ayushozha Claude Opus 4.6 commited on
Commit
65e91b3
·
1 Parent(s): 7ab6153

Update HF Space: final architecture diagram, frontend, and README

Browse files

- Add ReplicaLab_Architecture_Final.svg with full scoring engine,
reward formula, penalties, and 4-dimension breakdown
- Update README to reference final diagram and reflect 100% completion
- Include all latest frontend components (demo flow, training page,
episode results, compare page updates)
- Updated API client, types, and demo data for complete UI

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md CHANGED
@@ -16,7 +16,7 @@ pinned: false
16
 
17
  ReplicaLab tackles this by training an AI Scientist agent to negotiate feasible replication plans under realistic resource constraints. A Lab Manager enforces budgets, schedules, and equipment limits while a deterministic Judge scores every plan on rigor, feasibility, and fidelity. Through reinforcement learning, the Scientist learns to ask better questions, make smarter tradeoffs, and reach agreement faster -- all without sacrificing scientific quality.
18
 
19
- The initial domain focus is mathematics and machine learning, with offline finance and trading design as the third scenario family. Physics and biology remain future adapters after the core normalized scenario layer is stable.
20
 
21
  ## Team Ownership
22
 
@@ -32,7 +32,7 @@ The initial domain focus is mathematics and machine learning, with offline finan
32
  ## Architecture
33
 
34
  <p align="center">
35
- <img src="./ReplicaLab_Architecture_v2.svg" alt="ReplicaLab Hybrid Architecture" width="100%"/>
36
  </p>
37
 
38
  ReplicaLab uses a **hybrid Oracle architecture**:
@@ -174,7 +174,7 @@ RL training improves the Scientist agent’s ability to negotiate effective, fea
174
  - **Reduced-scale fallback:** `Qwen/Qwen3-4B`
175
  - **Decision record:** `docs/agt11_scientist_model_selection.md`
176
 
177
- ### Planned Training Path
178
 
179
  1. Use `notebooks/train_minimal_colab.ipynb` as the sponsor-facing minimal Colab script for the Unsloth / HF TRL requirement
180
  2. Use the judged notebook `notebooks/train_colab.ipynb` as the full readable driver
@@ -244,7 +244,7 @@ Difficulty scaling should mechanically tighten constraints, remove resources, or
244
  ```text
245
  replicalab-ai/
246
  ├── README.md
247
- ├── ReplicaLab_Architecture_v2.svg
248
  ├── pyproject.toml
249
  ├── openenv.yaml
250
  ├── replicalab/
 
16
 
17
  ReplicaLab tackles this by training an AI Scientist agent to negotiate feasible replication plans under realistic resource constraints. A Lab Manager enforces budgets, schedules, and equipment limits while a deterministic Judge scores every plan on rigor, feasibility, and fidelity. Through reinforcement learning, the Scientist learns to ask better questions, make smarter tradeoffs, and reach agreement faster -- all without sacrificing scientific quality.
18
 
19
+ Three scenario families ship today -- mathematics reasoning, ML benchmark replication, and offline finance/trading backtest design -- each with easy, medium, and hard difficulty scaling. Physics and biology remain future adapters after the core normalized scenario layer is stable.
20
 
21
  ## Team Ownership
22
 
 
32
  ## Architecture
33
 
34
  <p align="center">
35
+ <img src="./ReplicaLab_Architecture_Final.svg" alt="ReplicaLab Final System Architecture" width="100%"/>
36
  </p>
37
 
38
  ReplicaLab uses a **hybrid Oracle architecture**:
 
174
  - **Reduced-scale fallback:** `Qwen/Qwen3-4B`
175
  - **Decision record:** `docs/agt11_scientist_model_selection.md`
176
 
177
+ ### Training Path
178
 
179
  1. Use `notebooks/train_minimal_colab.ipynb` as the sponsor-facing minimal Colab script for the Unsloth / HF TRL requirement
180
  2. Use the judged notebook `notebooks/train_colab.ipynb` as the full readable driver
 
244
  ```text
245
  replicalab-ai/
246
  ├── README.md
247
+ ├── ReplicaLab_Architecture_Final.svg
248
  ├── pyproject.toml
249
  ├── openenv.yaml
250
  ├── replicalab/
ReplicaLab_Architecture_Final.svg ADDED

Git LFS Details

  • SHA256: fecb2c1df4e87cc0a4f5aeb90c9301bf22abf461d1c21e3c4a055340ab79f37f
  • Pointer size: 130 Bytes
  • Size of remote file: 38.8 kB
docs/changes.md CHANGED
@@ -81,4 +81,6 @@ Rules:
81
  | 2026-03-08 | Person B (Ayush) | Frontend episode kickoff UX | Added explicit first-round call-to-action controls to the live episode view after user testing showed the page looked stuck immediately after reset | The reset state loaded the paper and constraints correctly, but both the step controls and protocol-editor submit action could sit below the fold, making the UI appear frozen at round 0 | Added an `Episode ready` banner plus in-panel `Advance First Round` and `Open Protocol Editor` actions, and updated the negotiation placeholder so it no longer says `Start an episode` after an episode is already active | Keep a visible first action near the negotiation panel in future layout changes |
82
  | 2026-03-08 | Person B (Ayush) | One-click live demo automation | Extended the hero CTA flow so `Replicate a Paper` runs the seeded episode automatically instead of only opening the episode page | The hackathon demo needs a one-click narrative with live agent behavior and judge output; requiring manual reset and step clicks after entering the episode page weakens the demo | The dashboard now links to a seeded demo URL, the episode page auto-starts on demo routes, preserves demo query params in the shareable URL, and enables autoplay so the negotiation proceeds to judged completion with no extra clicks | If this is later generalized beyond the hero CTA, keep manual scenario-card entry as a separate non-demo path |
83
  | 2026-03-08 | Person B (Ayush) | Frontend backend-availability diagnostics | Added a startup health check and contextual network-error messaging to the replication setup flow after the live demo surfaced a generic `Failed to fetch` banner | The page gave no actionable explanation when the local API server on port `7860` was not running, which made a recoverable environment issue look like an application bug | `Controls.tsx` now checks backend health on load, `frontend/src/lib/api.ts` rewrites fetch-network failures into an explicit uvicorn startup instruction, the frontend was rebuilt, and the local FastAPI server was restarted and verified healthy on `127.0.0.1:7860` | Keep the backend running during demo prep or use the integrated backend-served frontend on `http://127.0.0.1:7860` |
 
 
84
 
 
81
  | 2026-03-08 | Person B (Ayush) | Frontend episode kickoff UX | Added explicit first-round call-to-action controls to the live episode view after user testing showed the page looked stuck immediately after reset | The reset state loaded the paper and constraints correctly, but both the step controls and protocol-editor submit action could sit below the fold, making the UI appear frozen at round 0 | Added an `Episode ready` banner plus in-panel `Advance First Round` and `Open Protocol Editor` actions, and updated the negotiation placeholder so it no longer says `Start an episode` after an episode is already active | Keep a visible first action near the negotiation panel in future layout changes |
82
  | 2026-03-08 | Person B (Ayush) | One-click live demo automation | Extended the hero CTA flow so `Replicate a Paper` runs the seeded episode automatically instead of only opening the episode page | The hackathon demo needs a one-click narrative with live agent behavior and judge output; requiring manual reset and step clicks after entering the episode page weakens the demo | The dashboard now links to a seeded demo URL, the episode page auto-starts on demo routes, preserves demo query params in the shareable URL, and enables autoplay so the negotiation proceeds to judged completion with no extra clicks | If this is later generalized beyond the hero CTA, keep manual scenario-card entry as a separate non-demo path |
83
  | 2026-03-08 | Person B (Ayush) | Frontend backend-availability diagnostics | Added a startup health check and contextual network-error messaging to the replication setup flow after the live demo surfaced a generic `Failed to fetch` banner | The page gave no actionable explanation when the local API server on port `7860` was not running, which made a recoverable environment issue look like an application bug | `Controls.tsx` now checks backend health on load, `frontend/src/lib/api.ts` rewrites fetch-network failures into an explicit uvicorn startup instruction, the frontend was rebuilt, and the local FastAPI server was restarted and verified healthy on `127.0.0.1:7860` | Keep the backend running during demo prep or use the integrated backend-served frontend on `http://127.0.0.1:7860` |
84
+ | 2026-03-08 | Person B (Ayush) | Frontend live demo outcomes and results report | Extended the one-click demo into three seeded story modes with a detailed post-episode report instead of a single autoplay path that stopped at the generic episode view | The hackathon demo needs to show distinct judged outcomes: immediate agreement, multi-round learning opportunity, and failure to reach agreement, all backed by real episode values rather than static mock copy | Added `fast-agreement`, `learning-opportunity`, and `no-agreement` dashboard launches; routed episode autoplay through scripted but backend-valid action sequences; added a results report with live reward charts, terminal score bars, training interpretation, reliability labeling, and tool-install suggestions; rebuilt the frontend and verified all three seeded ML runs against the live backend | If Oracle-backed narrative summaries are added later, keep the deterministic judge verdict and real score traces as the source of truth for the report |
85
+ | 2026-03-08 | Person B (Ayush) | Frontend training-status reporting | Replaced the packaged training teaser with an artifact-backed training page and honest improvement guidance | The demo needs a place to show training logs, real achieved values, and whether more training is still required; the earlier dashboard card implied progress but did not expose the real run outputs | Added a dedicated `/training` route, header navigation, a shared frontend data module sourced from real run summaries, and a new training page with checkpoint charts, compare bars, log highlights, preview-artifact status, and explicit `needs more training` analysis; rebuilt the frontend and reverified backend serving on `/training` | If future runs improve beyond baseline, update the shared training artifact data module first so the dashboard and training page stay consistent |
86
 
docs/demo_video_script_60s.md ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ReplicaLab 60s Demo Script
2
+
3
+ ## Voiceover
4
+
5
+ ReplicaLab starts from a research paper and turns it into a seeded replication benchmark. The Scientist proposes a protocol, the Lab Manager enforces budget, tools, and scheduling, and a deterministic Judge scores rigor, feasibility, and fidelity. In our first scenario, the agents agree immediately, so the paper looks replicable in this lab. In the second scenario, they negotiate across all six rounds, which creates a rich reinforcement learning signal. In the third, they never resolve the blockers, so the system rejects the paper for the current setup. Because every outcome is scored deterministically, we can train the Scientist with Unsloth and TRL, compare baseline versus trained runs, inspect real logs, and see exactly where more learning is still needed. The training page is intentionally honest: the live run reached positive rewards, but the held-out compare still shows that the trained Scientist has not beaten the deterministic baseline yet.
6
+
7
+ ## Shot List
8
+
9
+ 1. Dashboard hero: introduce ReplicaLab and the paper-to-training loop.
10
+ 2. First-round agreement: show a clean acceptance and high replicability score.
11
+ 3. Multi-round learning: show six-round negotiation and the learning-opportunity results panel.
12
+ 4. No agreement: show the timeout / rejection outcome and low reliability signal.
13
+ 5. Training page: show artifact-backed logs, checkpoints, baseline-vs-trained compare, and the explicit note that more training is still required.
docs/kush/notes.md CHANGED
@@ -17,3 +17,28 @@ Durable deviations belong in `docs/changes.md`.
17
  - The dashboard `Replicate a Paper` CTA now launches a seeded live demo automatically: reset, first proposal, autoplay, and judged completion all happen without extra clicks.
18
  - The replication setup card now performs a backend health check up front and surfaces a concrete startup command instead of the opaque browser-level `Failed to fetch` message when the API server is down.
19
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  - The dashboard `Replicate a Paper` CTA now launches a seeded live demo automatically: reset, first proposal, autoplay, and judged completion all happen without extra clicks.
18
  - The replication setup card now performs a backend health check up front and surfaces a concrete startup command instead of the opaque browser-level `Failed to fetch` message when the API server is down.
19
 
20
+ ## 2026-03-08 three-outcome live demo
21
+
22
+ - The live demo now has three seeded story modes on the dashboard: `fast-agreement`, `learning-opportunity`, and `no-agreement`.
23
+ - Each mode runs against the real backend with deterministic episode data and renders a post-episode results report instead of stopping at a generic terminal state.
24
+ - The results report now shows executed rounds, disagreement count, replicability score, paper reliability quality, reward and score charts, training interpretation, and next-tool suggestions.
25
+ - Verified backend-driven outputs for the current seeded ML demo cases:
26
+ - `fast-agreement` -> round `2`, verdict `accept`, cumulative reward `2.906845`
27
+ - `learning-opportunity` -> round `6`, verdict `accept`, cumulative reward `4.537097`
28
+ - `no-agreement` -> round `6`, verdict `timeout`, cumulative reward `0.366529`
29
+
30
+ ## 2026-03-08 training page with real artifacts
31
+
32
+ - Added a dedicated `/training` page instead of relying on the old packaged dashboard card.
33
+ - The new page is backed by real artifact values from the existing outputs:
34
+ - local deterministic baseline summary
35
+ - live ART/OpenEnv scientist checkpoints
36
+ - seeded hold-out compare summary
37
+ - scientist and lab-manager preview summaries
38
+ - The training story is now explicit and honest:
39
+ - the training pipeline works
40
+ - live reward moved positive by later checkpoints
41
+ - hold-out compare still shows the trained Scientist underperforming baseline
42
+ - more training and parser/invalid-action cleanup are still needed
43
+ - Header nav now includes `Training`, dashboard training CTA points there, and the dashboard training teaser uses the same artifact-backed data.
44
+
frontend/src/App.tsx CHANGED
@@ -2,6 +2,7 @@ import { Routes, Route } from 'react-router-dom';
2
  import EpisodePage from '@/pages/EpisodePage';
3
  import DashboardPage from '@/pages/DashboardPage';
4
  import ComparePage from '@/pages/ComparePage';
 
5
  import Header from '@/components/Header';
6
  import { ToastProvider } from '@/components/Toast';
7
  import Onboarding, { useOnboarding } from '@/components/Onboarding';
@@ -17,6 +18,7 @@ export default function App() {
17
  <Route path="/" element={<DashboardPage />} />
18
  <Route path="/episode" element={<EpisodePage />} />
19
  <Route path="/episode/:episodeId" element={<EpisodePage />} />
 
20
  <Route path="/compare" element={<ComparePage />} />
21
  </Routes>
22
  <Onboarding show={showOnboarding} onDismiss={dismissOnboarding} />
 
2
  import EpisodePage from '@/pages/EpisodePage';
3
  import DashboardPage from '@/pages/DashboardPage';
4
  import ComparePage from '@/pages/ComparePage';
5
+ import TrainingPage from '@/pages/TrainingPage';
6
  import Header from '@/components/Header';
7
  import { ToastProvider } from '@/components/Toast';
8
  import Onboarding, { useOnboarding } from '@/components/Onboarding';
 
18
  <Route path="/" element={<DashboardPage />} />
19
  <Route path="/episode" element={<EpisodePage />} />
20
  <Route path="/episode/:episodeId" element={<EpisodePage />} />
21
+ <Route path="/training" element={<TrainingPage />} />
22
  <Route path="/compare" element={<ComparePage />} />
23
  </Routes>
24
  <Onboarding show={showOnboarding} onDismiss={dismissOnboarding} />
frontend/src/components/EpisodeComparison.tsx CHANGED
@@ -41,7 +41,7 @@ export default function EpisodeComparison({ className }: { className?: string })
41
  // Auto-play through all rounds
42
  while (!state.done) {
43
  const isLastRound = state.round >= state.max_rounds - 1;
44
- const action = isLastRound ? buildAcceptAction() : buildDefaultScientistAction();
45
  state = await stepEpisode(state.session_id, action, state);
46
  }
47
 
 
41
  // Auto-play through all rounds
42
  while (!state.done) {
43
  const isLastRound = state.round >= state.max_rounds - 1;
44
+ const action = isLastRound ? buildAcceptAction() : buildDefaultScientistAction(state);
45
  state = await stepEpisode(state.session_id, action, state);
46
  }
47
 
frontend/src/components/EpisodeResultsReport.tsx ADDED
@@ -0,0 +1,465 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import type { ReactNode } from 'react';
2
+ import {
3
+ CheckCircle2,
4
+ AlertTriangle,
5
+ XCircle,
6
+ BrainCircuit,
7
+ TrendingUp,
8
+ Wrench,
9
+ Gauge,
10
+ FileCheck2,
11
+ } from 'lucide-react';
12
+ import {
13
+ ResponsiveContainer,
14
+ LineChart,
15
+ Line,
16
+ CartesianGrid,
17
+ XAxis,
18
+ YAxis,
19
+ Tooltip,
20
+ Legend,
21
+ BarChart,
22
+ Bar,
23
+ } from 'recharts';
24
+ import type { EpisodeState } from '@/types';
25
+ import { cn, formatReward } from '@/lib/utils';
26
+
27
+ interface EpisodeResultsReportProps {
28
+ episode: EpisodeState;
29
+ className?: string;
30
+ }
31
+
32
+ type OutcomeProfile = {
33
+ tone: 'good' | 'learning' | 'reject';
34
+ title: string;
35
+ subtitle: string;
36
+ badge: string;
37
+ icon: typeof CheckCircle2;
38
+ };
39
+
40
+ const TOOL_STACK: Record<string, { title: string; packages: string[]; commands: string[] }> = {
41
+ math_reasoning: {
42
+ title: 'Suggested proof-and-verification stack',
43
+ packages: ['SymPy', 'pytest', 'JupyterLab', 'LaTeX tooling'],
44
+ commands: [
45
+ 'pip install sympy pytest jupyterlab',
46
+ 'Install TeX distribution for reproducible proof write-ups',
47
+ ],
48
+ },
49
+ ml_benchmark: {
50
+ title: 'Suggested replication toolchain',
51
+ packages: ['PyTorch', 'torchvision', 'Weights & Biases', 'datasets'],
52
+ commands: [
53
+ 'pip install torch torchvision wandb datasets',
54
+ 'Install NVIDIA CUDA toolkit / drivers that match the available GPU node',
55
+ ],
56
+ },
57
+ finance_trading: {
58
+ title: 'Suggested backtest stack',
59
+ packages: ['pandas', 'numpy', 'vectorbt', 'matplotlib'],
60
+ commands: [
61
+ 'pip install pandas numpy vectorbt matplotlib',
62
+ 'Add a versioned market-data export before rerunning the benchmark',
63
+ ],
64
+ },
65
+ };
66
+
67
+ function computeConfidencePercent(episode: EpisodeState): number {
68
+ const scores = episode.scores;
69
+ if (!scores) {
70
+ return 0;
71
+ }
72
+
73
+ const weighted =
74
+ scores.rigor * 0.4 +
75
+ scores.feasibility * 0.35 +
76
+ scores.fidelity * 0.25;
77
+ return Math.round(weighted * 100);
78
+ }
79
+
80
+ function buildReliabilityLabel(episode: EpisodeState): string {
81
+ if (episode.judge_audit?.verdict === 'accept' && episode.step_history[0]?.lab_manager_action_type === 'accept') {
82
+ return 'Good';
83
+ }
84
+ if (episode.judge_audit?.verdict === 'accept') {
85
+ return 'Needs iteration';
86
+ }
87
+ return 'Bad';
88
+ }
89
+
90
+ function buildOutcomeProfile(episode: EpisodeState): OutcomeProfile {
91
+ const firstRoundAccepted =
92
+ episode.step_history[0]?.lab_manager_action_type === 'accept' &&
93
+ episode.judge_audit?.verdict === 'accept';
94
+ const disagreementRounds = episode.step_history.filter(
95
+ (trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
96
+ ).length;
97
+
98
+ if (firstRoundAccepted) {
99
+ return {
100
+ tone: 'good',
101
+ title: 'Completed: First-round agreement',
102
+ subtitle: 'The paper looks replicable under the current constraints, and the agents converged immediately.',
103
+ badge: 'Good paper / strong replication candidate',
104
+ icon: CheckCircle2,
105
+ };
106
+ }
107
+
108
+ if (episode.judge_audit?.verdict === 'accept') {
109
+ return {
110
+ tone: 'learning',
111
+ title: 'Completed: Multi-round learning opportunity',
112
+ subtitle: `${disagreementRounds} disagreement round(s) were resolved into a feasible final protocol.`,
113
+ badge: 'Heavy RL learning opportunity',
114
+ icon: AlertTriangle,
115
+ };
116
+ }
117
+
118
+ return {
119
+ tone: 'reject',
120
+ title: 'Completed: No agreement reached',
121
+ subtitle: 'The protocol never became acceptable within the six-round limit, so the paper is rejected for this lab setup.',
122
+ badge: 'Cannot replicate under current setup',
123
+ icon: XCircle,
124
+ };
125
+ }
126
+
127
+ function aggregateRewardComponents(episode: EpisodeState): Array<{ key: string; value: number }> {
128
+ const totals = new Map<string, number>();
129
+ for (const trace of episode.step_history) {
130
+ for (const [key, value] of Object.entries(trace.step_reward_components)) {
131
+ totals.set(key, (totals.get(key) ?? 0) + value);
132
+ }
133
+ }
134
+
135
+ return [...totals.entries()]
136
+ .map(([key, value]) => ({ key, value: Number(value.toFixed(3)) }))
137
+ .sort((a, b) => Math.abs(b.value) - Math.abs(a.value));
138
+ }
139
+
140
+ function buildLearningBullets(episode: EpisodeState): string[] {
141
+ const firstProtocol = episode.step_history.find((trace) => trace.protocol)?.protocol;
142
+ const finalProtocol = episode.protocol;
143
+ const bullets: string[] = [];
144
+
145
+ if (firstProtocol && finalProtocol && firstProtocol.sample_size !== finalProtocol.sample_size) {
146
+ bullets.push(
147
+ `Sample size moved from ${firstProtocol.sample_size} to ${finalProtocol.sample_size}, showing explicit adaptation to budget or staff feedback.`,
148
+ );
149
+ }
150
+
151
+ if (firstProtocol && finalProtocol && firstProtocol.controls.length !== finalProtocol.controls.length) {
152
+ bullets.push(
153
+ `Controls were narrowed from ${firstProtocol.controls.length} to ${finalProtocol.controls.length}, which reduced lab pressure while preserving the core benchmark check.`,
154
+ );
155
+ }
156
+
157
+ if (episode.step_history.some((trace) => trace.lab_manager_action_type === 'suggest_alternative')) {
158
+ bullets.push('The Scientist absorbed grounded Lab Manager alternatives instead of ignoring the resource constraints.');
159
+ }
160
+
161
+ if (episode.step_history.some((trace) => trace.step_reward_components.momentum_bonus)) {
162
+ bullets.push('Later revisions earned momentum bonuses, meaning the protocol improved rather than stalling.');
163
+ }
164
+
165
+ if (!bullets.length) {
166
+ bullets.push('The episode converged quickly, so the strongest training value is as a positive exemplar for future rollouts.');
167
+ }
168
+
169
+ return bullets;
170
+ }
171
+
172
+ function buildImprovementBullets(episode: EpisodeState): string[] {
173
+ const scores = episode.scores;
174
+ const verdict = episode.judge_audit?.verdict ?? 'unknown';
175
+ const suggestions: string[] = [];
176
+
177
+ if (!scores) {
178
+ suggestions.push('Provision more compute, staff, or schedule slack before attempting another replication run.');
179
+ suggestions.push('Reduce sample size and control scope earlier to avoid spending the full round budget on infeasible plans.');
180
+ return suggestions;
181
+ }
182
+
183
+ if (scores.rigor < 0.7) {
184
+ suggestions.push('Improve rigor by tightening success criteria, controls, and evaluation checkpoints in the proposed protocol.');
185
+ }
186
+ if (scores.feasibility < 0.9) {
187
+ suggestions.push('Improve feasibility by reducing workload earlier or aligning the protocol more closely with the available staff and equipment.');
188
+ }
189
+ if (scores.fidelity < 0.7) {
190
+ suggestions.push('Improve fidelity by matching the original method and baseline recipe more closely.');
191
+ }
192
+ if (verdict !== 'accept') {
193
+ suggestions.push('Use this rollout as a negative exemplar in RL training so the Scientist learns to avoid repeated non-feasible plans.');
194
+ }
195
+ if (!suggestions.length) {
196
+ suggestions.push('This is a strong positive exemplar; the next improvement is broader held-out evaluation rather than protocol repair.');
197
+ }
198
+
199
+ return suggestions;
200
+ }
201
+
202
+ export default function EpisodeResultsReport({ episode, className }: EpisodeResultsReportProps) {
203
+ const profile = buildOutcomeProfile(episode);
204
+ const confidencePercent = computeConfidencePercent(episode);
205
+ const reliabilityLabel = buildReliabilityLabel(episode);
206
+ const disagreementRounds = episode.step_history.filter(
207
+ (trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
208
+ ).length;
209
+ const chartData = episode.step_history.map((trace) => ({
210
+ round: `R${trace.round}`,
211
+ reward: Number(trace.reward.toFixed(3)),
212
+ cumulative: Number(trace.cumulative_reward.toFixed(3)),
213
+ }));
214
+ const scoreBars = [
215
+ { key: 'rigor', value: episode.scores?.rigor ?? 0 },
216
+ { key: 'feasibility', value: episode.scores?.feasibility ?? 0 },
217
+ { key: 'fidelity', value: episode.scores?.fidelity ?? 0 },
218
+ { key: 'parsimony', value: episode.scores?.parsimony ?? 0 },
219
+ ];
220
+ const rewardComponents = aggregateRewardComponents(episode);
221
+ const toolStack = TOOL_STACK[episode.template];
222
+ const ProfileIcon = profile.icon;
223
+
224
+ return (
225
+ <section className={cn('rounded-2xl border border-border bg-card p-6', className)}>
226
+ <div className="flex flex-col gap-4 lg:flex-row lg:items-start lg:justify-between">
227
+ <div>
228
+ <div
229
+ className={cn(
230
+ 'mb-3 inline-flex items-center gap-2 rounded-full px-3 py-1 text-xs font-semibold',
231
+ profile.tone === 'good' && 'bg-lab-manager/10 text-lab-manager',
232
+ profile.tone === 'learning' && 'bg-judge/10 text-judge',
233
+ profile.tone === 'reject' && 'bg-destructive/10 text-destructive',
234
+ )}
235
+ >
236
+ <ProfileIcon className="h-4 w-4" />
237
+ {profile.badge}
238
+ </div>
239
+ <h2 className="text-2xl font-bold">{profile.title}</h2>
240
+ <p className="mt-2 max-w-3xl text-sm text-muted-foreground">{profile.subtitle}</p>
241
+ <p className="mt-3 max-w-3xl text-sm">
242
+ {profile.tone !== 'reject'
243
+ ? `ReplicaLab scores this paper as replicable in the current lab setup with ${confidencePercent}% confidence.`
244
+ : `ReplicaLab rejects this paper for the current setup. The paper reliability score is ${confidencePercent}% until the blocking constraints are addressed.`}
245
+ </p>
246
+ </div>
247
+
248
+ <div className="grid gap-3 sm:grid-cols-2 lg:w-[360px]">
249
+ <MetricCard
250
+ icon={<Gauge className="h-4 w-4" />}
251
+ label="Replicability score"
252
+ value={`${confidencePercent}%`}
253
+ hint="Derived from rigor, feasibility, and fidelity."
254
+ />
255
+ <MetricCard
256
+ icon={<CheckCircle2 className="h-4 w-4" />}
257
+ label="Paper reliability quality"
258
+ value={reliabilityLabel}
259
+ hint="Good, learning opportunity, or bad based on the judged outcome."
260
+ />
261
+ <MetricCard
262
+ icon={<FileCheck2 className="h-4 w-4" />}
263
+ label="Judge verdict"
264
+ value={episode.judge_audit?.verdict ?? 'unknown'}
265
+ hint="Canonical deterministic decision."
266
+ />
267
+ <MetricCard
268
+ icon={<TrendingUp className="h-4 w-4" />}
269
+ label="Total reward"
270
+ value={formatReward(episode.cumulative_reward)}
271
+ hint={`Cumulative reward across ${episode.step_history.length} executed round(s).`}
272
+ />
273
+ </div>
274
+ </div>
275
+
276
+ <div className="mt-4 rounded-xl border border-border bg-muted/30 p-4">
277
+ <div className="flex flex-wrap items-center gap-2 text-xs font-medium">
278
+ <span className="rounded-full bg-background px-2.5 py-1 text-primary">
279
+ Executed rounds: {episode.step_history.length} / {episode.max_rounds}
280
+ </span>
281
+ <span className="rounded-full bg-background px-2.5 py-1 text-primary">
282
+ Disagreement rounds: {disagreementRounds}
283
+ </span>
284
+ <span className="rounded-full bg-background px-2.5 py-1 text-primary">
285
+ Final status: {episode.done ? 'completed' : 'in progress'}
286
+ </span>
287
+ </div>
288
+ </div>
289
+
290
+ <div className="mt-6 grid gap-4 xl:grid-cols-2">
291
+ <div className="rounded-xl border border-border bg-muted/30 p-4">
292
+ <h3 className="mb-2 text-sm font-semibold">Round-by-round reward trajectory</h3>
293
+ <p className="mb-4 text-xs text-muted-foreground">
294
+ This is real episode data from the run you just watched, not static demo text.
295
+ </p>
296
+ <div className="h-60">
297
+ <ResponsiveContainer width="100%" height="100%">
298
+ <LineChart data={chartData}>
299
+ <CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
300
+ <XAxis dataKey="round" tick={{ fontSize: 11 }} />
301
+ <YAxis tick={{ fontSize: 11 }} />
302
+ <Tooltip />
303
+ <Legend wrapperStyle={{ fontSize: '12px' }} />
304
+ <Line type="monotone" dataKey="reward" stroke="var(--color-primary)" strokeWidth={2.5} name="Step reward" />
305
+ <Line type="monotone" dataKey="cumulative" stroke="var(--color-lab-manager)" strokeWidth={2.5} name="Cumulative reward" />
306
+ </LineChart>
307
+ </ResponsiveContainer>
308
+ </div>
309
+ </div>
310
+
311
+ <div className="rounded-xl border border-border bg-muted/30 p-4">
312
+ <h3 className="mb-2 text-sm font-semibold">Terminal score profile</h3>
313
+ <p className="mb-4 text-xs text-muted-foreground">
314
+ The final verdict depends on these component scores plus the terminal rubric.
315
+ </p>
316
+ <div className="h-60">
317
+ <ResponsiveContainer width="100%" height="100%">
318
+ <BarChart data={scoreBars}>
319
+ <CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
320
+ <XAxis dataKey="key" tick={{ fontSize: 11 }} />
321
+ <YAxis domain={[0, 1]} tickFormatter={(value) => `${Math.round(value * 100)}%`} tick={{ fontSize: 11 }} />
322
+ <Tooltip />
323
+ <Bar dataKey="value" fill="var(--color-judge)" radius={[6, 6, 0, 0]} />
324
+ </BarChart>
325
+ </ResponsiveContainer>
326
+ </div>
327
+ </div>
328
+ </div>
329
+
330
+ <div className="mt-6 grid gap-4 xl:grid-cols-3">
331
+ <InsightCard
332
+ title="What the agents learned"
333
+ icon={<BrainCircuit className="h-4 w-4" />}
334
+ items={buildLearningBullets(episode)}
335
+ />
336
+ <InsightCard
337
+ title="What to improve next"
338
+ icon={<TrendingUp className="h-4 w-4" />}
339
+ items={buildImprovementBullets(episode)}
340
+ />
341
+ <InsightCard
342
+ title="Next tools to install"
343
+ icon={<Wrench className="h-4 w-4" />}
344
+ items={[
345
+ toolStack.title,
346
+ ...toolStack.packages.map((item) => `Install ${item}`),
347
+ ...toolStack.commands,
348
+ ]}
349
+ />
350
+ </div>
351
+
352
+ <div className="mt-6 grid gap-4 xl:grid-cols-[1.5fr_1fr]">
353
+ <div className="rounded-xl border border-border bg-muted/30 p-4">
354
+ <h3 className="mb-3 text-sm font-semibold">Episode timeline</h3>
355
+ <div className="space-y-3">
356
+ {episode.step_history.map((trace) => (
357
+ <div key={trace.round} className="rounded-lg border border-border bg-background p-3">
358
+ <div className="flex flex-wrap items-center gap-2">
359
+ <span className="rounded-full bg-primary/10 px-2 py-0.5 text-[11px] font-semibold text-primary">
360
+ Round {trace.round}
361
+ </span>
362
+ <span className="rounded-full bg-muted px-2 py-0.5 text-[11px] text-muted-foreground">
363
+ Scientist: {trace.action_type.replace(/_/g, ' ')}
364
+ </span>
365
+ {trace.lab_manager_action_type && (
366
+ <span className="rounded-full bg-muted px-2 py-0.5 text-[11px] text-muted-foreground">
367
+ Lab Manager: {trace.lab_manager_action_type.replace(/_/g, ' ')}
368
+ </span>
369
+ )}
370
+ <span className="ml-auto text-xs font-semibold text-foreground">
371
+ {formatReward(trace.reward)}
372
+ </span>
373
+ </div>
374
+ <p className="mt-2 text-xs text-muted-foreground">{trace.scientist_message}</p>
375
+ {trace.lab_manager_message && (
376
+ <p className="mt-1 text-xs text-muted-foreground">{trace.lab_manager_message}</p>
377
+ )}
378
+ </div>
379
+ ))}
380
+ </div>
381
+ </div>
382
+
383
+ <div className="rounded-xl border border-border bg-muted/30 p-4">
384
+ <h3 className="mb-3 text-sm font-semibold">Reward component ledger</h3>
385
+ <div className="space-y-2">
386
+ {rewardComponents.length ? rewardComponents.map((entry) => (
387
+ <div key={entry.key} className="flex items-center justify-between rounded-lg border border-border bg-background px-3 py-2 text-xs">
388
+ <span className="capitalize text-muted-foreground">{entry.key.replace(/_/g, ' ')}</span>
389
+ <span className={cn('font-semibold', entry.value >= 0 ? 'text-lab-manager' : 'text-destructive')}>
390
+ {entry.value >= 0 ? '+' : ''}
391
+ {entry.value.toFixed(3)}
392
+ </span>
393
+ </div>
394
+ )) : (
395
+ <div className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
396
+ No intermediate reward components were emitted in this episode.
397
+ </div>
398
+ )}
399
+ </div>
400
+
401
+ <div className="mt-4 rounded-lg border border-border bg-background p-3">
402
+ <div className="mb-2 text-xs font-semibold">Training interpretation</div>
403
+ <p className="text-xs text-muted-foreground">
404
+ {profile.tone === 'good' &&
405
+ 'Use this run as a positive exemplar: the policy found a feasible replication plan with minimal negotiation overhead.'}
406
+ {profile.tone === 'learning' &&
407
+ 'This is the richest RL case: multiple disagreement rounds created dense reward signals for the Scientist to learn from.'}
408
+ {profile.tone === 'reject' &&
409
+ 'This is a strong negative exemplar: the policy spent all six rounds without resolving the key feasibility blockers.'}
410
+ </p>
411
+ </div>
412
+ </div>
413
+ </div>
414
+ </section>
415
+ );
416
+ }
417
+
418
+ function MetricCard({
419
+ icon,
420
+ label,
421
+ value,
422
+ hint,
423
+ }: {
424
+ icon: ReactNode;
425
+ label: string;
426
+ value: string;
427
+ hint: string;
428
+ }) {
429
+ return (
430
+ <div className="rounded-xl border border-border bg-muted/30 p-3">
431
+ <div className="mb-2 flex items-center gap-2 text-xs text-muted-foreground">
432
+ {icon}
433
+ {label}
434
+ </div>
435
+ <div className="text-lg font-bold">{value}</div>
436
+ <p className="mt-1 text-[11px] text-muted-foreground">{hint}</p>
437
+ </div>
438
+ );
439
+ }
440
+
441
+ function InsightCard({
442
+ title,
443
+ icon,
444
+ items,
445
+ }: {
446
+ title: string;
447
+ icon: ReactNode;
448
+ items: string[];
449
+ }) {
450
+ return (
451
+ <div className="rounded-xl border border-border bg-muted/30 p-4">
452
+ <div className="mb-3 flex items-center gap-2 text-sm font-semibold">
453
+ {icon}
454
+ {title}
455
+ </div>
456
+ <div className="space-y-2">
457
+ {items.map((item) => (
458
+ <div key={item} className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
459
+ {item}
460
+ </div>
461
+ ))}
462
+ </div>
463
+ </div>
464
+ );
465
+ }
frontend/src/components/Header.tsx CHANGED
@@ -1,6 +1,6 @@
1
  import { useState } from 'react';
2
  import { Link, useLocation } from 'react-router-dom';
3
- import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows } from 'lucide-react';
4
  import { cn } from '@/lib/utils';
5
  import { useTheme } from '@/lib/useTheme';
6
  import { toggleMute, isMuted } from '@/lib/audio';
@@ -8,6 +8,7 @@ import { toggleMute, isMuted } from '@/lib/audio';
8
  const navItems = [
9
  { to: '/', label: 'Dashboard', icon: LayoutDashboard },
10
  { to: '/episode', label: 'Episode', icon: Play },
 
11
  { to: '/compare', label: 'Compare', icon: GitCompareArrows },
12
  ];
13
 
 
1
  import { useState } from 'react';
2
  import { Link, useLocation } from 'react-router-dom';
3
+ import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows, GraduationCap } from 'lucide-react';
4
  import { cn } from '@/lib/utils';
5
  import { useTheme } from '@/lib/useTheme';
6
  import { toggleMute, isMuted } from '@/lib/audio';
 
8
  const navItems = [
9
  { to: '/', label: 'Dashboard', icon: LayoutDashboard },
10
  { to: '/episode', label: 'Episode', icon: Play },
11
+ { to: '/training', label: 'Training', icon: GraduationCap },
12
  { to: '/compare', label: 'Compare', icon: GitCompareArrows },
13
  ];
14
 
frontend/src/components/TrainingResults.tsx CHANGED
@@ -20,6 +20,7 @@ import {
20
  Legend,
21
  } from 'recharts';
22
  import type { TrainingComparison } from '@/types';
 
23
  import { cn, formatReward, formatScore } from '@/lib/utils';
24
 
25
  interface TrainingResultsProps {
@@ -27,49 +28,8 @@ interface TrainingResultsProps {
27
  className?: string;
28
  }
29
 
30
- const DEMO_COMPARISON: TrainingComparison = {
31
- baseline: [
32
- { episode: 1, reward: 3.8, rigor: 0.58, feasibility: 0.46, fidelity: 0.55, rounds_used: 5, agreement: false, invalid_actions: 2 },
33
- { episode: 2, reward: 4.0, rigor: 0.6, feasibility: 0.48, fidelity: 0.56, rounds_used: 5, agreement: false, invalid_actions: 2 },
34
- { episode: 3, reward: 4.1, rigor: 0.61, feasibility: 0.5, fidelity: 0.57, rounds_used: 5, agreement: true, invalid_actions: 1 },
35
- { episode: 4, reward: 4.2, rigor: 0.62, feasibility: 0.5, fidelity: 0.58, rounds_used: 4, agreement: true, invalid_actions: 1 },
36
- { episode: 5, reward: 4.0, rigor: 0.6, feasibility: 0.49, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
37
- { episode: 6, reward: 4.3, rigor: 0.63, feasibility: 0.52, fidelity: 0.59, rounds_used: 4, agreement: true, invalid_actions: 1 },
38
- { episode: 7, reward: 4.1, rigor: 0.61, feasibility: 0.5, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
39
- { episode: 8, reward: 4.4, rigor: 0.64, feasibility: 0.53, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
40
- { episode: 9, reward: 4.2, rigor: 0.62, feasibility: 0.51, fidelity: 0.59, rounds_used: 4, agreement: true, invalid_actions: 1 },
41
- { episode: 10, reward: 4.0, rigor: 0.6, feasibility: 0.48, fidelity: 0.57, rounds_used: 5, agreement: false, invalid_actions: 2 },
42
- { episode: 11, reward: 4.3, rigor: 0.64, feasibility: 0.52, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
43
- { episode: 12, reward: 4.1, rigor: 0.61, feasibility: 0.49, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
44
- ],
45
- trained: [
46
- { episode: 1, reward: 4.5, rigor: 0.64, feasibility: 0.56, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
47
- { episode: 2, reward: 4.8, rigor: 0.67, feasibility: 0.58, fidelity: 0.62, rounds_used: 4, agreement: true, invalid_actions: 1 },
48
- { episode: 3, reward: 5.0, rigor: 0.69, feasibility: 0.61, fidelity: 0.64, rounds_used: 4, agreement: true, invalid_actions: 1 },
49
- { episode: 4, reward: 5.2, rigor: 0.71, feasibility: 0.63, fidelity: 0.66, rounds_used: 3, agreement: true, invalid_actions: 1 },
50
- { episode: 5, reward: 5.5, rigor: 0.73, feasibility: 0.66, fidelity: 0.68, rounds_used: 3, agreement: true, invalid_actions: 0 },
51
- { episode: 6, reward: 5.7, rigor: 0.75, feasibility: 0.68, fidelity: 0.69, rounds_used: 3, agreement: true, invalid_actions: 0 },
52
- { episode: 7, reward: 5.9, rigor: 0.76, feasibility: 0.7, fidelity: 0.7, rounds_used: 3, agreement: true, invalid_actions: 0 },
53
- { episode: 8, reward: 6.1, rigor: 0.78, feasibility: 0.72, fidelity: 0.72, rounds_used: 3, agreement: true, invalid_actions: 0 },
54
- { episode: 9, reward: 6.3, rigor: 0.79, feasibility: 0.74, fidelity: 0.73, rounds_used: 3, agreement: true, invalid_actions: 0 },
55
- { episode: 10, reward: 6.5, rigor: 0.8, feasibility: 0.76, fidelity: 0.75, rounds_used: 3, agreement: true, invalid_actions: 0 },
56
- { episode: 11, reward: 6.6, rigor: 0.81, feasibility: 0.77, fidelity: 0.76, rounds_used: 2, agreement: true, invalid_actions: 0 },
57
- { episode: 12, reward: 6.8, rigor: 0.83, feasibility: 0.79, fidelity: 0.78, rounds_used: 2, agreement: true, invalid_actions: 0 },
58
- ],
59
- summary: {
60
- baseline_avg_reward: 4.13,
61
- trained_avg_reward: 5.74,
62
- baseline_agreement_rate: 0.5,
63
- trained_agreement_rate: 0.92,
64
- baseline_avg_rounds: 4.58,
65
- trained_avg_rounds: 3.0,
66
- baseline_invalid_rate: 0.15,
67
- trained_invalid_rate: 0.03,
68
- },
69
- };
70
-
71
  export default function TrainingResults({ data, className }: TrainingResultsProps) {
72
- const comparison = data ?? DEMO_COMPARISON;
73
  const [showTrained, setShowTrained] = useState(true);
74
  const chartData = comparison.baseline.map((baselinePoint, index) => ({
75
  episode: baselinePoint.episode,
@@ -96,18 +56,21 @@ export default function TrainingResults({ data, className }: TrainingResultsProp
96
  <span className="rounded-full bg-primary/10 px-2 py-1 text-[11px] font-medium text-primary">Minimal Colab notebook</span>
97
  <span className="rounded-full bg-scientist/10 px-2 py-1 text-[11px] font-medium text-scientist">Unsloth + HF TRL</span>
98
  <span className="rounded-full bg-judge/10 px-2 py-1 text-[11px] font-medium text-judge">Deterministic judge reward</span>
 
 
 
99
  </div>
100
 
101
  <p className="mb-4 text-sm text-muted-foreground">
102
- The same seeded replication tasks are used for baseline and trained runs. The Scientist only improves if it
103
- can negotiate stronger protocols under the exact same judge rubric.
104
  </p>
105
 
106
  <div className="mb-4 grid gap-2 sm:grid-cols-2 xl:grid-cols-4">
107
  <StoryCard icon={<FileText className="h-3.5 w-3.5" />} title="Notebook" detail="`train_minimal_colab.ipynb` is the sponsor-facing minimal GRPO path." />
108
  <StoryCard icon={<Cpu className="h-3.5 w-3.5" />} title="Trainer" detail="Unsloth + TRL update the Scientist LoRA adapter on seeded tasks." />
109
  <StoryCard icon={<FlaskConical className="h-3.5 w-3.5" />} title="Environment" detail="Episodes come from the same paper-derived ReplicaLab benchmark families." />
110
- <StoryCard icon={<Scale className="h-3.5 w-3.5" />} title="Reward" detail="Rigor, feasibility, and fidelity stay deterministic for clean comparisons." />
111
  </div>
112
 
113
  <div className="mb-4 h-48">
@@ -155,8 +118,8 @@ export default function TrainingResults({ data, className }: TrainingResultsProp
155
  </div>
156
 
157
  <p className="mt-4 text-xs text-muted-foreground">
158
- This packaged panel is the demo view for fixed-seed training metrics. When live run artifacts are wired in,
159
- the same layout can render real summary JSON from the training outputs.
160
  </p>
161
  </div>
162
  );
 
20
  Legend,
21
  } from 'recharts';
22
  import type { TrainingComparison } from '@/types';
23
+ import { HOLDOUT_COMPARE, LIVE_CHECKPOINTS, TRAINING_ASSESSMENT } from '@/data/trainingArtifacts';
24
  import { cn, formatReward, formatScore } from '@/lib/utils';
25
 
26
  interface TrainingResultsProps {
 
28
  className?: string;
29
  }
30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  export default function TrainingResults({ data, className }: TrainingResultsProps) {
32
+ const comparison = data ?? HOLDOUT_COMPARE;
33
  const [showTrained, setShowTrained] = useState(true);
34
  const chartData = comparison.baseline.map((baselinePoint, index) => ({
35
  episode: baselinePoint.episode,
 
56
  <span className="rounded-full bg-primary/10 px-2 py-1 text-[11px] font-medium text-primary">Minimal Colab notebook</span>
57
  <span className="rounded-full bg-scientist/10 px-2 py-1 text-[11px] font-medium text-scientist">Unsloth + HF TRL</span>
58
  <span className="rounded-full bg-judge/10 px-2 py-1 text-[11px] font-medium text-judge">Deterministic judge reward</span>
59
+ <span className="rounded-full bg-destructive/10 px-2 py-1 text-[11px] font-medium text-destructive">
60
+ More training required
61
+ </span>
62
  </div>
63
 
64
  <p className="mb-4 text-sm text-muted-foreground">
65
+ This panel now uses the real hold-out compare artifact. The training stack ran successfully, but the current
66
+ trained Scientist still underperforms the deterministic baseline on fixed seeded evaluation.
67
  </p>
68
 
69
  <div className="mb-4 grid gap-2 sm:grid-cols-2 xl:grid-cols-4">
70
  <StoryCard icon={<FileText className="h-3.5 w-3.5" />} title="Notebook" detail="`train_minimal_colab.ipynb` is the sponsor-facing minimal GRPO path." />
71
  <StoryCard icon={<Cpu className="h-3.5 w-3.5" />} title="Trainer" detail="Unsloth + TRL update the Scientist LoRA adapter on seeded tasks." />
72
  <StoryCard icon={<FlaskConical className="h-3.5 w-3.5" />} title="Environment" detail="Episodes come from the same paper-derived ReplicaLab benchmark families." />
73
+ <StoryCard icon={<Scale className="h-3.5 w-3.5" />} title="Reward" detail={`Latest live checkpoint reached ${formatReward(LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1].averageReward)} average reward at ${(LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1].agreementRate * 100).toFixed(0)}% agreement.`} />
74
  </div>
75
 
76
  <div className="mb-4 h-48">
 
118
  </div>
119
 
120
  <p className="mt-4 text-xs text-muted-foreground">
121
+ Current status: {TRAINING_ASSESSMENT.achieved[2]} The next gate is reducing invalid actions and rerunning
122
+ the hold-out compare until trained performance overtakes baseline.
123
  </p>
124
  </div>
125
  );
frontend/src/data/trainingArtifacts.ts ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import type { TrainingComparison } from '@/types';
2
+
3
+ export interface TrainingCheckpoint {
4
+ label: string;
5
+ artifactStep: number;
6
+ averageReward: number;
7
+ agreementRate: number;
8
+ averageRounds: number;
9
+ rolloutsPerGroup: number;
10
+ scenarioCount: number;
11
+ finishedAt: string;
12
+ }
13
+
14
+ export interface TrainingLogRow {
15
+ label: string;
16
+ trainingStep: number;
17
+ seed: number;
18
+ paperTitle: string;
19
+ reward: number;
20
+ roundsUsed: number;
21
+ invalidActionCount: number;
22
+ parseErrorCount: number;
23
+ verdict: string | null;
24
+ note: string;
25
+ }
26
+
27
+ export interface PreviewArtifact {
28
+ runName: string;
29
+ modelName: string;
30
+ datasetSize: number;
31
+ evidencePackVersion: string;
32
+ config: {
33
+ maxSteps?: number;
34
+ numTrainEpochs?: number;
35
+ learningRate: number;
36
+ loraRank: number;
37
+ maxSeqLength: number;
38
+ templates: string[];
39
+ difficulties: string[];
40
+ };
41
+ }
42
+
43
+ export const HOLDOUT_COMPARE: TrainingComparison = {
44
+ baseline: [
45
+ { episode: 1, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
46
+ { episode: 2, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
47
+ { episode: 3, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
48
+ { episode: 4, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
49
+ ],
50
+ trained: [
51
+ { episode: 1, reward: -5.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
52
+ { episode: 2, reward: -4.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
53
+ { episode: 3, reward: -5.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
54
+ { episode: 4, reward: -4.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
55
+ ],
56
+ summary: {
57
+ baseline_avg_reward: 4.925,
58
+ trained_avg_reward: -4.5,
59
+ baseline_agreement_rate: 1.0,
60
+ trained_agreement_rate: 0.0,
61
+ baseline_avg_rounds: 2.0,
62
+ trained_avg_rounds: 20.0,
63
+ baseline_invalid_rate: 0.0,
64
+ trained_invalid_rate: 1.0,
65
+ },
66
+ };
67
+
68
+ export const LIVE_CHECKPOINTS: TrainingCheckpoint[] = [
69
+ {
70
+ label: 'Checkpoint 1',
71
+ artifactStep: 1,
72
+ averageReward: -1.0,
73
+ agreementRate: 0.0,
74
+ averageRounds: 1.75,
75
+ rolloutsPerGroup: 2,
76
+ scenarioCount: 2,
77
+ finishedAt: '2026-03-08T17:55:41.971589+00:00',
78
+ },
79
+ {
80
+ label: 'Checkpoint 2',
81
+ artifactStep: 2,
82
+ averageReward: 0.387202,
83
+ agreementRate: 0.25,
84
+ averageRounds: 3.0,
85
+ rolloutsPerGroup: 2,
86
+ scenarioCount: 2,
87
+ finishedAt: '2026-03-08T17:59:47.820056+00:00',
88
+ },
89
+ {
90
+ label: 'Checkpoint 5',
91
+ artifactStep: 5,
92
+ averageReward: 0.596966,
93
+ agreementRate: 0.305556,
94
+ averageRounds: 3.055556,
95
+ rolloutsPerGroup: 3,
96
+ scenarioCount: 4,
97
+ finishedAt: '2026-03-08T18:12:40.950355+00:00',
98
+ },
99
+ ];
100
+
101
+ export const TRAINING_LOG_ROWS: TrainingLogRow[] = [
102
+ {
103
+ label: 'clean_accept_seed0_step1',
104
+ trainingStep: 1,
105
+ seed: 0,
106
+ paperTitle: 'Reproducing a CIFAR-10 ResNet-18 baseline',
107
+ reward: 4.54881,
108
+ roundsUsed: 3,
109
+ invalidActionCount: 0,
110
+ parseErrorCount: 0,
111
+ verdict: 'accept',
112
+ note: 'The policy produced a valid protocol and reached a clean judged acceptance.',
113
+ },
114
+ {
115
+ label: 'ag_news_invalids_step1',
116
+ trainingStep: 1,
117
+ seed: 1,
118
+ paperTitle: 'Reproducing an AG News TinyBERT baseline',
119
+ reward: -0.25,
120
+ roundsUsed: 4,
121
+ invalidActionCount: 4,
122
+ parseErrorCount: 0,
123
+ verdict: null,
124
+ note: 'The main failure mode was repeated invalid actions on the medium AG News case.',
125
+ },
126
+ {
127
+ label: 'diffusion_parse_failure_step2',
128
+ trainingStep: 2,
129
+ seed: 2,
130
+ paperTitle: 'Reproducing an AG News TinyBERT baseline',
131
+ reward: -1.0,
132
+ roundsUsed: 2,
133
+ invalidActionCount: 0,
134
+ parseErrorCount: 1,
135
+ verdict: null,
136
+ note: 'Parser instability still caused zero-score rollouts after training had already begun.',
137
+ },
138
+ {
139
+ label: 'checkpoint5_best_snapshot',
140
+ trainingStep: 5,
141
+ seed: 0,
142
+ paperTitle: 'replicalab-scientist-art-live:step5',
143
+ reward: 0.596966,
144
+ roundsUsed: 3,
145
+ invalidActionCount: 0,
146
+ parseErrorCount: 0,
147
+ verdict: 'mixed',
148
+ note: 'By checkpoint 5 the live run had moved into positive average reward, but not enough to beat baseline on hold-out compare.',
149
+ },
150
+ ];
151
+
152
+ export const SCIENTIST_PREVIEW_ARTIFACT: PreviewArtifact = {
153
+ runName: 'scientist-preview-smoke-20260308b',
154
+ modelName: 'Qwen/Qwen3-8B',
155
+ datasetSize: 18,
156
+ evidencePackVersion: '6a0802447dc4',
157
+ config: {
158
+ maxSteps: 12,
159
+ learningRate: 5e-6,
160
+ loraRank: 32,
161
+ maxSeqLength: 4096,
162
+ templates: ['math_reasoning', 'ml_benchmark', 'finance_trading'],
163
+ difficulties: ['easy', 'medium', 'hard'],
164
+ },
165
+ };
166
+
167
+ export const LAB_MANAGER_PREVIEW_ARTIFACT: PreviewArtifact = {
168
+ runName: 'lab-manager-preview-smoke-20260308b',
169
+ modelName: 'Qwen/Qwen3-8B',
170
+ datasetSize: 54,
171
+ evidencePackVersion: '6a0802447dc4',
172
+ config: {
173
+ numTrainEpochs: 1.0,
174
+ learningRate: 2e-5,
175
+ loraRank: 32,
176
+ maxSeqLength: 3072,
177
+ templates: ['math_reasoning', 'ml_benchmark', 'finance_trading'],
178
+ difficulties: ['easy', 'medium', 'hard'],
179
+ },
180
+ };
181
+
182
+ export const LOCAL_BASELINE_SUMMARY = {
183
+ averageReward: 4.600926,
184
+ agreementRate: 1.0,
185
+ averageRounds: 2.0,
186
+ averageRigor: 0.805556,
187
+ averageFeasibility: 1.0,
188
+ averageFidelity: 0.438889,
189
+ episodeCount: 3,
190
+ };
191
+
192
+ export const TRAINING_ASSESSMENT = {
193
+ needsMoreTraining: true,
194
+ achieved: [
195
+ 'The minimal Colab path and the reusable training modules are both in place.',
196
+ 'Scientist preview data and Lab Manager preview data were generated successfully on the frozen evidence packs.',
197
+ 'The live ART/OpenEnv Scientist run reached positive average reward by checkpoint 5.',
198
+ ],
199
+ gaps: [
200
+ 'Hold-out compare still strongly favors the deterministic baseline over the trained scientist policy.',
201
+ 'Invalid-action rate on the hold-out compare is still 1.0 for the trained policy.',
202
+ 'Lab Manager has preview artifacts but does not yet have a live trained-and-evaluated adapter in the demo.',
203
+ ],
204
+ improvements: [
205
+ 'Reduce invalid JSON and invalid action rate before extending training length.',
206
+ 'Run more train steps on the same frozen evidence version and compare on fixed held-out seeds after each checkpoint.',
207
+ 'Add curriculum or parser-focused reward shaping for the medium ML benchmark cases.',
208
+ 'Finish the Lab Manager SFT run and evaluate Scientist-plus-Lab-Manager together instead of only Scientist RL.',
209
+ ],
210
+ };
frontend/src/lib/api.ts CHANGED
@@ -15,6 +15,7 @@ import type {
15
  BackendScenarioFamily,
16
  ScenarioTemplate,
17
  Difficulty,
 
18
  } from '@/types';
19
 
20
  const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? '/api';
@@ -121,6 +122,34 @@ function observationToEpisodeState(
121
  conversation: adaptConversation(sci.conversation_history),
122
  scores: null,
123
  judge_audit: null,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
  };
125
  }
126
 
@@ -222,9 +251,11 @@ export async function stepEpisode(
222
  : prevState.conversation;
223
  const protocol = obs?.scientist?.current_protocol ?? prevState.protocol;
224
  const round = obs?.scientist?.round_number ?? prevState.round + 1;
 
225
 
226
  // Update lab constraints if available
227
  const labConstraints = obs ? adaptLabConstraints(obs) : prevState.lab_constraints;
 
228
 
229
  return {
230
  ...prevState,
@@ -235,6 +266,8 @@ export async function stepEpisode(
235
  lab_constraints: labConstraints,
236
  scores,
237
  judge_audit: judgeAudit,
 
 
238
  };
239
  }
240
 
@@ -297,41 +330,61 @@ export function sendWsMessage(ws: WebSocket, msg: WebSocketMessage) {
297
  // ---------------------------------------------------------------------------
298
 
299
  export function buildDefaultScientistAction(state?: EpisodeState): ScientistAction {
300
- if (state?.protocol) {
301
- return buildAcceptAction();
302
- }
303
-
304
  const durationLimit = Math.max(1, state?.lab_constraints.time_limit_days ?? 3);
305
- const originalDuration = state?.paper.original_duration_days ?? 0;
306
- const durationDays = Math.max(1, Math.min(durationLimit, originalDuration || durationLimit));
307
  const template = state?.template;
 
 
 
 
 
 
 
 
308
 
309
  const technique =
310
- state?.paper.original_technique && state.paper.original_technique !== 'N/A'
311
- ? state.paper.original_technique
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
312
  : template === 'math_reasoning'
313
- ? 'structured_proof_check'
314
- : template === 'finance_trading'
315
- ? 'offline_backtest'
316
- : 'published_training_recipe';
317
-
318
- const controls = state?.paper.original_controls.length
319
- ? state.paper.original_controls
320
- : ['baseline'];
321
 
322
- const requiredEquipment = state?.lab_constraints.equipment_available.slice(0, 1) ?? [];
323
- const requiredReagents = state?.lab_constraints.reagents_available.slice(0, 1) ?? [];
 
 
 
 
324
 
325
  return {
326
- action_type: 'propose_protocol',
327
- sample_size: 3,
328
  controls,
329
  technique,
330
  duration_days: durationDays,
331
  required_equipment: requiredEquipment,
332
  required_reagents: requiredReagents,
333
  questions: [],
334
- rationale: `Replicate the source result within the available lab window of ${durationLimit} days using currently available resources.`,
 
 
335
  };
336
  }
337
 
 
15
  BackendScenarioFamily,
16
  ScenarioTemplate,
17
  Difficulty,
18
+ EpisodeStepTrace,
19
  } from '@/types';
20
 
21
  const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? '/api';
 
122
  conversation: adaptConversation(sci.conversation_history),
123
  scores: null,
124
  judge_audit: null,
125
+ cumulative_reward: 0,
126
+ step_history: [],
127
+ };
128
+ }
129
+
130
+ function buildRoundTrace(
131
+ prevState: EpisodeState,
132
+ data: BackendStepResult,
133
+ ): EpisodeStepTrace {
134
+ const round = data.info.round;
135
+ const rawHistory = data.observation?.scientist?.conversation_history ?? [];
136
+ const roundEntries = rawHistory.filter((entry) => entry.round_number === round);
137
+ const scientistEntry = roundEntries.find((entry) => entry.role === 'scientist');
138
+ const labManagerEntry = roundEntries.find((entry) => entry.role === 'lab_manager');
139
+
140
+ return {
141
+ round,
142
+ reward: data.reward,
143
+ cumulative_reward: data.info.cumulative_reward ?? prevState.cumulative_reward + data.reward,
144
+ action_type: scientistEntry?.action_type ?? 'unknown',
145
+ scientist_message: scientistEntry?.message ?? '',
146
+ lab_manager_action_type: labManagerEntry?.action_type ?? undefined,
147
+ lab_manager_message: labManagerEntry?.message ?? undefined,
148
+ step_reward_components: data.info.step_reward_components ?? {},
149
+ protocol: data.observation?.scientist?.current_protocol ?? prevState.protocol,
150
+ oracle_round_score: data.info.oracle_round_score ?? null,
151
+ oracle_post_mortem: data.info.oracle_post_mortem ?? null,
152
+ oracle_event: data.info.oracle_event ?? null,
153
  };
154
  }
155
 
 
251
  : prevState.conversation;
252
  const protocol = obs?.scientist?.current_protocol ?? prevState.protocol;
253
  const round = obs?.scientist?.round_number ?? prevState.round + 1;
254
+ const cumulativeReward = data.info.cumulative_reward ?? prevState.cumulative_reward + data.reward;
255
 
256
  // Update lab constraints if available
257
  const labConstraints = obs ? adaptLabConstraints(obs) : prevState.lab_constraints;
258
+ const roundTrace = buildRoundTrace(prevState, data);
259
 
260
  return {
261
  ...prevState,
 
266
  lab_constraints: labConstraints,
267
  scores,
268
  judge_audit: judgeAudit,
269
+ cumulative_reward: cumulativeReward,
270
+ step_history: [...prevState.step_history, roundTrace],
271
  };
272
  }
273
 
 
330
  // ---------------------------------------------------------------------------
331
 
332
  export function buildDefaultScientistAction(state?: EpisodeState): ScientistAction {
 
 
 
 
333
  const durationLimit = Math.max(1, state?.lab_constraints.time_limit_days ?? 3);
 
 
334
  const template = state?.template;
335
+ const currentProtocol = state?.protocol;
336
+
337
+ const originalDuration = state?.paper.original_duration_days ?? 0;
338
+ const preferredDuration = currentProtocol?.duration_days ?? (originalDuration || durationLimit);
339
+ const durationDays = Math.max(
340
+ 1,
341
+ Math.min(durationLimit, preferredDuration),
342
+ );
343
 
344
  const technique =
345
+ currentProtocol?.technique
346
+ ?? (state?.paper.original_technique && state.paper.original_technique !== 'N/A'
347
+ ? state.paper.original_technique
348
+ : template === 'math_reasoning'
349
+ ? 'structured_proof_check'
350
+ : template === 'finance_trading'
351
+ ? 'offline_backtest'
352
+ : 'published_training_recipe');
353
+
354
+ const controls =
355
+ currentProtocol?.controls.length
356
+ ? currentProtocol.controls
357
+ : state?.paper.original_controls.length
358
+ ? state.paper.original_controls
359
+ : ['baseline'];
360
+
361
+ const baseSampleSize = currentProtocol?.sample_size ?? 3;
362
+ const sampleSize =
363
+ state?.round && state.round > 0
364
+ ? Math.max(3, Math.min(baseSampleSize + (state.round % 2 === 0 ? 1 : -1), 12))
365
  : template === 'math_reasoning'
366
+ ? 4
367
+ : 3;
 
 
 
 
 
 
368
 
369
+ const requiredEquipment = currentProtocol?.required_equipment.length
370
+ ? currentProtocol.required_equipment
371
+ : state?.lab_constraints.equipment_available.slice(0, 1) ?? [];
372
+ const requiredReagents = currentProtocol?.required_reagents.length
373
+ ? currentProtocol.required_reagents
374
+ : state?.lab_constraints.reagents_available.slice(0, 1) ?? [];
375
 
376
  return {
377
+ action_type: currentProtocol ? 'revise_protocol' : 'propose_protocol',
378
+ sample_size: sampleSize,
379
  controls,
380
  technique,
381
  duration_days: durationDays,
382
  required_equipment: requiredEquipment,
383
  required_reagents: requiredReagents,
384
  questions: [],
385
+ rationale: currentProtocol
386
+ ? `Refine the existing protocol for round ${state?.round ?? 0} while staying inside the ${durationLimit}-day lab window.`
387
+ : `Replicate the source result within the available lab window of ${durationLimit} days using currently available resources.`,
388
  };
389
  }
390
 
frontend/src/lib/demo.ts ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import type { DemoCase, EpisodeState, ScientistAction } from '@/types';
2
+ import { buildAcceptAction, buildDefaultScientistAction } from '@/lib/api';
3
+
4
+ export interface DemoCaseMeta {
5
+ id: DemoCase;
6
+ title: string;
7
+ subtitle: string;
8
+ summary: string;
9
+ seed: number;
10
+ }
11
+
12
+ export const DEMO_CASES: DemoCaseMeta[] = [
13
+ {
14
+ id: 'fast-agreement',
15
+ title: 'Scenario 1: First-Round Agreement',
16
+ subtitle: 'Good paper, fast agreement',
17
+ summary: 'A strong paper-derived benchmark where the first protocol is feasible and the agents converge immediately.',
18
+ seed: 101,
19
+ },
20
+ {
21
+ id: 'learning-opportunity',
22
+ title: 'Scenario 2: Multi-Round Learning',
23
+ subtitle: 'Heavy RL learning opportunity',
24
+ summary: 'The Scientist starts overly ambitious, absorbs constraint feedback across several rounds, then lands on a feasible protocol.',
25
+ seed: 202,
26
+ },
27
+ {
28
+ id: 'no-agreement',
29
+ title: 'Scenario 3: No Agreement',
30
+ subtitle: 'Reject, not replicable under current setup',
31
+ summary: 'The protocol never becomes acceptable within the six-round budget, producing a clean rejection case for the demo.',
32
+ seed: 303,
33
+ },
34
+ ];
35
+
36
+ export function parseDemoCase(value: string | null): DemoCase | undefined {
37
+ return DEMO_CASES.find((item) => item.id === value)?.id;
38
+ }
39
+
40
+ function buildScenarioAwareProposal(
41
+ state: EpisodeState,
42
+ overrides: Partial<ScientistAction>,
43
+ ): ScientistAction {
44
+ const baseAction = buildDefaultScientistAction(state);
45
+ const actionType = state.protocol ? 'revise_protocol' : 'propose_protocol';
46
+
47
+ return {
48
+ ...baseAction,
49
+ action_type: actionType,
50
+ ...overrides,
51
+ };
52
+ }
53
+
54
+ export function buildDemoScientistAction(
55
+ state: EpisodeState,
56
+ demoCase?: DemoCase,
57
+ ): ScientistAction {
58
+ if (!demoCase) {
59
+ return state.round >= state.max_rounds - 1
60
+ ? buildAcceptAction()
61
+ : buildDefaultScientistAction(state);
62
+ }
63
+
64
+ const defaultEquipment = state.lab_constraints.equipment_available.slice(0, 1);
65
+ const defaultReagents = state.lab_constraints.reagents_available.slice(0, 1);
66
+ const defaultTechnique =
67
+ state.protocol?.technique ||
68
+ (state.paper.original_technique !== 'N/A'
69
+ ? state.paper.original_technique
70
+ : 'published_training_recipe');
71
+
72
+ if (demoCase === 'fast-agreement') {
73
+ if (state.round >= 1) {
74
+ return buildAcceptAction();
75
+ }
76
+ return buildScenarioAwareProposal(state, {
77
+ sample_size: 3,
78
+ controls: ['baseline'],
79
+ technique: defaultTechnique,
80
+ duration_days: Math.min(4, state.lab_constraints.time_limit_days),
81
+ required_equipment: defaultEquipment,
82
+ required_reagents: defaultReagents,
83
+ rationale: 'Reproduce the published baseline exactly once with the available reviewer pass and hardware.',
84
+ });
85
+ }
86
+
87
+ if (demoCase === 'learning-opportunity') {
88
+ const learningPlan: Record<number, Partial<ScientistAction>> = {
89
+ 0: {
90
+ sample_size: 96,
91
+ controls: ['baseline', 'ablation', 'sanity_check'],
92
+ rationale: 'Start with an expansive benchmark reproduction to maximize coverage before trimming.',
93
+ },
94
+ 1: {
95
+ sample_size: 72,
96
+ controls: ['baseline', 'ablation', 'sanity_check'],
97
+ rationale: 'Reduce the experimental load, but keep the extra controls while the constraints are still being learned.',
98
+ },
99
+ 2: {
100
+ sample_size: 48,
101
+ controls: ['baseline', 'ablation', 'sanity_check'],
102
+ rationale: 'Continue lowering the sample size while preserving the full validation set of controls.',
103
+ },
104
+ 3: {
105
+ sample_size: 32,
106
+ controls: ['baseline', 'ablation', 'sanity_check'],
107
+ rationale: 'The Scientist now converges toward feasibility by balancing budget with stronger controls.',
108
+ },
109
+ 4: {
110
+ sample_size: 24,
111
+ controls: ['baseline', 'ablation'],
112
+ rationale: 'Final revision: keep the baseline plus one ablation and fit the protocol cleanly inside the lab capacity.',
113
+ },
114
+ };
115
+
116
+ if (state.round >= 5) {
117
+ return buildAcceptAction();
118
+ }
119
+ return buildScenarioAwareProposal(state, {
120
+ technique: defaultTechnique,
121
+ duration_days: Math.min(4, state.lab_constraints.time_limit_days),
122
+ required_equipment: defaultEquipment,
123
+ required_reagents: defaultReagents,
124
+ ...learningPlan[state.round],
125
+ });
126
+ }
127
+
128
+ const rejectionPlan: Record<number, Partial<ScientistAction>> = {
129
+ 0: {
130
+ sample_size: 96,
131
+ controls: ['baseline', 'ablation', 'sanity_check'],
132
+ rationale: 'Push for the full benchmark replication despite the current resource limits.',
133
+ },
134
+ 1: {
135
+ sample_size: 88,
136
+ controls: ['baseline', 'ablation', 'sanity_check'],
137
+ rationale: 'Trim slightly, but keep the high-coverage plan intact.',
138
+ },
139
+ 2: {
140
+ sample_size: 80,
141
+ controls: ['baseline', 'ablation', 'sanity_check'],
142
+ rationale: 'Still prioritize coverage over the budget warnings.',
143
+ },
144
+ 3: {
145
+ sample_size: 76,
146
+ controls: ['baseline', 'ablation', 'sanity_check'],
147
+ rationale: 'The Scientist refuses to reduce scope enough to become feasible.',
148
+ },
149
+ 4: {
150
+ sample_size: 72,
151
+ controls: ['baseline', 'ablation', 'sanity_check'],
152
+ rationale: 'The protocol remains too expensive and staff-heavy for this lab.',
153
+ },
154
+ 5: {
155
+ sample_size: 68,
156
+ controls: ['baseline', 'ablation', 'sanity_check'],
157
+ rationale: 'Final round: still no viable path to agreement under the current constraints.',
158
+ },
159
+ };
160
+
161
+ return buildScenarioAwareProposal(state, {
162
+ technique: defaultTechnique,
163
+ duration_days: Math.min(4, state.lab_constraints.time_limit_days),
164
+ required_equipment: defaultEquipment,
165
+ required_reagents: defaultReagents,
166
+ ...(rejectionPlan[state.round] ?? rejectionPlan[5]),
167
+ });
168
+ }
frontend/src/pages/ComparePage.tsx CHANGED
@@ -17,7 +17,8 @@ export default function ComparePage() {
17
  <h1 className="mb-2 text-3xl font-bold tracking-tight">Seeded Benchmark Comparison</h1>
18
  <p className="mx-auto max-w-lg text-muted-foreground">
19
  Use the same frontend to replay fixed benchmark cases across domains or difficulty levels.
20
- This page is the evaluation bench for demo runs, while the training panel holds the baseline-vs-trained story.
 
21
  </p>
22
  </motion.div>
23
 
 
17
  <h1 className="mb-2 text-3xl font-bold tracking-tight">Seeded Benchmark Comparison</h1>
18
  <p className="mx-auto max-w-lg text-muted-foreground">
19
  Use the same frontend to replay fixed benchmark cases across domains or difficulty levels.
20
+ This page is the evaluation bench for demo runs, while the dedicated Training page holds the artifact-backed
21
+ training logs and improvement analysis.
22
  </p>
23
  </motion.div>
24
 
frontend/src/pages/DashboardPage.tsx CHANGED
@@ -10,12 +10,16 @@ import {
10
  MessageSquare,
11
  Scale,
12
  TrendingUp,
 
 
 
13
  } from 'lucide-react';
14
  import TrainingResults from '@/components/TrainingResults';
15
  import AnimatedCharacter from '@/components/AnimatedCharacter';
16
  import MoleculeScene from '@/components/MoleculeScene';
17
  import TiltCard from '@/components/TiltCard';
18
  import { cn } from '@/lib/utils';
 
19
 
20
  const SCENARIOS = [
21
  {
@@ -50,7 +54,7 @@ const SCENARIOS = [
50
  const ROLES = [
51
  {
52
  key: 'scientist',
53
- name: 'Dr. Elara The Scientist',
54
  tagline: 'Protects the Science',
55
  description: 'A brilliant researcher who proposes, revises, and asks questions to build the best replication plan without compromising scientific validity.',
56
  color: 'text-scientist',
@@ -59,7 +63,7 @@ const ROLES = [
59
  },
60
  {
61
  key: 'lab_manager',
62
- name: 'Takuma The Lab Manager',
63
  tagline: 'Guards Feasibility',
64
  description: 'A pragmatic organizer who reports budget, equipment, scheduling, and staffing constraints. He ensures plans are actually executable.',
65
  color: 'text-lab-manager',
@@ -68,7 +72,7 @@ const ROLES = [
68
  },
69
  {
70
  key: 'judge',
71
- name: 'Aldric The Judge',
72
  tagline: 'Delivers the Verdict',
73
  description: 'An impartial arbiter who scores the final protocol on rigor, feasibility, and fidelity using a deterministic rubric engine.',
74
  color: 'text-judge',
@@ -176,19 +180,19 @@ export default function DashboardPage() {
176
 
177
  <div className="mt-8 flex items-center justify-center gap-3">
178
  <Link
179
- to="/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1"
180
  className="inline-flex items-center gap-2 rounded-lg bg-primary px-5 py-2.5 text-sm font-medium text-primary-foreground transition-colors hover:bg-primary/90 shadow-lg shadow-primary/25"
181
  >
182
  <Play className="h-4 w-4" />
183
  Replicate a Paper
184
  </Link>
185
- <a
186
- href="#training"
187
  className="inline-flex items-center gap-2 rounded-lg border border-border bg-background/80 backdrop-blur-sm px-5 py-2.5 text-sm font-medium text-foreground transition-colors hover:bg-muted"
188
  >
189
  <BarChart3 className="h-4 w-4" />
190
- See Training Flow
191
- </a>
192
  </div>
193
 
194
  <div className="mx-auto mt-8 grid max-w-4xl gap-3 text-left md:grid-cols-4">
@@ -296,6 +300,41 @@ export default function DashboardPage() {
296
  </div>
297
  </section>
298
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
299
  {/* Training Results */}
300
  <section id="training" className="mb-12">
301
  <h2 className="mb-2 text-center text-xl font-semibold">Training Results</h2>
@@ -307,10 +346,10 @@ export default function DashboardPage() {
307
  </div>
308
  <div className="mt-4 text-center">
309
  <Link
310
- to="/compare"
311
  className="inline-flex items-center gap-2 text-sm font-medium text-primary transition-colors hover:text-primary/80"
312
  >
313
- Open the seeded evaluation bench
314
  <ArrowRight className="h-4 w-4" />
315
  </Link>
316
  </div>
 
10
  MessageSquare,
11
  Scale,
12
  TrendingUp,
13
+ CheckCircle2,
14
+ AlertTriangle,
15
+ XCircle,
16
  } from 'lucide-react';
17
  import TrainingResults from '@/components/TrainingResults';
18
  import AnimatedCharacter from '@/components/AnimatedCharacter';
19
  import MoleculeScene from '@/components/MoleculeScene';
20
  import TiltCard from '@/components/TiltCard';
21
  import { cn } from '@/lib/utils';
22
+ import { DEMO_CASES } from '@/lib/demo';
23
 
24
  const SCENARIOS = [
25
  {
 
54
  const ROLES = [
55
  {
56
  key: 'scientist',
57
+ name: 'Dr. Elara - The Scientist',
58
  tagline: 'Protects the Science',
59
  description: 'A brilliant researcher who proposes, revises, and asks questions to build the best replication plan without compromising scientific validity.',
60
  color: 'text-scientist',
 
63
  },
64
  {
65
  key: 'lab_manager',
66
+ name: 'Takuma - The Lab Manager',
67
  tagline: 'Guards Feasibility',
68
  description: 'A pragmatic organizer who reports budget, equipment, scheduling, and staffing constraints. He ensures plans are actually executable.',
69
  color: 'text-lab-manager',
 
72
  },
73
  {
74
  key: 'judge',
75
+ name: 'Aldric - The Judge',
76
  tagline: 'Delivers the Verdict',
77
  description: 'An impartial arbiter who scores the final protocol on rigor, feasibility, and fidelity using a deterministic rubric engine.',
78
  color: 'text-judge',
 
180
 
181
  <div className="mt-8 flex items-center justify-center gap-3">
182
  <Link
183
+ to="/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1&demoCase=fast-agreement"
184
  className="inline-flex items-center gap-2 rounded-lg bg-primary px-5 py-2.5 text-sm font-medium text-primary-foreground transition-colors hover:bg-primary/90 shadow-lg shadow-primary/25"
185
  >
186
  <Play className="h-4 w-4" />
187
  Replicate a Paper
188
  </Link>
189
+ <Link
190
+ to="/training"
191
  className="inline-flex items-center gap-2 rounded-lg border border-border bg-background/80 backdrop-blur-sm px-5 py-2.5 text-sm font-medium text-foreground transition-colors hover:bg-muted"
192
  >
193
  <BarChart3 className="h-4 w-4" />
194
+ See Training Logs
195
+ </Link>
196
  </div>
197
 
198
  <div className="mx-auto mt-8 grid max-w-4xl gap-3 text-left md:grid-cols-4">
 
300
  </div>
301
  </section>
302
 
303
+ <section className="mb-16">
304
+ <h2 className="mb-2 text-center text-xl font-semibold">Live Demo Outcomes</h2>
305
+ <p className="mb-8 text-center text-sm text-muted-foreground">
306
+ Pick the exact story you want to show: immediate agreement, multi-round learning, or clear rejection.
307
+ </p>
308
+ <div className="grid gap-4 md:grid-cols-3">
309
+ {DEMO_CASES.map((demo) => {
310
+ const Icon =
311
+ demo.id === 'fast-agreement'
312
+ ? CheckCircle2
313
+ : demo.id === 'learning-opportunity'
314
+ ? AlertTriangle
315
+ : XCircle;
316
+
317
+ return (
318
+ <Link
319
+ key={demo.id}
320
+ to={`/episode?template=ml_benchmark&difficulty=medium&seed=${demo.seed}&demo=1&autoplay=1&demoCase=${demo.id}`}
321
+ className="rounded-xl border border-border bg-card p-5 transition-colors hover:border-primary/40 hover:bg-muted/40"
322
+ >
323
+ <div className="mb-3 inline-flex rounded-full bg-primary/10 p-2 text-primary">
324
+ <Icon className="h-4 w-4" />
325
+ </div>
326
+ <h3 className="text-sm font-semibold">{demo.title}</h3>
327
+ <p className="mt-1 text-xs font-medium text-primary">{demo.subtitle}</p>
328
+ <p className="mt-3 text-sm text-muted-foreground">{demo.summary}</p>
329
+ <div className="mt-4 inline-flex items-center gap-1 text-xs font-medium text-primary">
330
+ Launch demo <ArrowRight className="h-3 w-3" />
331
+ </div>
332
+ </Link>
333
+ );
334
+ })}
335
+ </div>
336
+ </section>
337
+
338
  {/* Training Results */}
339
  <section id="training" className="mb-12">
340
  <h2 className="mb-2 text-center text-xl font-semibold">Training Results</h2>
 
346
  </div>
347
  <div className="mt-4 text-center">
348
  <Link
349
+ to="/training"
350
  className="inline-flex items-center gap-2 text-sm font-medium text-primary transition-colors hover:text-primary/80"
351
  >
352
+ Open the detailed training page
353
  <ArrowRight className="h-4 w-4" />
354
  </Link>
355
  </div>
frontend/src/pages/EpisodePage.tsx CHANGED
@@ -3,9 +3,10 @@ import { useNavigate, useSearchParams } from 'react-router-dom';
3
  import { motion, AnimatePresence } from 'framer-motion';
4
  import { FileText, FlaskConical, Scale, TrendingUp, Play, Pencil } from 'lucide-react';
5
  import type { EpisodeState, ResetParams, ScientistAction } from '@/types';
6
- import { resetEpisode, stepEpisode, buildDefaultScientistAction, buildAcceptAction } from '@/lib/api';
7
  import { sfx, startAmbient, stopAmbient } from '@/lib/audio';
8
  import { fireSuccessConfetti, fireGavelConfetti } from '@/lib/confetti';
 
9
  import { useToast } from '@/components/Toast';
10
  import { useKeyboardShortcuts, ShortcutsOverlay, ShortcutHint } from '@/components/KeyboardShortcuts';
11
  import AutoPlayControls, { useAutoPlay } from '@/components/AutoPlayControls';
@@ -24,6 +25,7 @@ import ProtocolTimeline from '@/components/ProtocolTimeline';
24
  import AgentThoughts from '@/components/AgentThoughts';
25
  import ProtocolEditor from '@/components/ProtocolEditor';
26
  import LabScene3D from '@/components/LabScene3D';
 
27
 
28
  const TEMPLATE_OPTIONS = ['math_reasoning', 'ml_benchmark', 'finance_trading'] as const;
29
  const DIFFICULTY_OPTIONS = ['easy', 'medium', 'hard'] as const;
@@ -66,6 +68,11 @@ export default function EpisodePage() {
66
  () => searchParams.get('demo') === '1' || searchParams.get('autostart') === '1',
67
  [searchParams],
68
  );
 
 
 
 
 
69
  const autoPlayRequested = useMemo(
70
  () => demoRequested || searchParams.get('autoplay') === '1',
71
  [demoRequested, searchParams],
@@ -86,6 +93,7 @@ export default function EpisodePage() {
86
  // Sound effects for phase transitions
87
  const prevPhaseRef = useRef(phase);
88
  const prevMsgCountRef = useRef(0);
 
89
 
90
  useEffect(() => {
91
  const prev = prevPhaseRef.current;
@@ -109,10 +117,10 @@ export default function EpisodePage() {
109
  sfx.success();
110
  fireSuccessConfetti();
111
  }, 400);
112
- toast('Episode complete Agreement reached!', 'success');
113
  } else if (verdict) {
114
  setTimeout(() => sfx.failure(), 400);
115
- toast(`Episode complete Verdict: ${verdict}`, 'warning');
116
  }
117
  }
118
  }, [phase, episode?.judge_audit?.verdict, toast]);
@@ -134,6 +142,12 @@ export default function EpisodePage() {
134
  return () => stopAmbient();
135
  }, []);
136
 
 
 
 
 
 
 
137
  const handleStart = useCallback(async (params: ResetParams) => {
138
  setLoading(true);
139
  setIsJudging(false);
@@ -142,7 +156,8 @@ export default function EpisodePage() {
142
  sfx.click();
143
  try {
144
  const state = await resetEpisode(params);
145
- setEpisode(state);
 
146
  prevMsgCountRef.current = 0;
147
  toast('Episode started!', 'info');
148
  // Feature 9: Update URL for shareable link
@@ -156,6 +171,9 @@ export default function EpisodePage() {
156
  if (autoPlayRequested) {
157
  nextSearch.set('autoplay', '1');
158
  }
 
 
 
159
  navigate(`/episode/${state.episode_id}?${nextSearch.toString()}`, { replace: true });
160
  } catch (err) {
161
  console.error('Failed to start episode:', err);
@@ -165,7 +183,7 @@ export default function EpisodePage() {
165
  } finally {
166
  setLoading(false);
167
  }
168
- }, [autoPlayRequested, demoRequested, navigate, toast]);
169
 
170
  const handleStepWithAction = useCallback(async (action?: ScientistAction) => {
171
  if (!episode || episode.done) return;
@@ -178,7 +196,7 @@ export default function EpisodePage() {
178
 
179
  try {
180
  const isLastRound = episode.round >= episode.max_rounds - 1;
181
- const finalAction = action ?? (isLastRound ? buildAcceptAction() : buildDefaultScientistAction(episode));
182
 
183
  if (isLastRound && !action) {
184
  setIsJudging(true);
@@ -205,7 +223,7 @@ export default function EpisodePage() {
205
  } finally {
206
  setLoading(false);
207
  }
208
- }, [episode, toast]);
209
 
210
  useEffect(() => {
211
  if (!demoRequested || autoStartTriggered || episode || loading) {
@@ -395,6 +413,28 @@ export default function EpisodePage() {
395
  ))}
396
  </motion.div>
397
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
398
  {!episode.done && episode.conversation.length === 0 && (
399
  <motion.div
400
  className="mb-4 rounded-lg border border-primary/20 bg-primary/5 p-4"
@@ -578,6 +618,18 @@ export default function EpisodePage() {
578
  {episode.done && <ReplayViewer messages={episode.conversation} />}
579
  </motion.div>
580
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
581
  </div>
582
  );
583
  }
 
3
  import { motion, AnimatePresence } from 'framer-motion';
4
  import { FileText, FlaskConical, Scale, TrendingUp, Play, Pencil } from 'lucide-react';
5
  import type { EpisodeState, ResetParams, ScientistAction } from '@/types';
6
+ import { resetEpisode, stepEpisode } from '@/lib/api';
7
  import { sfx, startAmbient, stopAmbient } from '@/lib/audio';
8
  import { fireSuccessConfetti, fireGavelConfetti } from '@/lib/confetti';
9
+ import { buildDemoScientistAction, DEMO_CASES, parseDemoCase } from '@/lib/demo';
10
  import { useToast } from '@/components/Toast';
11
  import { useKeyboardShortcuts, ShortcutsOverlay, ShortcutHint } from '@/components/KeyboardShortcuts';
12
  import AutoPlayControls, { useAutoPlay } from '@/components/AutoPlayControls';
 
25
  import AgentThoughts from '@/components/AgentThoughts';
26
  import ProtocolEditor from '@/components/ProtocolEditor';
27
  import LabScene3D from '@/components/LabScene3D';
28
+ import EpisodeResultsReport from '@/components/EpisodeResultsReport';
29
 
30
  const TEMPLATE_OPTIONS = ['math_reasoning', 'ml_benchmark', 'finance_trading'] as const;
31
  const DIFFICULTY_OPTIONS = ['easy', 'medium', 'hard'] as const;
 
68
  () => searchParams.get('demo') === '1' || searchParams.get('autostart') === '1',
69
  [searchParams],
70
  );
71
+ const demoCase = useMemo(() => parseDemoCase(searchParams.get('demoCase')), [searchParams]);
72
+ const demoMeta = useMemo(
73
+ () => DEMO_CASES.find((item) => item.id === demoCase),
74
+ [demoCase],
75
+ );
76
  const autoPlayRequested = useMemo(
77
  () => demoRequested || searchParams.get('autoplay') === '1',
78
  [demoRequested, searchParams],
 
93
  // Sound effects for phase transitions
94
  const prevPhaseRef = useRef(phase);
95
  const prevMsgCountRef = useRef(0);
96
+ const resultsRef = useRef<HTMLDivElement | null>(null);
97
 
98
  useEffect(() => {
99
  const prev = prevPhaseRef.current;
 
117
  sfx.success();
118
  fireSuccessConfetti();
119
  }, 400);
120
+ toast('Episode complete - Agreement reached!', 'success');
121
  } else if (verdict) {
122
  setTimeout(() => sfx.failure(), 400);
123
+ toast(`Episode complete - Verdict: ${verdict}`, 'warning');
124
  }
125
  }
126
  }, [phase, episode?.judge_audit?.verdict, toast]);
 
142
  return () => stopAmbient();
143
  }, []);
144
 
145
+ useEffect(() => {
146
+ if (episode?.done) {
147
+ resultsRef.current?.scrollIntoView({ behavior: 'smooth', block: 'start' });
148
+ }
149
+ }, [episode?.done]);
150
+
151
  const handleStart = useCallback(async (params: ResetParams) => {
152
  setLoading(true);
153
  setIsJudging(false);
 
156
  sfx.click();
157
  try {
158
  const state = await resetEpisode(params);
159
+ const nextState = demoCase ? { ...state, demo_case: demoCase } : state;
160
+ setEpisode(nextState);
161
  prevMsgCountRef.current = 0;
162
  toast('Episode started!', 'info');
163
  // Feature 9: Update URL for shareable link
 
171
  if (autoPlayRequested) {
172
  nextSearch.set('autoplay', '1');
173
  }
174
+ if (demoCase) {
175
+ nextSearch.set('demoCase', demoCase);
176
+ }
177
  navigate(`/episode/${state.episode_id}?${nextSearch.toString()}`, { replace: true });
178
  } catch (err) {
179
  console.error('Failed to start episode:', err);
 
183
  } finally {
184
  setLoading(false);
185
  }
186
+ }, [autoPlayRequested, demoCase, demoRequested, navigate, toast]);
187
 
188
  const handleStepWithAction = useCallback(async (action?: ScientistAction) => {
189
  if (!episode || episode.done) return;
 
196
 
197
  try {
198
  const isLastRound = episode.round >= episode.max_rounds - 1;
199
+ const finalAction = action ?? buildDemoScientistAction(episode, demoCase);
200
 
201
  if (isLastRound && !action) {
202
  setIsJudging(true);
 
223
  } finally {
224
  setLoading(false);
225
  }
226
+ }, [demoCase, episode, toast]);
227
 
228
  useEffect(() => {
229
  if (!demoRequested || autoStartTriggered || episode || loading) {
 
413
  ))}
414
  </motion.div>
415
 
416
+ {demoMeta && (
417
+ <motion.div
418
+ className="mb-4 rounded-xl border border-primary/20 bg-primary/5 p-4"
419
+ initial={{ opacity: 0, y: 8 }}
420
+ animate={{ opacity: 1, y: 0 }}
421
+ transition={{ delay: 0.14 }}
422
+ >
423
+ <div className="flex flex-col gap-2 md:flex-row md:items-center md:justify-between">
424
+ <div>
425
+ <div className="text-xs font-semibold uppercase tracking-[0.18em] text-primary">
426
+ {demoMeta.title}
427
+ </div>
428
+ <h2 className="mt-1 text-base font-semibold">{demoMeta.subtitle}</h2>
429
+ <p className="mt-1 text-sm text-muted-foreground">{demoMeta.summary}</p>
430
+ </div>
431
+ <div className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
432
+ Seed {episode.seed} · {episode.template.replace(/_/g, ' ')} · {episode.difficulty}
433
+ </div>
434
+ </div>
435
+ </motion.div>
436
+ )}
437
+
438
  {!episode.done && episode.conversation.length === 0 && (
439
  <motion.div
440
  className="mb-4 rounded-lg border border-primary/20 bg-primary/5 p-4"
 
618
  {episode.done && <ReplayViewer messages={episode.conversation} />}
619
  </motion.div>
620
  </div>
621
+
622
+ {episode.done && (
623
+ <motion.div
624
+ ref={resultsRef}
625
+ className="mt-6"
626
+ initial={{ opacity: 0, y: 12 }}
627
+ animate={{ opacity: 1, y: 0 }}
628
+ transition={{ delay: 0.35 }}
629
+ >
630
+ <EpisodeResultsReport episode={episode} />
631
+ </motion.div>
632
+ )}
633
  </div>
634
  );
635
  }
frontend/src/pages/TrainingPage.tsx ADDED
@@ -0,0 +1,365 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import type { ReactNode } from 'react';
2
+ import { motion } from 'framer-motion';
3
+ import {
4
+ BrainCircuit,
5
+ CheckCircle2,
6
+ FileCode2,
7
+ FlaskConical,
8
+ Gauge,
9
+ GraduationCap,
10
+ AlertTriangle,
11
+ Wrench,
12
+ TerminalSquare,
13
+ } from 'lucide-react';
14
+ import {
15
+ ResponsiveContainer,
16
+ LineChart,
17
+ Line,
18
+ CartesianGrid,
19
+ XAxis,
20
+ YAxis,
21
+ Tooltip,
22
+ Legend,
23
+ BarChart,
24
+ Bar,
25
+ } from 'recharts';
26
+ import {
27
+ HOLDOUT_COMPARE,
28
+ LAB_MANAGER_PREVIEW_ARTIFACT,
29
+ LIVE_CHECKPOINTS,
30
+ LOCAL_BASELINE_SUMMARY,
31
+ SCIENTIST_PREVIEW_ARTIFACT,
32
+ TRAINING_ASSESSMENT,
33
+ TRAINING_LOG_ROWS,
34
+ } from '@/data/trainingArtifacts';
35
+ import { cn, formatReward, formatScore } from '@/lib/utils';
36
+
37
+ const compareBars = [
38
+ {
39
+ metric: 'Avg reward',
40
+ baseline: HOLDOUT_COMPARE.summary.baseline_avg_reward,
41
+ trained: HOLDOUT_COMPARE.summary.trained_avg_reward,
42
+ },
43
+ {
44
+ metric: 'Agreement %',
45
+ baseline: HOLDOUT_COMPARE.summary.baseline_agreement_rate * 100,
46
+ trained: HOLDOUT_COMPARE.summary.trained_agreement_rate * 100,
47
+ },
48
+ {
49
+ metric: 'Invalid %',
50
+ baseline: HOLDOUT_COMPARE.summary.baseline_invalid_rate * 100,
51
+ trained: HOLDOUT_COMPARE.summary.trained_invalid_rate * 100,
52
+ },
53
+ {
54
+ metric: 'Avg rounds',
55
+ baseline: HOLDOUT_COMPARE.summary.baseline_avg_rounds,
56
+ trained: HOLDOUT_COMPARE.summary.trained_avg_rounds,
57
+ },
58
+ ];
59
+
60
+ const checkpointChart = LIVE_CHECKPOINTS.map((checkpoint) => ({
61
+ label: checkpoint.label,
62
+ reward: Number(checkpoint.averageReward.toFixed(3)),
63
+ agreementPercent: Number((checkpoint.agreementRate * 100).toFixed(1)),
64
+ rounds: Number(checkpoint.averageRounds.toFixed(2)),
65
+ }));
66
+
67
+ export default function TrainingPage() {
68
+ const latestCheckpoint = LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1];
69
+
70
+ return (
71
+ <div className="mx-auto max-w-screen-xl px-4 py-8">
72
+ <motion.div
73
+ className="mb-8 text-center"
74
+ initial={{ opacity: 0, y: -10 }}
75
+ animate={{ opacity: 1, y: 0 }}
76
+ >
77
+ <div className="mb-3 inline-flex items-center gap-2 rounded-full bg-primary/10 px-4 py-1.5 text-sm font-medium text-primary">
78
+ <GraduationCap className="h-4 w-4" />
79
+ Training Logs And Analysis
80
+ </div>
81
+ <h1 className="mb-2 text-3xl font-bold tracking-tight">Scientist And Lab Manager Training Status</h1>
82
+ <p className="mx-auto max-w-3xl text-muted-foreground">
83
+ This page is backed by the real ReplicaLab training artifacts already generated in the repo. It shows what
84
+ the training stack achieved, where it is still failing, and whether more training is needed before claiming
85
+ the agents are actually better.
86
+ </p>
87
+ </motion.div>
88
+
89
+ <motion.div
90
+ className="mb-6 grid gap-4 lg:grid-cols-4"
91
+ initial={{ opacity: 0, y: 10 }}
92
+ animate={{ opacity: 1, y: 0 }}
93
+ transition={{ delay: 0.05 }}
94
+ >
95
+ <StatusCard
96
+ icon={<FileCode2 className="h-4 w-4" />}
97
+ title="Scientist preview"
98
+ value={`${SCIENTIST_PREVIEW_ARTIFACT.datasetSize} prompts`}
99
+ detail={`${SCIENTIST_PREVIEW_ARTIFACT.modelName} · max steps ${SCIENTIST_PREVIEW_ARTIFACT.config.maxSteps}`}
100
+ />
101
+ <StatusCard
102
+ icon={<FlaskConical className="h-4 w-4" />}
103
+ title="Lab Manager preview"
104
+ value={`${LAB_MANAGER_PREVIEW_ARTIFACT.datasetSize} samples`}
105
+ detail={`${LAB_MANAGER_PREVIEW_ARTIFACT.modelName} · ${LAB_MANAGER_PREVIEW_ARTIFACT.config.numTrainEpochs} epoch preview`}
106
+ />
107
+ <StatusCard
108
+ icon={<Gauge className="h-4 w-4" />}
109
+ title="Latest live RL checkpoint"
110
+ value={formatReward(latestCheckpoint.averageReward)}
111
+ detail={`${formatScore(latestCheckpoint.agreementRate)} agreement at artifact step ${latestCheckpoint.artifactStep}`}
112
+ />
113
+ <StatusCard
114
+ icon={<AlertTriangle className="h-4 w-4" />}
115
+ title="Do we need more training?"
116
+ value="Yes"
117
+ detail="Hold-out compare still favors baseline."
118
+ tone="warn"
119
+ />
120
+ </motion.div>
121
+
122
+ <motion.div
123
+ className="mb-6 grid gap-4 lg:grid-cols-2"
124
+ initial={{ opacity: 0, y: 12 }}
125
+ animate={{ opacity: 1, y: 0 }}
126
+ transition={{ delay: 0.1 }}
127
+ >
128
+ <ChartCard
129
+ title="Live RL checkpoint progression"
130
+ subtitle="Real checkpoint summaries from the ART/OpenEnv scientist run."
131
+ >
132
+ <div className="h-72">
133
+ <ResponsiveContainer width="100%" height="100%">
134
+ <LineChart data={checkpointChart}>
135
+ <CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
136
+ <XAxis dataKey="label" tick={{ fontSize: 11 }} />
137
+ <YAxis yAxisId="left" tick={{ fontSize: 11 }} />
138
+ <YAxis
139
+ yAxisId="right"
140
+ orientation="right"
141
+ tick={{ fontSize: 11 }}
142
+ tickFormatter={(value) => `${value}%`}
143
+ />
144
+ <Tooltip />
145
+ <Legend wrapperStyle={{ fontSize: '12px' }} />
146
+ <Line yAxisId="left" type="monotone" dataKey="reward" stroke="var(--color-primary)" strokeWidth={2.5} name="Avg reward" />
147
+ <Line yAxisId="right" type="monotone" dataKey="agreementPercent" stroke="var(--color-lab-manager)" strokeWidth={2.5} name="Agreement %" />
148
+ <Line yAxisId="left" type="monotone" dataKey="rounds" stroke="var(--color-judge)" strokeWidth={2} name="Avg rounds" />
149
+ </LineChart>
150
+ </ResponsiveContainer>
151
+ </div>
152
+ </ChartCard>
153
+
154
+ <ChartCard
155
+ title="Hold-out compare"
156
+ subtitle="Real seeded comparison between the deterministic baseline and the current trained scientist."
157
+ >
158
+ <div className="h-72">
159
+ <ResponsiveContainer width="100%" height="100%">
160
+ <BarChart data={compareBars}>
161
+ <CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
162
+ <XAxis dataKey="metric" tick={{ fontSize: 11 }} />
163
+ <YAxis tick={{ fontSize: 11 }} />
164
+ <Tooltip />
165
+ <Legend wrapperStyle={{ fontSize: '12px' }} />
166
+ <Bar dataKey="baseline" fill="var(--color-lab-manager)" radius={[6, 6, 0, 0]} name="Baseline" />
167
+ <Bar dataKey="trained" fill="var(--color-destructive)" radius={[6, 6, 0, 0]} name="Trained" />
168
+ </BarChart>
169
+ </ResponsiveContainer>
170
+ </div>
171
+ </ChartCard>
172
+ </motion.div>
173
+
174
+ <motion.div
175
+ className="mb-6 grid gap-4 lg:grid-cols-[1.35fr_1fr]"
176
+ initial={{ opacity: 0, y: 12 }}
177
+ animate={{ opacity: 1, y: 0 }}
178
+ transition={{ delay: 0.16 }}
179
+ >
180
+ <div className="rounded-xl border border-border bg-card p-5">
181
+ <div className="mb-4 flex items-center gap-2">
182
+ <TerminalSquare className="h-4 w-4 text-primary" />
183
+ <h2 className="text-base font-semibold">Training log highlights</h2>
184
+ </div>
185
+ <div className="space-y-3">
186
+ {TRAINING_LOG_ROWS.map((row) => (
187
+ <div key={row.label} className="rounded-lg border border-border bg-muted/30 p-3">
188
+ <div className="flex flex-wrap items-center gap-2 text-[11px] font-medium">
189
+ <span className="rounded-full bg-background px-2 py-1 text-primary">step {row.trainingStep}</span>
190
+ <span className="rounded-full bg-background px-2 py-1 text-primary">seed {row.seed}</span>
191
+ <span className={cn(
192
+ 'rounded-full px-2 py-1',
193
+ row.verdict === 'accept' ? 'bg-lab-manager/10 text-lab-manager' : 'bg-destructive/10 text-destructive',
194
+ )}>
195
+ {row.verdict ?? 'no verdict'}
196
+ </span>
197
+ </div>
198
+ <div className="mt-2 font-medium">{row.paperTitle}</div>
199
+ <div className="mt-2 grid gap-2 text-xs text-muted-foreground sm:grid-cols-4">
200
+ <span>reward {formatReward(row.reward)}</span>
201
+ <span>rounds {row.roundsUsed}</span>
202
+ <span>invalid {row.invalidActionCount}</span>
203
+ <span>parse {row.parseErrorCount}</span>
204
+ </div>
205
+ <p className="mt-2 text-xs text-muted-foreground">{row.note}</p>
206
+ </div>
207
+ ))}
208
+ </div>
209
+ </div>
210
+
211
+ <div className="space-y-4">
212
+ <SummaryCard
213
+ title="What we achieved"
214
+ icon={<CheckCircle2 className="h-4 w-4" />}
215
+ items={TRAINING_ASSESSMENT.achieved}
216
+ tone="good"
217
+ />
218
+ <SummaryCard
219
+ title="Why the current model still fails"
220
+ icon={<BrainCircuit className="h-4 w-4" />}
221
+ items={TRAINING_ASSESSMENT.gaps}
222
+ tone="warn"
223
+ />
224
+ <SummaryCard
225
+ title="What to improve next"
226
+ icon={<Wrench className="h-4 w-4" />}
227
+ items={TRAINING_ASSESSMENT.improvements}
228
+ tone="neutral"
229
+ />
230
+ </div>
231
+ </motion.div>
232
+
233
+ <motion.div
234
+ className="grid gap-4 lg:grid-cols-3"
235
+ initial={{ opacity: 0, y: 12 }}
236
+ animate={{ opacity: 1, y: 0 }}
237
+ transition={{ delay: 0.22 }}
238
+ >
239
+ <InfoPanel
240
+ title="Local deterministic baseline"
241
+ lines={[
242
+ `${LOCAL_BASELINE_SUMMARY.episodeCount} episodes across all three scenario families`,
243
+ `Average reward ${formatReward(LOCAL_BASELINE_SUMMARY.averageReward)}`,
244
+ `Agreement ${formatScore(LOCAL_BASELINE_SUMMARY.agreementRate)}`,
245
+ `Average rounds ${LOCAL_BASELINE_SUMMARY.averageRounds.toFixed(1)}`,
246
+ ]}
247
+ />
248
+ <InfoPanel
249
+ title="Scientist run config"
250
+ lines={[
251
+ SCIENTIST_PREVIEW_ARTIFACT.modelName,
252
+ `LoRA rank ${SCIENTIST_PREVIEW_ARTIFACT.config.loraRank}`,
253
+ `Max seq length ${SCIENTIST_PREVIEW_ARTIFACT.config.maxSeqLength}`,
254
+ `Learning rate ${SCIENTIST_PREVIEW_ARTIFACT.config.learningRate}`,
255
+ ]}
256
+ />
257
+ <InfoPanel
258
+ title="Lab Manager status"
259
+ lines={[
260
+ `Preview dataset ${LAB_MANAGER_PREVIEW_ARTIFACT.datasetSize} samples`,
261
+ `Evidence pack ${LAB_MANAGER_PREVIEW_ARTIFACT.evidencePackVersion}`,
262
+ 'Preview artifact exists, but no live evaluated SFT adapter is shown in the demo yet',
263
+ 'Next step is a real trained-plus-evaluated Lab Manager run',
264
+ ]}
265
+ />
266
+ </motion.div>
267
+ </div>
268
+ );
269
+ }
270
+
271
+ function StatusCard({
272
+ icon,
273
+ title,
274
+ value,
275
+ detail,
276
+ tone = 'neutral',
277
+ }: {
278
+ icon: ReactNode;
279
+ title: string;
280
+ value: string;
281
+ detail: string;
282
+ tone?: 'neutral' | 'warn';
283
+ }) {
284
+ return (
285
+ <div className={cn(
286
+ 'rounded-xl border bg-card p-4',
287
+ tone === 'warn' ? 'border-judge/30' : 'border-border',
288
+ )}>
289
+ <div className="mb-2 flex items-center gap-2 text-sm font-medium text-muted-foreground">
290
+ {icon}
291
+ {title}
292
+ </div>
293
+ <div className={cn('text-2xl font-bold', tone === 'warn' ? 'text-judge' : 'text-foreground')}>
294
+ {value}
295
+ </div>
296
+ <p className="mt-1 text-sm text-muted-foreground">{detail}</p>
297
+ </div>
298
+ );
299
+ }
300
+
301
+ function ChartCard({
302
+ title,
303
+ subtitle,
304
+ children,
305
+ }: {
306
+ title: string;
307
+ subtitle: string;
308
+ children: ReactNode;
309
+ }) {
310
+ return (
311
+ <div className="rounded-xl border border-border bg-card p-5">
312
+ <h2 className="text-base font-semibold">{title}</h2>
313
+ <p className="mt-1 text-sm text-muted-foreground">{subtitle}</p>
314
+ <div className="mt-4">{children}</div>
315
+ </div>
316
+ );
317
+ }
318
+
319
+ function SummaryCard({
320
+ title,
321
+ icon,
322
+ items,
323
+ tone,
324
+ }: {
325
+ title: string;
326
+ icon: ReactNode;
327
+ items: string[];
328
+ tone: 'good' | 'warn' | 'neutral';
329
+ }) {
330
+ return (
331
+ <div className={cn(
332
+ 'rounded-xl border bg-card p-4',
333
+ tone === 'good' && 'border-lab-manager/30',
334
+ tone === 'warn' && 'border-judge/30',
335
+ tone === 'neutral' && 'border-border',
336
+ )}>
337
+ <div className="mb-3 flex items-center gap-2 text-sm font-semibold">
338
+ {icon}
339
+ {title}
340
+ </div>
341
+ <div className="space-y-2">
342
+ {items.map((item) => (
343
+ <div key={item} className="rounded-lg border border-border bg-muted/30 px-3 py-2 text-xs text-muted-foreground">
344
+ {item}
345
+ </div>
346
+ ))}
347
+ </div>
348
+ </div>
349
+ );
350
+ }
351
+
352
+ function InfoPanel({ title, lines }: { title: string; lines: string[] }) {
353
+ return (
354
+ <div className="rounded-xl border border-border bg-card p-4">
355
+ <h2 className="text-sm font-semibold">{title}</h2>
356
+ <div className="mt-3 space-y-2">
357
+ {lines.map((line) => (
358
+ <div key={line} className="rounded-lg border border-border bg-muted/30 px-3 py-2 text-xs text-muted-foreground">
359
+ {line}
360
+ </div>
361
+ ))}
362
+ </div>
363
+ </div>
364
+ );
365
+ }
frontend/src/types/index.ts CHANGED
@@ -3,6 +3,7 @@ export type LabManagerActionType = 'report_feasibility' | 'suggest_alternative'
3
  export type Difficulty = 'easy' | 'medium' | 'hard';
4
  export type ScenarioTemplate = 'math_reasoning' | 'ml_benchmark' | 'finance_trading';
5
  export type Role = 'scientist' | 'lab_manager' | 'judge' | 'system';
 
6
 
7
  // --- Backend-aligned action contracts ---
8
 
@@ -103,6 +104,21 @@ export interface JudgeAudit {
103
  score_breakdown: ScoreBreakdown;
104
  }
105
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
  // --- Frontend episode state (assembled from backend responses) ---
107
 
108
  export interface EpisodeState {
@@ -120,6 +136,9 @@ export interface EpisodeState {
120
  conversation: NegotiationMessage[];
121
  scores: ScoreBreakdown | null;
122
  judge_audit: JudgeAudit | null;
 
 
 
123
  }
124
 
125
  export interface ResetParams {
@@ -190,8 +209,12 @@ export interface BackendStepInfo {
190
  verdict: string | null;
191
  top_failure_reasons: string[];
192
  round: number;
193
- stub: boolean;
194
  episode_id: string;
 
 
 
 
 
195
  }
196
 
197
  export interface BackendStepResult {
 
3
  export type Difficulty = 'easy' | 'medium' | 'hard';
4
  export type ScenarioTemplate = 'math_reasoning' | 'ml_benchmark' | 'finance_trading';
5
  export type Role = 'scientist' | 'lab_manager' | 'judge' | 'system';
6
+ export type DemoCase = 'fast-agreement' | 'learning-opportunity' | 'no-agreement';
7
 
8
  // --- Backend-aligned action contracts ---
9
 
 
104
  score_breakdown: ScoreBreakdown;
105
  }
106
 
107
+ export interface EpisodeStepTrace {
108
+ round: number;
109
+ reward: number;
110
+ cumulative_reward: number;
111
+ action_type: string;
112
+ scientist_message: string;
113
+ lab_manager_action_type?: string;
114
+ lab_manager_message?: string;
115
+ step_reward_components: Record<string, number>;
116
+ protocol: Protocol | null;
117
+ oracle_round_score?: Record<string, unknown> | null;
118
+ oracle_post_mortem?: Record<string, unknown> | null;
119
+ oracle_event?: Record<string, unknown> | null;
120
+ }
121
+
122
  // --- Frontend episode state (assembled from backend responses) ---
123
 
124
  export interface EpisodeState {
 
136
  conversation: NegotiationMessage[];
137
  scores: ScoreBreakdown | null;
138
  judge_audit: JudgeAudit | null;
139
+ cumulative_reward: number;
140
+ step_history: EpisodeStepTrace[];
141
+ demo_case?: DemoCase;
142
  }
143
 
144
  export interface ResetParams {
 
209
  verdict: string | null;
210
  top_failure_reasons: string[];
211
  round: number;
 
212
  episode_id: string;
213
+ step_reward_components?: Record<string, number>;
214
+ cumulative_reward?: number;
215
+ oracle_round_score?: Record<string, unknown> | null;
216
+ oracle_post_mortem?: Record<string, unknown> | null;
217
+ oracle_event?: Record<string, unknown> | null;
218
  }
219
 
220
  export interface BackendStepResult {
scripts/build_demo_video.py ADDED
@@ -0,0 +1,476 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ import os
5
+ import shutil
6
+ import subprocess
7
+ import sys
8
+ import textwrap
9
+ import time
10
+ import urllib.parse
11
+ import urllib.request
12
+ from dataclasses import dataclass
13
+ from pathlib import Path
14
+ from typing import Iterable
15
+
16
+ from PIL import Image, ImageDraw, ImageFont, ImageOps
17
+ from selenium import webdriver
18
+ from selenium.common.exceptions import TimeoutException
19
+ from selenium.webdriver.chrome.options import Options
20
+ from selenium.webdriver.common.by import By
21
+ from selenium.webdriver.support.ui import WebDriverWait
22
+
23
+
24
+ ROOT = Path(__file__).resolve().parents[1]
25
+ OUTPUT_DIR = ROOT / "replicalab" / "outputs" / "demo_video"
26
+ SCREENS_DIR = OUTPUT_DIR / "screens"
27
+ SLIDES_DIR = OUTPUT_DIR / "slides"
28
+ AUDIO_DIR = OUTPUT_DIR / "audio"
29
+ TEXT_DIR = OUTPUT_DIR / "text"
30
+ CHROME_PATH = Path(r"C:\Program Files\Google\Chrome\Application\chrome.exe")
31
+ BASE_URL = "http://127.0.0.1:7860"
32
+ ONBOARDING_KEY = "replicalab-onboarded"
33
+ THEME_KEY = "replicalab-theme"
34
+ VIDEO_SIZE = (1920, 1080)
35
+ NARRATION = (
36
+ "ReplicaLab starts from a research paper and turns it into a seeded replication benchmark. "
37
+ "The Scientist proposes a protocol, the Lab Manager enforces budget, tools, and scheduling, "
38
+ "and a deterministic Judge scores rigor, feasibility, and fidelity. In our first scenario, "
39
+ "the agents agree immediately, so the paper looks replicable in this lab. In the second scenario, "
40
+ "they negotiate across all six rounds, which creates a rich reinforcement learning signal. "
41
+ "In the third, they never resolve the blockers, so the system rejects the paper for the current setup. "
42
+ "Because every outcome is scored deterministically, we can train the Scientist with Unsloth and TRL, "
43
+ "compare baseline versus trained runs, inspect real logs, and see exactly where more learning is still needed. "
44
+ "The training page is intentionally honest: the live run reached positive rewards, but the held-out compare still "
45
+ "shows that the trained Scientist has not beaten the deterministic baseline yet."
46
+ )
47
+
48
+
49
+ @dataclass(frozen=True)
50
+ class Scene:
51
+ id: str
52
+ url: str
53
+ title: str
54
+ subtitle: str
55
+ duration: float
56
+ expected_text: str | None = None
57
+
58
+
59
+ SCENES: tuple[Scene, ...] = (
60
+ Scene(
61
+ id="dashboard",
62
+ url=f"{BASE_URL}/",
63
+ title="Paper to benchmark",
64
+ subtitle="ReplicaLab turns a paper into a seeded replication benchmark.",
65
+ duration=8.0,
66
+ expected_text="ReplicaLab",
67
+ ),
68
+ Scene(
69
+ id="fast_agreement",
70
+ url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1&demoCase=fast-agreement",
71
+ title="Scenario 1: first-round agreement",
72
+ subtitle="The agents converge quickly and the paper scores as a strong replication candidate.",
73
+ duration=11.0,
74
+ expected_text="Completed: First-round agreement",
75
+ ),
76
+ Scene(
77
+ id="learning_opportunity",
78
+ url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=202&demo=1&autoplay=1&demoCase=learning-opportunity",
79
+ title="Scenario 2: multi-round learning",
80
+ subtitle="Six rounds of disagreement create a rich RL signal before the final acceptance.",
81
+ duration=13.0,
82
+ expected_text="Completed: Multi-round learning opportunity",
83
+ ),
84
+ Scene(
85
+ id="no_agreement",
86
+ url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=303&demo=1&autoplay=1&demoCase=no-agreement",
87
+ title="Scenario 3: no agreement",
88
+ subtitle="The blockers remain unresolved, so the system rejects replication for this setup.",
89
+ duration=12.0,
90
+ expected_text="Completed: No agreement reached",
91
+ ),
92
+ Scene(
93
+ id="training",
94
+ url=f"{BASE_URL}/training",
95
+ title="Artifact-backed training review",
96
+ subtitle="The training page shows real checkpoints, real compare metrics, and what still needs improvement.",
97
+ duration=16.0,
98
+ expected_text="Training Logs And Analysis",
99
+ ),
100
+ )
101
+
102
+
103
+ def load_env_value(key: str) -> str | None:
104
+ if os.getenv(key):
105
+ return os.getenv(key)
106
+
107
+ for path in (ROOT / ".env", ROOT / ".env.local", ROOT / "frontend" / ".env"):
108
+ if not path.exists():
109
+ continue
110
+ for raw in path.read_text(encoding="utf-8").splitlines():
111
+ line = raw.strip()
112
+ if not line or line.startswith("#") or "=" not in line:
113
+ continue
114
+ env_key, env_value = line.split("=", 1)
115
+ if env_key.strip() == key:
116
+ return env_value.strip().strip('"').strip("'")
117
+ return None
118
+
119
+
120
+ def ensure_backend() -> None:
121
+ try:
122
+ with urllib.request.urlopen(f"{BASE_URL}/health", timeout=10) as response:
123
+ data = json.loads(response.read().decode("utf-8"))
124
+ except Exception as exc: # pragma: no cover - user-facing failure
125
+ raise RuntimeError(
126
+ f"Backend unavailable at {BASE_URL}. Start it with "
127
+ "\"python -m uvicorn server.app:app --host 127.0.0.1 --port 7860\"."
128
+ ) from exc
129
+ if data.get("status") != "ok":
130
+ raise RuntimeError(f"Unexpected backend health payload: {data}")
131
+
132
+
133
+ def ensure_output_dirs() -> None:
134
+ for directory in (OUTPUT_DIR, SCREENS_DIR, SLIDES_DIR, AUDIO_DIR, TEXT_DIR):
135
+ directory.mkdir(parents=True, exist_ok=True)
136
+
137
+
138
+ def get_font(size: int, bold: bool = False) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
139
+ candidates = [
140
+ Path(r"C:\Windows\Fonts\segoeuib.ttf" if bold else r"C:\Windows\Fonts\segoeui.ttf"),
141
+ Path(r"C:\Windows\Fonts\arialbd.ttf" if bold else r"C:\Windows\Fonts\arial.ttf"),
142
+ ]
143
+ for candidate in candidates:
144
+ if candidate.exists():
145
+ return ImageFont.truetype(str(candidate), size=size)
146
+ return ImageFont.load_default()
147
+
148
+
149
+ def pick_voice(api_key: str, preferred_voice_id: str | None) -> str:
150
+ if preferred_voice_id:
151
+ return preferred_voice_id
152
+
153
+ request = urllib.request.Request(
154
+ "https://api.elevenlabs.io/v1/voices",
155
+ headers={"xi-api-key": api_key, "Accept": "application/json"},
156
+ method="GET",
157
+ )
158
+ with urllib.request.urlopen(request, timeout=30) as response:
159
+ payload = json.loads(response.read().decode("utf-8"))
160
+
161
+ voices = payload.get("voices", [])
162
+ if not voices:
163
+ raise RuntimeError("ElevenLabs returned no voices for the current API key.")
164
+
165
+ preferred_names = ("Rachel", "Aria", "Sarah", "Charlie", "George")
166
+ for name in preferred_names:
167
+ for voice in voices:
168
+ if voice.get("name") == name and voice.get("voice_id"):
169
+ return voice["voice_id"]
170
+ for voice in voices:
171
+ if voice.get("voice_id"):
172
+ return voice["voice_id"]
173
+ raise RuntimeError("No usable ElevenLabs voice_id found.")
174
+
175
+
176
+ def synthesize_voiceover(api_key: str, voice_id: str, text: str, output_path: Path) -> None:
177
+ payload = {
178
+ "text": text,
179
+ "model_id": "eleven_multilingual_v2",
180
+ "voice_settings": {
181
+ "stability": 0.4,
182
+ "similarity_boost": 0.8,
183
+ "style": 0.25,
184
+ "use_speaker_boost": True,
185
+ },
186
+ }
187
+ body = json.dumps(payload).encode("utf-8")
188
+ urls = (
189
+ f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}?output_format=mp3_44100_128",
190
+ f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
191
+ )
192
+
193
+ last_error: Exception | None = None
194
+ for url in urls:
195
+ request = urllib.request.Request(
196
+ url,
197
+ data=body,
198
+ headers={
199
+ "xi-api-key": api_key,
200
+ "Accept": "audio/mpeg",
201
+ "Content-Type": "application/json",
202
+ },
203
+ method="POST",
204
+ )
205
+ try:
206
+ with urllib.request.urlopen(request, timeout=90) as response:
207
+ audio = response.read()
208
+ if not audio:
209
+ raise RuntimeError("ElevenLabs returned an empty audio payload.")
210
+ output_path.write_bytes(audio)
211
+ return
212
+ except Exception as exc: # pragma: no cover - fallback path
213
+ last_error = exc
214
+ raise RuntimeError(f"Failed to synthesize ElevenLabs audio: {last_error}")
215
+
216
+
217
+ def build_driver() -> webdriver.Chrome:
218
+ options = Options()
219
+ options.binary_location = str(CHROME_PATH)
220
+ options.add_argument("--headless=new")
221
+ options.add_argument("--disable-gpu")
222
+ options.add_argument("--window-size=1600,1200")
223
+ options.add_argument("--force-device-scale-factor=1")
224
+ options.add_argument("--hide-scrollbars")
225
+ options.add_argument("--mute-audio")
226
+ options.add_argument("--no-first-run")
227
+ options.add_argument("--no-default-browser-check")
228
+ options.add_argument(f"--user-data-dir={OUTPUT_DIR / 'chrome_profile'}")
229
+ driver = webdriver.Chrome(options=options)
230
+ driver.set_window_size(1600, 1200)
231
+ return driver
232
+
233
+
234
+ def capture_screens() -> list[Path]:
235
+ driver = build_driver()
236
+ files: list[Path] = []
237
+ try:
238
+ driver.get(BASE_URL)
239
+ WebDriverWait(driver, 20).until(
240
+ lambda d: d.execute_script("return document.readyState") == "complete"
241
+ )
242
+ driver.execute_script(
243
+ "window.localStorage.setItem(arguments[0], '1');"
244
+ "window.localStorage.setItem(arguments[1], 'light');",
245
+ ONBOARDING_KEY,
246
+ THEME_KEY,
247
+ )
248
+
249
+ for scene in SCENES:
250
+ driver.get(scene.url)
251
+ WebDriverWait(driver, 30).until(
252
+ lambda d: d.execute_script("return document.readyState") == "complete"
253
+ )
254
+ if scene.expected_text:
255
+ try:
256
+ WebDriverWait(driver, 35).until(
257
+ lambda d: scene.expected_text in d.page_source
258
+ )
259
+ except TimeoutException:
260
+ pass
261
+ time.sleep(1.5)
262
+ output = SCREENS_DIR / f"{scene.id}.png"
263
+ driver.save_screenshot(str(output))
264
+ files.append(output)
265
+ finally:
266
+ driver.quit()
267
+ return files
268
+
269
+
270
+ def wrap_text(draw: ImageDraw.ImageDraw, text: str, font: ImageFont.ImageFont, width: int) -> list[str]:
271
+ words = text.split()
272
+ lines: list[str] = []
273
+ current = ""
274
+ for word in words:
275
+ candidate = f"{current} {word}".strip()
276
+ if draw.textlength(candidate, font=font) <= width:
277
+ current = candidate
278
+ else:
279
+ if current:
280
+ lines.append(current)
281
+ current = word
282
+ if current:
283
+ lines.append(current)
284
+ return lines
285
+
286
+
287
+ def create_slides() -> list[Path]:
288
+ title_font = get_font(46, bold=True)
289
+ subtitle_font = get_font(28)
290
+ badge_font = get_font(24, bold=True)
291
+ output_paths: list[Path] = []
292
+
293
+ for scene in SCENES:
294
+ raw_image = Image.open(SCREENS_DIR / f"{scene.id}.png").convert("RGB")
295
+ canvas = ImageOps.fit(raw_image, VIDEO_SIZE, method=Image.Resampling.LANCZOS)
296
+ overlay = Image.new("RGBA", VIDEO_SIZE, (0, 0, 0, 0))
297
+ draw = ImageDraw.Draw(overlay)
298
+ draw.rounded_rectangle((60, 780, 1860, 1020), radius=36, fill=(7, 13, 36, 190))
299
+ draw.rounded_rectangle((60, 56, 520, 116), radius=30, fill=(99, 102, 241, 220))
300
+ draw.text((92, 72), "ReplicaLab - 60 second demo", font=badge_font, fill=(255, 255, 255))
301
+ draw.text((96, 820), scene.title, font=title_font, fill=(255, 255, 255))
302
+ subtitle_lines = wrap_text(draw, scene.subtitle, subtitle_font, width=1620)
303
+ y = 888
304
+ for line in subtitle_lines:
305
+ draw.text((96, y), line, font=subtitle_font, fill=(226, 232, 240))
306
+ y += 40
307
+
308
+ final = Image.alpha_composite(canvas.convert("RGBA"), overlay).convert("RGB")
309
+ slide_path = SLIDES_DIR / f"{scene.id}.png"
310
+ final.save(slide_path, quality=95)
311
+ output_paths.append(slide_path)
312
+ return output_paths
313
+
314
+
315
+ def write_concat_file(paths: Iterable[Path]) -> Path:
316
+ concat_path = TEXT_DIR / "slides.txt"
317
+ lines: list[str] = []
318
+ ordered = list(paths)
319
+ for scene, path in zip(SCENES, ordered):
320
+ lines.append(f"file '{path.as_posix()}'")
321
+ lines.append(f"duration {scene.duration:.2f}")
322
+ lines.append(f"file '{ordered[-1].as_posix()}'")
323
+ concat_path.write_text("\n".join(lines), encoding="utf-8")
324
+ return concat_path
325
+
326
+
327
+ def write_script_assets() -> None:
328
+ (TEXT_DIR / "voiceover.txt").write_text(NARRATION, encoding="utf-8")
329
+ (TEXT_DIR / "shot_list.json").write_text(
330
+ json.dumps(
331
+ [
332
+ {
333
+ "id": scene.id,
334
+ "title": scene.title,
335
+ "subtitle": scene.subtitle,
336
+ "url": scene.url,
337
+ "duration_seconds": scene.duration,
338
+ }
339
+ for scene in SCENES
340
+ ],
341
+ indent=2,
342
+ ),
343
+ encoding="utf-8",
344
+ )
345
+
346
+
347
+ def seconds_to_srt(value: float) -> str:
348
+ millis = int(round(value * 1000))
349
+ hours, millis = divmod(millis, 3_600_000)
350
+ minutes, millis = divmod(millis, 60_000)
351
+ seconds, millis = divmod(millis, 1000)
352
+ return f"{hours:02}:{minutes:02}:{seconds:02},{millis:03}"
353
+
354
+
355
+ def write_srt() -> None:
356
+ lines = wrap_text(ImageDraw.Draw(Image.new("RGB", (1, 1))), NARRATION, get_font(30), 72 * 18)
357
+ segment_count = max(1, len(lines))
358
+ total_duration = sum(scene.duration for scene in SCENES)
359
+ step = total_duration / segment_count
360
+ chunks = []
361
+ start = 0.0
362
+ for index, line in enumerate(lines, start=1):
363
+ end = min(total_duration, start + step)
364
+ chunks.append(f"{index}\n{seconds_to_srt(start)} --> {seconds_to_srt(end)}\n{line}\n")
365
+ start = end
366
+ (TEXT_DIR / "voiceover.srt").write_text("\n".join(chunks), encoding="utf-8")
367
+
368
+
369
+ def ffprobe_duration(path: Path) -> float:
370
+ result = subprocess.run(
371
+ [
372
+ "ffprobe",
373
+ "-v",
374
+ "error",
375
+ "-show_entries",
376
+ "format=duration",
377
+ "-of",
378
+ "default=noprint_wrappers=1:nokey=1",
379
+ str(path),
380
+ ],
381
+ check=True,
382
+ capture_output=True,
383
+ text=True,
384
+ )
385
+ return float(result.stdout.strip())
386
+
387
+
388
+ def run_ffmpeg(audio_path: Path, concat_path: Path) -> Path:
389
+ silent_video = OUTPUT_DIR / "replicalab_demo_60s_silent.mp4"
390
+ final_video = OUTPUT_DIR / "replicalab_demo_60s.mp4"
391
+
392
+ subprocess.run(
393
+ [
394
+ "ffmpeg",
395
+ "-y",
396
+ "-f",
397
+ "concat",
398
+ "-safe",
399
+ "0",
400
+ "-i",
401
+ str(concat_path),
402
+ "-vf",
403
+ "fps=30,format=yuv420p",
404
+ "-c:v",
405
+ "libx264",
406
+ "-pix_fmt",
407
+ "yuv420p",
408
+ str(silent_video),
409
+ ],
410
+ check=True,
411
+ )
412
+ subprocess.run(
413
+ [
414
+ "ffmpeg",
415
+ "-y",
416
+ "-i",
417
+ str(silent_video),
418
+ "-i",
419
+ str(audio_path),
420
+ "-c:v",
421
+ "copy",
422
+ "-c:a",
423
+ "aac",
424
+ "-b:a",
425
+ "192k",
426
+ "-shortest",
427
+ str(final_video),
428
+ ],
429
+ check=True,
430
+ )
431
+ return final_video
432
+
433
+
434
+ def main() -> int:
435
+ ensure_backend()
436
+ ensure_output_dirs()
437
+ write_script_assets()
438
+ write_srt()
439
+
440
+ api_key = load_env_value("ELEVENLABS_API_KEY")
441
+ if not api_key:
442
+ raise RuntimeError("ELEVENLABS_API_KEY was not found in the environment or .env file.")
443
+
444
+ voice_id = pick_voice(api_key, load_env_value("ELEVENLABS_VOICE_ID"))
445
+ audio_path = AUDIO_DIR / "voiceover.mp3"
446
+ synthesize_voiceover(api_key, voice_id, NARRATION, audio_path)
447
+
448
+ capture_screens()
449
+ slides = create_slides()
450
+ concat_path = write_concat_file(slides)
451
+ final_video = run_ffmpeg(audio_path, concat_path)
452
+
453
+ metadata = {
454
+ "voice_id": voice_id,
455
+ "audio_duration_seconds": round(ffprobe_duration(audio_path), 3),
456
+ "video_path": str(final_video),
457
+ "slides": [str(path) for path in slides],
458
+ }
459
+ (TEXT_DIR / "build_metadata.json").write_text(json.dumps(metadata, indent=2), encoding="utf-8")
460
+
461
+ print(textwrap.dedent(f"""
462
+ Built demo assets:
463
+ audio: {audio_path}
464
+ video: {final_video}
465
+ script: {TEXT_DIR / 'voiceover.txt'}
466
+ subtitles: {TEXT_DIR / 'voiceover.srt'}
467
+ """).strip())
468
+ return 0
469
+
470
+
471
+ if __name__ == "__main__":
472
+ try:
473
+ raise SystemExit(main())
474
+ except Exception as exc: # pragma: no cover - CLI path
475
+ print(f"ERROR: {exc}", file=sys.stderr)
476
+ raise