Spaces:
Running
Running
Update HF Space: final architecture diagram, frontend, and README
Browse files- Add ReplicaLab_Architecture_Final.svg with full scoring engine,
reward formula, penalties, and 4-dimension breakdown
- Update README to reference final diagram and reflect 100% completion
- Include all latest frontend components (demo flow, training page,
episode results, compare page updates)
- Updated API client, types, and demo data for complete UI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- README.md +4 -4
- ReplicaLab_Architecture_Final.svg +3 -0
- docs/changes.md +2 -0
- docs/demo_video_script_60s.md +13 -0
- docs/kush/notes.md +25 -0
- frontend/src/App.tsx +2 -0
- frontend/src/components/EpisodeComparison.tsx +1 -1
- frontend/src/components/EpisodeResultsReport.tsx +465 -0
- frontend/src/components/Header.tsx +2 -1
- frontend/src/components/TrainingResults.tsx +10 -47
- frontend/src/data/trainingArtifacts.ts +210 -0
- frontend/src/lib/api.ts +74 -21
- frontend/src/lib/demo.ts +168 -0
- frontend/src/pages/ComparePage.tsx +2 -1
- frontend/src/pages/DashboardPage.tsx +49 -10
- frontend/src/pages/EpisodePage.tsx +59 -7
- frontend/src/pages/TrainingPage.tsx +365 -0
- frontend/src/types/index.ts +24 -1
- scripts/build_demo_video.py +476 -0
README.md
CHANGED
|
@@ -16,7 +16,7 @@ pinned: false
|
|
| 16 |
|
| 17 |
ReplicaLab tackles this by training an AI Scientist agent to negotiate feasible replication plans under realistic resource constraints. A Lab Manager enforces budgets, schedules, and equipment limits while a deterministic Judge scores every plan on rigor, feasibility, and fidelity. Through reinforcement learning, the Scientist learns to ask better questions, make smarter tradeoffs, and reach agreement faster -- all without sacrificing scientific quality.
|
| 18 |
|
| 19 |
-
|
| 20 |
|
| 21 |
## Team Ownership
|
| 22 |
|
|
@@ -32,7 +32,7 @@ The initial domain focus is mathematics and machine learning, with offline finan
|
|
| 32 |
## Architecture
|
| 33 |
|
| 34 |
<p align="center">
|
| 35 |
-
<img src="./
|
| 36 |
</p>
|
| 37 |
|
| 38 |
ReplicaLab uses a **hybrid Oracle architecture**:
|
|
@@ -174,7 +174,7 @@ RL training improves the Scientist agent’s ability to negotiate effective, fea
|
|
| 174 |
- **Reduced-scale fallback:** `Qwen/Qwen3-4B`
|
| 175 |
- **Decision record:** `docs/agt11_scientist_model_selection.md`
|
| 176 |
|
| 177 |
-
###
|
| 178 |
|
| 179 |
1. Use `notebooks/train_minimal_colab.ipynb` as the sponsor-facing minimal Colab script for the Unsloth / HF TRL requirement
|
| 180 |
2. Use the judged notebook `notebooks/train_colab.ipynb` as the full readable driver
|
|
@@ -244,7 +244,7 @@ Difficulty scaling should mechanically tighten constraints, remove resources, or
|
|
| 244 |
```text
|
| 245 |
replicalab-ai/
|
| 246 |
├── README.md
|
| 247 |
-
├──
|
| 248 |
├── pyproject.toml
|
| 249 |
├── openenv.yaml
|
| 250 |
├── replicalab/
|
|
|
|
| 16 |
|
| 17 |
ReplicaLab tackles this by training an AI Scientist agent to negotiate feasible replication plans under realistic resource constraints. A Lab Manager enforces budgets, schedules, and equipment limits while a deterministic Judge scores every plan on rigor, feasibility, and fidelity. Through reinforcement learning, the Scientist learns to ask better questions, make smarter tradeoffs, and reach agreement faster -- all without sacrificing scientific quality.
|
| 18 |
|
| 19 |
+
Three scenario families ship today -- mathematics reasoning, ML benchmark replication, and offline finance/trading backtest design -- each with easy, medium, and hard difficulty scaling. Physics and biology remain future adapters after the core normalized scenario layer is stable.
|
| 20 |
|
| 21 |
## Team Ownership
|
| 22 |
|
|
|
|
| 32 |
## Architecture
|
| 33 |
|
| 34 |
<p align="center">
|
| 35 |
+
<img src="./ReplicaLab_Architecture_Final.svg" alt="ReplicaLab Final System Architecture" width="100%"/>
|
| 36 |
</p>
|
| 37 |
|
| 38 |
ReplicaLab uses a **hybrid Oracle architecture**:
|
|
|
|
| 174 |
- **Reduced-scale fallback:** `Qwen/Qwen3-4B`
|
| 175 |
- **Decision record:** `docs/agt11_scientist_model_selection.md`
|
| 176 |
|
| 177 |
+
### Training Path
|
| 178 |
|
| 179 |
1. Use `notebooks/train_minimal_colab.ipynb` as the sponsor-facing minimal Colab script for the Unsloth / HF TRL requirement
|
| 180 |
2. Use the judged notebook `notebooks/train_colab.ipynb` as the full readable driver
|
|
|
|
| 244 |
```text
|
| 245 |
replicalab-ai/
|
| 246 |
├── README.md
|
| 247 |
+
├── ReplicaLab_Architecture_Final.svg
|
| 248 |
├── pyproject.toml
|
| 249 |
├── openenv.yaml
|
| 250 |
├── replicalab/
|
ReplicaLab_Architecture_Final.svg
ADDED
|
|
Git LFS Details
|
docs/changes.md
CHANGED
|
@@ -81,4 +81,6 @@ Rules:
|
|
| 81 |
| 2026-03-08 | Person B (Ayush) | Frontend episode kickoff UX | Added explicit first-round call-to-action controls to the live episode view after user testing showed the page looked stuck immediately after reset | The reset state loaded the paper and constraints correctly, but both the step controls and protocol-editor submit action could sit below the fold, making the UI appear frozen at round 0 | Added an `Episode ready` banner plus in-panel `Advance First Round` and `Open Protocol Editor` actions, and updated the negotiation placeholder so it no longer says `Start an episode` after an episode is already active | Keep a visible first action near the negotiation panel in future layout changes |
|
| 82 |
| 2026-03-08 | Person B (Ayush) | One-click live demo automation | Extended the hero CTA flow so `Replicate a Paper` runs the seeded episode automatically instead of only opening the episode page | The hackathon demo needs a one-click narrative with live agent behavior and judge output; requiring manual reset and step clicks after entering the episode page weakens the demo | The dashboard now links to a seeded demo URL, the episode page auto-starts on demo routes, preserves demo query params in the shareable URL, and enables autoplay so the negotiation proceeds to judged completion with no extra clicks | If this is later generalized beyond the hero CTA, keep manual scenario-card entry as a separate non-demo path |
|
| 83 |
| 2026-03-08 | Person B (Ayush) | Frontend backend-availability diagnostics | Added a startup health check and contextual network-error messaging to the replication setup flow after the live demo surfaced a generic `Failed to fetch` banner | The page gave no actionable explanation when the local API server on port `7860` was not running, which made a recoverable environment issue look like an application bug | `Controls.tsx` now checks backend health on load, `frontend/src/lib/api.ts` rewrites fetch-network failures into an explicit uvicorn startup instruction, the frontend was rebuilt, and the local FastAPI server was restarted and verified healthy on `127.0.0.1:7860` | Keep the backend running during demo prep or use the integrated backend-served frontend on `http://127.0.0.1:7860` |
|
|
|
|
|
|
|
| 84 |
|
|
|
|
| 81 |
| 2026-03-08 | Person B (Ayush) | Frontend episode kickoff UX | Added explicit first-round call-to-action controls to the live episode view after user testing showed the page looked stuck immediately after reset | The reset state loaded the paper and constraints correctly, but both the step controls and protocol-editor submit action could sit below the fold, making the UI appear frozen at round 0 | Added an `Episode ready` banner plus in-panel `Advance First Round` and `Open Protocol Editor` actions, and updated the negotiation placeholder so it no longer says `Start an episode` after an episode is already active | Keep a visible first action near the negotiation panel in future layout changes |
|
| 82 |
| 2026-03-08 | Person B (Ayush) | One-click live demo automation | Extended the hero CTA flow so `Replicate a Paper` runs the seeded episode automatically instead of only opening the episode page | The hackathon demo needs a one-click narrative with live agent behavior and judge output; requiring manual reset and step clicks after entering the episode page weakens the demo | The dashboard now links to a seeded demo URL, the episode page auto-starts on demo routes, preserves demo query params in the shareable URL, and enables autoplay so the negotiation proceeds to judged completion with no extra clicks | If this is later generalized beyond the hero CTA, keep manual scenario-card entry as a separate non-demo path |
|
| 83 |
| 2026-03-08 | Person B (Ayush) | Frontend backend-availability diagnostics | Added a startup health check and contextual network-error messaging to the replication setup flow after the live demo surfaced a generic `Failed to fetch` banner | The page gave no actionable explanation when the local API server on port `7860` was not running, which made a recoverable environment issue look like an application bug | `Controls.tsx` now checks backend health on load, `frontend/src/lib/api.ts` rewrites fetch-network failures into an explicit uvicorn startup instruction, the frontend was rebuilt, and the local FastAPI server was restarted and verified healthy on `127.0.0.1:7860` | Keep the backend running during demo prep or use the integrated backend-served frontend on `http://127.0.0.1:7860` |
|
| 84 |
+
| 2026-03-08 | Person B (Ayush) | Frontend live demo outcomes and results report | Extended the one-click demo into three seeded story modes with a detailed post-episode report instead of a single autoplay path that stopped at the generic episode view | The hackathon demo needs to show distinct judged outcomes: immediate agreement, multi-round learning opportunity, and failure to reach agreement, all backed by real episode values rather than static mock copy | Added `fast-agreement`, `learning-opportunity`, and `no-agreement` dashboard launches; routed episode autoplay through scripted but backend-valid action sequences; added a results report with live reward charts, terminal score bars, training interpretation, reliability labeling, and tool-install suggestions; rebuilt the frontend and verified all three seeded ML runs against the live backend | If Oracle-backed narrative summaries are added later, keep the deterministic judge verdict and real score traces as the source of truth for the report |
|
| 85 |
+
| 2026-03-08 | Person B (Ayush) | Frontend training-status reporting | Replaced the packaged training teaser with an artifact-backed training page and honest improvement guidance | The demo needs a place to show training logs, real achieved values, and whether more training is still required; the earlier dashboard card implied progress but did not expose the real run outputs | Added a dedicated `/training` route, header navigation, a shared frontend data module sourced from real run summaries, and a new training page with checkpoint charts, compare bars, log highlights, preview-artifact status, and explicit `needs more training` analysis; rebuilt the frontend and reverified backend serving on `/training` | If future runs improve beyond baseline, update the shared training artifact data module first so the dashboard and training page stay consistent |
|
| 86 |
|
docs/demo_video_script_60s.md
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ReplicaLab 60s Demo Script
|
| 2 |
+
|
| 3 |
+
## Voiceover
|
| 4 |
+
|
| 5 |
+
ReplicaLab starts from a research paper and turns it into a seeded replication benchmark. The Scientist proposes a protocol, the Lab Manager enforces budget, tools, and scheduling, and a deterministic Judge scores rigor, feasibility, and fidelity. In our first scenario, the agents agree immediately, so the paper looks replicable in this lab. In the second scenario, they negotiate across all six rounds, which creates a rich reinforcement learning signal. In the third, they never resolve the blockers, so the system rejects the paper for the current setup. Because every outcome is scored deterministically, we can train the Scientist with Unsloth and TRL, compare baseline versus trained runs, inspect real logs, and see exactly where more learning is still needed. The training page is intentionally honest: the live run reached positive rewards, but the held-out compare still shows that the trained Scientist has not beaten the deterministic baseline yet.
|
| 6 |
+
|
| 7 |
+
## Shot List
|
| 8 |
+
|
| 9 |
+
1. Dashboard hero: introduce ReplicaLab and the paper-to-training loop.
|
| 10 |
+
2. First-round agreement: show a clean acceptance and high replicability score.
|
| 11 |
+
3. Multi-round learning: show six-round negotiation and the learning-opportunity results panel.
|
| 12 |
+
4. No agreement: show the timeout / rejection outcome and low reliability signal.
|
| 13 |
+
5. Training page: show artifact-backed logs, checkpoints, baseline-vs-trained compare, and the explicit note that more training is still required.
|
docs/kush/notes.md
CHANGED
|
@@ -17,3 +17,28 @@ Durable deviations belong in `docs/changes.md`.
|
|
| 17 |
- The dashboard `Replicate a Paper` CTA now launches a seeded live demo automatically: reset, first proposal, autoplay, and judged completion all happen without extra clicks.
|
| 18 |
- The replication setup card now performs a backend health check up front and surfaces a concrete startup command instead of the opaque browser-level `Failed to fetch` message when the API server is down.
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
- The dashboard `Replicate a Paper` CTA now launches a seeded live demo automatically: reset, first proposal, autoplay, and judged completion all happen without extra clicks.
|
| 18 |
- The replication setup card now performs a backend health check up front and surfaces a concrete startup command instead of the opaque browser-level `Failed to fetch` message when the API server is down.
|
| 19 |
|
| 20 |
+
## 2026-03-08 three-outcome live demo
|
| 21 |
+
|
| 22 |
+
- The live demo now has three seeded story modes on the dashboard: `fast-agreement`, `learning-opportunity`, and `no-agreement`.
|
| 23 |
+
- Each mode runs against the real backend with deterministic episode data and renders a post-episode results report instead of stopping at a generic terminal state.
|
| 24 |
+
- The results report now shows executed rounds, disagreement count, replicability score, paper reliability quality, reward and score charts, training interpretation, and next-tool suggestions.
|
| 25 |
+
- Verified backend-driven outputs for the current seeded ML demo cases:
|
| 26 |
+
- `fast-agreement` -> round `2`, verdict `accept`, cumulative reward `2.906845`
|
| 27 |
+
- `learning-opportunity` -> round `6`, verdict `accept`, cumulative reward `4.537097`
|
| 28 |
+
- `no-agreement` -> round `6`, verdict `timeout`, cumulative reward `0.366529`
|
| 29 |
+
|
| 30 |
+
## 2026-03-08 training page with real artifacts
|
| 31 |
+
|
| 32 |
+
- Added a dedicated `/training` page instead of relying on the old packaged dashboard card.
|
| 33 |
+
- The new page is backed by real artifact values from the existing outputs:
|
| 34 |
+
- local deterministic baseline summary
|
| 35 |
+
- live ART/OpenEnv scientist checkpoints
|
| 36 |
+
- seeded hold-out compare summary
|
| 37 |
+
- scientist and lab-manager preview summaries
|
| 38 |
+
- The training story is now explicit and honest:
|
| 39 |
+
- the training pipeline works
|
| 40 |
+
- live reward moved positive by later checkpoints
|
| 41 |
+
- hold-out compare still shows the trained Scientist underperforming baseline
|
| 42 |
+
- more training and parser/invalid-action cleanup are still needed
|
| 43 |
+
- Header nav now includes `Training`, dashboard training CTA points there, and the dashboard training teaser uses the same artifact-backed data.
|
| 44 |
+
|
frontend/src/App.tsx
CHANGED
|
@@ -2,6 +2,7 @@ import { Routes, Route } from 'react-router-dom';
|
|
| 2 |
import EpisodePage from '@/pages/EpisodePage';
|
| 3 |
import DashboardPage from '@/pages/DashboardPage';
|
| 4 |
import ComparePage from '@/pages/ComparePage';
|
|
|
|
| 5 |
import Header from '@/components/Header';
|
| 6 |
import { ToastProvider } from '@/components/Toast';
|
| 7 |
import Onboarding, { useOnboarding } from '@/components/Onboarding';
|
|
@@ -17,6 +18,7 @@ export default function App() {
|
|
| 17 |
<Route path="/" element={<DashboardPage />} />
|
| 18 |
<Route path="/episode" element={<EpisodePage />} />
|
| 19 |
<Route path="/episode/:episodeId" element={<EpisodePage />} />
|
|
|
|
| 20 |
<Route path="/compare" element={<ComparePage />} />
|
| 21 |
</Routes>
|
| 22 |
<Onboarding show={showOnboarding} onDismiss={dismissOnboarding} />
|
|
|
|
| 2 |
import EpisodePage from '@/pages/EpisodePage';
|
| 3 |
import DashboardPage from '@/pages/DashboardPage';
|
| 4 |
import ComparePage from '@/pages/ComparePage';
|
| 5 |
+
import TrainingPage from '@/pages/TrainingPage';
|
| 6 |
import Header from '@/components/Header';
|
| 7 |
import { ToastProvider } from '@/components/Toast';
|
| 8 |
import Onboarding, { useOnboarding } from '@/components/Onboarding';
|
|
|
|
| 18 |
<Route path="/" element={<DashboardPage />} />
|
| 19 |
<Route path="/episode" element={<EpisodePage />} />
|
| 20 |
<Route path="/episode/:episodeId" element={<EpisodePage />} />
|
| 21 |
+
<Route path="/training" element={<TrainingPage />} />
|
| 22 |
<Route path="/compare" element={<ComparePage />} />
|
| 23 |
</Routes>
|
| 24 |
<Onboarding show={showOnboarding} onDismiss={dismissOnboarding} />
|
frontend/src/components/EpisodeComparison.tsx
CHANGED
|
@@ -41,7 +41,7 @@ export default function EpisodeComparison({ className }: { className?: string })
|
|
| 41 |
// Auto-play through all rounds
|
| 42 |
while (!state.done) {
|
| 43 |
const isLastRound = state.round >= state.max_rounds - 1;
|
| 44 |
-
const action = isLastRound ? buildAcceptAction() : buildDefaultScientistAction();
|
| 45 |
state = await stepEpisode(state.session_id, action, state);
|
| 46 |
}
|
| 47 |
|
|
|
|
| 41 |
// Auto-play through all rounds
|
| 42 |
while (!state.done) {
|
| 43 |
const isLastRound = state.round >= state.max_rounds - 1;
|
| 44 |
+
const action = isLastRound ? buildAcceptAction() : buildDefaultScientistAction(state);
|
| 45 |
state = await stepEpisode(state.session_id, action, state);
|
| 46 |
}
|
| 47 |
|
frontend/src/components/EpisodeResultsReport.tsx
ADDED
|
@@ -0,0 +1,465 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import type { ReactNode } from 'react';
|
| 2 |
+
import {
|
| 3 |
+
CheckCircle2,
|
| 4 |
+
AlertTriangle,
|
| 5 |
+
XCircle,
|
| 6 |
+
BrainCircuit,
|
| 7 |
+
TrendingUp,
|
| 8 |
+
Wrench,
|
| 9 |
+
Gauge,
|
| 10 |
+
FileCheck2,
|
| 11 |
+
} from 'lucide-react';
|
| 12 |
+
import {
|
| 13 |
+
ResponsiveContainer,
|
| 14 |
+
LineChart,
|
| 15 |
+
Line,
|
| 16 |
+
CartesianGrid,
|
| 17 |
+
XAxis,
|
| 18 |
+
YAxis,
|
| 19 |
+
Tooltip,
|
| 20 |
+
Legend,
|
| 21 |
+
BarChart,
|
| 22 |
+
Bar,
|
| 23 |
+
} from 'recharts';
|
| 24 |
+
import type { EpisodeState } from '@/types';
|
| 25 |
+
import { cn, formatReward } from '@/lib/utils';
|
| 26 |
+
|
| 27 |
+
interface EpisodeResultsReportProps {
|
| 28 |
+
episode: EpisodeState;
|
| 29 |
+
className?: string;
|
| 30 |
+
}
|
| 31 |
+
|
| 32 |
+
type OutcomeProfile = {
|
| 33 |
+
tone: 'good' | 'learning' | 'reject';
|
| 34 |
+
title: string;
|
| 35 |
+
subtitle: string;
|
| 36 |
+
badge: string;
|
| 37 |
+
icon: typeof CheckCircle2;
|
| 38 |
+
};
|
| 39 |
+
|
| 40 |
+
const TOOL_STACK: Record<string, { title: string; packages: string[]; commands: string[] }> = {
|
| 41 |
+
math_reasoning: {
|
| 42 |
+
title: 'Suggested proof-and-verification stack',
|
| 43 |
+
packages: ['SymPy', 'pytest', 'JupyterLab', 'LaTeX tooling'],
|
| 44 |
+
commands: [
|
| 45 |
+
'pip install sympy pytest jupyterlab',
|
| 46 |
+
'Install TeX distribution for reproducible proof write-ups',
|
| 47 |
+
],
|
| 48 |
+
},
|
| 49 |
+
ml_benchmark: {
|
| 50 |
+
title: 'Suggested replication toolchain',
|
| 51 |
+
packages: ['PyTorch', 'torchvision', 'Weights & Biases', 'datasets'],
|
| 52 |
+
commands: [
|
| 53 |
+
'pip install torch torchvision wandb datasets',
|
| 54 |
+
'Install NVIDIA CUDA toolkit / drivers that match the available GPU node',
|
| 55 |
+
],
|
| 56 |
+
},
|
| 57 |
+
finance_trading: {
|
| 58 |
+
title: 'Suggested backtest stack',
|
| 59 |
+
packages: ['pandas', 'numpy', 'vectorbt', 'matplotlib'],
|
| 60 |
+
commands: [
|
| 61 |
+
'pip install pandas numpy vectorbt matplotlib',
|
| 62 |
+
'Add a versioned market-data export before rerunning the benchmark',
|
| 63 |
+
],
|
| 64 |
+
},
|
| 65 |
+
};
|
| 66 |
+
|
| 67 |
+
function computeConfidencePercent(episode: EpisodeState): number {
|
| 68 |
+
const scores = episode.scores;
|
| 69 |
+
if (!scores) {
|
| 70 |
+
return 0;
|
| 71 |
+
}
|
| 72 |
+
|
| 73 |
+
const weighted =
|
| 74 |
+
scores.rigor * 0.4 +
|
| 75 |
+
scores.feasibility * 0.35 +
|
| 76 |
+
scores.fidelity * 0.25;
|
| 77 |
+
return Math.round(weighted * 100);
|
| 78 |
+
}
|
| 79 |
+
|
| 80 |
+
function buildReliabilityLabel(episode: EpisodeState): string {
|
| 81 |
+
if (episode.judge_audit?.verdict === 'accept' && episode.step_history[0]?.lab_manager_action_type === 'accept') {
|
| 82 |
+
return 'Good';
|
| 83 |
+
}
|
| 84 |
+
if (episode.judge_audit?.verdict === 'accept') {
|
| 85 |
+
return 'Needs iteration';
|
| 86 |
+
}
|
| 87 |
+
return 'Bad';
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
function buildOutcomeProfile(episode: EpisodeState): OutcomeProfile {
|
| 91 |
+
const firstRoundAccepted =
|
| 92 |
+
episode.step_history[0]?.lab_manager_action_type === 'accept' &&
|
| 93 |
+
episode.judge_audit?.verdict === 'accept';
|
| 94 |
+
const disagreementRounds = episode.step_history.filter(
|
| 95 |
+
(trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
|
| 96 |
+
).length;
|
| 97 |
+
|
| 98 |
+
if (firstRoundAccepted) {
|
| 99 |
+
return {
|
| 100 |
+
tone: 'good',
|
| 101 |
+
title: 'Completed: First-round agreement',
|
| 102 |
+
subtitle: 'The paper looks replicable under the current constraints, and the agents converged immediately.',
|
| 103 |
+
badge: 'Good paper / strong replication candidate',
|
| 104 |
+
icon: CheckCircle2,
|
| 105 |
+
};
|
| 106 |
+
}
|
| 107 |
+
|
| 108 |
+
if (episode.judge_audit?.verdict === 'accept') {
|
| 109 |
+
return {
|
| 110 |
+
tone: 'learning',
|
| 111 |
+
title: 'Completed: Multi-round learning opportunity',
|
| 112 |
+
subtitle: `${disagreementRounds} disagreement round(s) were resolved into a feasible final protocol.`,
|
| 113 |
+
badge: 'Heavy RL learning opportunity',
|
| 114 |
+
icon: AlertTriangle,
|
| 115 |
+
};
|
| 116 |
+
}
|
| 117 |
+
|
| 118 |
+
return {
|
| 119 |
+
tone: 'reject',
|
| 120 |
+
title: 'Completed: No agreement reached',
|
| 121 |
+
subtitle: 'The protocol never became acceptable within the six-round limit, so the paper is rejected for this lab setup.',
|
| 122 |
+
badge: 'Cannot replicate under current setup',
|
| 123 |
+
icon: XCircle,
|
| 124 |
+
};
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
function aggregateRewardComponents(episode: EpisodeState): Array<{ key: string; value: number }> {
|
| 128 |
+
const totals = new Map<string, number>();
|
| 129 |
+
for (const trace of episode.step_history) {
|
| 130 |
+
for (const [key, value] of Object.entries(trace.step_reward_components)) {
|
| 131 |
+
totals.set(key, (totals.get(key) ?? 0) + value);
|
| 132 |
+
}
|
| 133 |
+
}
|
| 134 |
+
|
| 135 |
+
return [...totals.entries()]
|
| 136 |
+
.map(([key, value]) => ({ key, value: Number(value.toFixed(3)) }))
|
| 137 |
+
.sort((a, b) => Math.abs(b.value) - Math.abs(a.value));
|
| 138 |
+
}
|
| 139 |
+
|
| 140 |
+
function buildLearningBullets(episode: EpisodeState): string[] {
|
| 141 |
+
const firstProtocol = episode.step_history.find((trace) => trace.protocol)?.protocol;
|
| 142 |
+
const finalProtocol = episode.protocol;
|
| 143 |
+
const bullets: string[] = [];
|
| 144 |
+
|
| 145 |
+
if (firstProtocol && finalProtocol && firstProtocol.sample_size !== finalProtocol.sample_size) {
|
| 146 |
+
bullets.push(
|
| 147 |
+
`Sample size moved from ${firstProtocol.sample_size} to ${finalProtocol.sample_size}, showing explicit adaptation to budget or staff feedback.`,
|
| 148 |
+
);
|
| 149 |
+
}
|
| 150 |
+
|
| 151 |
+
if (firstProtocol && finalProtocol && firstProtocol.controls.length !== finalProtocol.controls.length) {
|
| 152 |
+
bullets.push(
|
| 153 |
+
`Controls were narrowed from ${firstProtocol.controls.length} to ${finalProtocol.controls.length}, which reduced lab pressure while preserving the core benchmark check.`,
|
| 154 |
+
);
|
| 155 |
+
}
|
| 156 |
+
|
| 157 |
+
if (episode.step_history.some((trace) => trace.lab_manager_action_type === 'suggest_alternative')) {
|
| 158 |
+
bullets.push('The Scientist absorbed grounded Lab Manager alternatives instead of ignoring the resource constraints.');
|
| 159 |
+
}
|
| 160 |
+
|
| 161 |
+
if (episode.step_history.some((trace) => trace.step_reward_components.momentum_bonus)) {
|
| 162 |
+
bullets.push('Later revisions earned momentum bonuses, meaning the protocol improved rather than stalling.');
|
| 163 |
+
}
|
| 164 |
+
|
| 165 |
+
if (!bullets.length) {
|
| 166 |
+
bullets.push('The episode converged quickly, so the strongest training value is as a positive exemplar for future rollouts.');
|
| 167 |
+
}
|
| 168 |
+
|
| 169 |
+
return bullets;
|
| 170 |
+
}
|
| 171 |
+
|
| 172 |
+
function buildImprovementBullets(episode: EpisodeState): string[] {
|
| 173 |
+
const scores = episode.scores;
|
| 174 |
+
const verdict = episode.judge_audit?.verdict ?? 'unknown';
|
| 175 |
+
const suggestions: string[] = [];
|
| 176 |
+
|
| 177 |
+
if (!scores) {
|
| 178 |
+
suggestions.push('Provision more compute, staff, or schedule slack before attempting another replication run.');
|
| 179 |
+
suggestions.push('Reduce sample size and control scope earlier to avoid spending the full round budget on infeasible plans.');
|
| 180 |
+
return suggestions;
|
| 181 |
+
}
|
| 182 |
+
|
| 183 |
+
if (scores.rigor < 0.7) {
|
| 184 |
+
suggestions.push('Improve rigor by tightening success criteria, controls, and evaluation checkpoints in the proposed protocol.');
|
| 185 |
+
}
|
| 186 |
+
if (scores.feasibility < 0.9) {
|
| 187 |
+
suggestions.push('Improve feasibility by reducing workload earlier or aligning the protocol more closely with the available staff and equipment.');
|
| 188 |
+
}
|
| 189 |
+
if (scores.fidelity < 0.7) {
|
| 190 |
+
suggestions.push('Improve fidelity by matching the original method and baseline recipe more closely.');
|
| 191 |
+
}
|
| 192 |
+
if (verdict !== 'accept') {
|
| 193 |
+
suggestions.push('Use this rollout as a negative exemplar in RL training so the Scientist learns to avoid repeated non-feasible plans.');
|
| 194 |
+
}
|
| 195 |
+
if (!suggestions.length) {
|
| 196 |
+
suggestions.push('This is a strong positive exemplar; the next improvement is broader held-out evaluation rather than protocol repair.');
|
| 197 |
+
}
|
| 198 |
+
|
| 199 |
+
return suggestions;
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
export default function EpisodeResultsReport({ episode, className }: EpisodeResultsReportProps) {
|
| 203 |
+
const profile = buildOutcomeProfile(episode);
|
| 204 |
+
const confidencePercent = computeConfidencePercent(episode);
|
| 205 |
+
const reliabilityLabel = buildReliabilityLabel(episode);
|
| 206 |
+
const disagreementRounds = episode.step_history.filter(
|
| 207 |
+
(trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
|
| 208 |
+
).length;
|
| 209 |
+
const chartData = episode.step_history.map((trace) => ({
|
| 210 |
+
round: `R${trace.round}`,
|
| 211 |
+
reward: Number(trace.reward.toFixed(3)),
|
| 212 |
+
cumulative: Number(trace.cumulative_reward.toFixed(3)),
|
| 213 |
+
}));
|
| 214 |
+
const scoreBars = [
|
| 215 |
+
{ key: 'rigor', value: episode.scores?.rigor ?? 0 },
|
| 216 |
+
{ key: 'feasibility', value: episode.scores?.feasibility ?? 0 },
|
| 217 |
+
{ key: 'fidelity', value: episode.scores?.fidelity ?? 0 },
|
| 218 |
+
{ key: 'parsimony', value: episode.scores?.parsimony ?? 0 },
|
| 219 |
+
];
|
| 220 |
+
const rewardComponents = aggregateRewardComponents(episode);
|
| 221 |
+
const toolStack = TOOL_STACK[episode.template];
|
| 222 |
+
const ProfileIcon = profile.icon;
|
| 223 |
+
|
| 224 |
+
return (
|
| 225 |
+
<section className={cn('rounded-2xl border border-border bg-card p-6', className)}>
|
| 226 |
+
<div className="flex flex-col gap-4 lg:flex-row lg:items-start lg:justify-between">
|
| 227 |
+
<div>
|
| 228 |
+
<div
|
| 229 |
+
className={cn(
|
| 230 |
+
'mb-3 inline-flex items-center gap-2 rounded-full px-3 py-1 text-xs font-semibold',
|
| 231 |
+
profile.tone === 'good' && 'bg-lab-manager/10 text-lab-manager',
|
| 232 |
+
profile.tone === 'learning' && 'bg-judge/10 text-judge',
|
| 233 |
+
profile.tone === 'reject' && 'bg-destructive/10 text-destructive',
|
| 234 |
+
)}
|
| 235 |
+
>
|
| 236 |
+
<ProfileIcon className="h-4 w-4" />
|
| 237 |
+
{profile.badge}
|
| 238 |
+
</div>
|
| 239 |
+
<h2 className="text-2xl font-bold">{profile.title}</h2>
|
| 240 |
+
<p className="mt-2 max-w-3xl text-sm text-muted-foreground">{profile.subtitle}</p>
|
| 241 |
+
<p className="mt-3 max-w-3xl text-sm">
|
| 242 |
+
{profile.tone !== 'reject'
|
| 243 |
+
? `ReplicaLab scores this paper as replicable in the current lab setup with ${confidencePercent}% confidence.`
|
| 244 |
+
: `ReplicaLab rejects this paper for the current setup. The paper reliability score is ${confidencePercent}% until the blocking constraints are addressed.`}
|
| 245 |
+
</p>
|
| 246 |
+
</div>
|
| 247 |
+
|
| 248 |
+
<div className="grid gap-3 sm:grid-cols-2 lg:w-[360px]">
|
| 249 |
+
<MetricCard
|
| 250 |
+
icon={<Gauge className="h-4 w-4" />}
|
| 251 |
+
label="Replicability score"
|
| 252 |
+
value={`${confidencePercent}%`}
|
| 253 |
+
hint="Derived from rigor, feasibility, and fidelity."
|
| 254 |
+
/>
|
| 255 |
+
<MetricCard
|
| 256 |
+
icon={<CheckCircle2 className="h-4 w-4" />}
|
| 257 |
+
label="Paper reliability quality"
|
| 258 |
+
value={reliabilityLabel}
|
| 259 |
+
hint="Good, learning opportunity, or bad based on the judged outcome."
|
| 260 |
+
/>
|
| 261 |
+
<MetricCard
|
| 262 |
+
icon={<FileCheck2 className="h-4 w-4" />}
|
| 263 |
+
label="Judge verdict"
|
| 264 |
+
value={episode.judge_audit?.verdict ?? 'unknown'}
|
| 265 |
+
hint="Canonical deterministic decision."
|
| 266 |
+
/>
|
| 267 |
+
<MetricCard
|
| 268 |
+
icon={<TrendingUp className="h-4 w-4" />}
|
| 269 |
+
label="Total reward"
|
| 270 |
+
value={formatReward(episode.cumulative_reward)}
|
| 271 |
+
hint={`Cumulative reward across ${episode.step_history.length} executed round(s).`}
|
| 272 |
+
/>
|
| 273 |
+
</div>
|
| 274 |
+
</div>
|
| 275 |
+
|
| 276 |
+
<div className="mt-4 rounded-xl border border-border bg-muted/30 p-4">
|
| 277 |
+
<div className="flex flex-wrap items-center gap-2 text-xs font-medium">
|
| 278 |
+
<span className="rounded-full bg-background px-2.5 py-1 text-primary">
|
| 279 |
+
Executed rounds: {episode.step_history.length} / {episode.max_rounds}
|
| 280 |
+
</span>
|
| 281 |
+
<span className="rounded-full bg-background px-2.5 py-1 text-primary">
|
| 282 |
+
Disagreement rounds: {disagreementRounds}
|
| 283 |
+
</span>
|
| 284 |
+
<span className="rounded-full bg-background px-2.5 py-1 text-primary">
|
| 285 |
+
Final status: {episode.done ? 'completed' : 'in progress'}
|
| 286 |
+
</span>
|
| 287 |
+
</div>
|
| 288 |
+
</div>
|
| 289 |
+
|
| 290 |
+
<div className="mt-6 grid gap-4 xl:grid-cols-2">
|
| 291 |
+
<div className="rounded-xl border border-border bg-muted/30 p-4">
|
| 292 |
+
<h3 className="mb-2 text-sm font-semibold">Round-by-round reward trajectory</h3>
|
| 293 |
+
<p className="mb-4 text-xs text-muted-foreground">
|
| 294 |
+
This is real episode data from the run you just watched, not static demo text.
|
| 295 |
+
</p>
|
| 296 |
+
<div className="h-60">
|
| 297 |
+
<ResponsiveContainer width="100%" height="100%">
|
| 298 |
+
<LineChart data={chartData}>
|
| 299 |
+
<CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
|
| 300 |
+
<XAxis dataKey="round" tick={{ fontSize: 11 }} />
|
| 301 |
+
<YAxis tick={{ fontSize: 11 }} />
|
| 302 |
+
<Tooltip />
|
| 303 |
+
<Legend wrapperStyle={{ fontSize: '12px' }} />
|
| 304 |
+
<Line type="monotone" dataKey="reward" stroke="var(--color-primary)" strokeWidth={2.5} name="Step reward" />
|
| 305 |
+
<Line type="monotone" dataKey="cumulative" stroke="var(--color-lab-manager)" strokeWidth={2.5} name="Cumulative reward" />
|
| 306 |
+
</LineChart>
|
| 307 |
+
</ResponsiveContainer>
|
| 308 |
+
</div>
|
| 309 |
+
</div>
|
| 310 |
+
|
| 311 |
+
<div className="rounded-xl border border-border bg-muted/30 p-4">
|
| 312 |
+
<h3 className="mb-2 text-sm font-semibold">Terminal score profile</h3>
|
| 313 |
+
<p className="mb-4 text-xs text-muted-foreground">
|
| 314 |
+
The final verdict depends on these component scores plus the terminal rubric.
|
| 315 |
+
</p>
|
| 316 |
+
<div className="h-60">
|
| 317 |
+
<ResponsiveContainer width="100%" height="100%">
|
| 318 |
+
<BarChart data={scoreBars}>
|
| 319 |
+
<CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
|
| 320 |
+
<XAxis dataKey="key" tick={{ fontSize: 11 }} />
|
| 321 |
+
<YAxis domain={[0, 1]} tickFormatter={(value) => `${Math.round(value * 100)}%`} tick={{ fontSize: 11 }} />
|
| 322 |
+
<Tooltip />
|
| 323 |
+
<Bar dataKey="value" fill="var(--color-judge)" radius={[6, 6, 0, 0]} />
|
| 324 |
+
</BarChart>
|
| 325 |
+
</ResponsiveContainer>
|
| 326 |
+
</div>
|
| 327 |
+
</div>
|
| 328 |
+
</div>
|
| 329 |
+
|
| 330 |
+
<div className="mt-6 grid gap-4 xl:grid-cols-3">
|
| 331 |
+
<InsightCard
|
| 332 |
+
title="What the agents learned"
|
| 333 |
+
icon={<BrainCircuit className="h-4 w-4" />}
|
| 334 |
+
items={buildLearningBullets(episode)}
|
| 335 |
+
/>
|
| 336 |
+
<InsightCard
|
| 337 |
+
title="What to improve next"
|
| 338 |
+
icon={<TrendingUp className="h-4 w-4" />}
|
| 339 |
+
items={buildImprovementBullets(episode)}
|
| 340 |
+
/>
|
| 341 |
+
<InsightCard
|
| 342 |
+
title="Next tools to install"
|
| 343 |
+
icon={<Wrench className="h-4 w-4" />}
|
| 344 |
+
items={[
|
| 345 |
+
toolStack.title,
|
| 346 |
+
...toolStack.packages.map((item) => `Install ${item}`),
|
| 347 |
+
...toolStack.commands,
|
| 348 |
+
]}
|
| 349 |
+
/>
|
| 350 |
+
</div>
|
| 351 |
+
|
| 352 |
+
<div className="mt-6 grid gap-4 xl:grid-cols-[1.5fr_1fr]">
|
| 353 |
+
<div className="rounded-xl border border-border bg-muted/30 p-4">
|
| 354 |
+
<h3 className="mb-3 text-sm font-semibold">Episode timeline</h3>
|
| 355 |
+
<div className="space-y-3">
|
| 356 |
+
{episode.step_history.map((trace) => (
|
| 357 |
+
<div key={trace.round} className="rounded-lg border border-border bg-background p-3">
|
| 358 |
+
<div className="flex flex-wrap items-center gap-2">
|
| 359 |
+
<span className="rounded-full bg-primary/10 px-2 py-0.5 text-[11px] font-semibold text-primary">
|
| 360 |
+
Round {trace.round}
|
| 361 |
+
</span>
|
| 362 |
+
<span className="rounded-full bg-muted px-2 py-0.5 text-[11px] text-muted-foreground">
|
| 363 |
+
Scientist: {trace.action_type.replace(/_/g, ' ')}
|
| 364 |
+
</span>
|
| 365 |
+
{trace.lab_manager_action_type && (
|
| 366 |
+
<span className="rounded-full bg-muted px-2 py-0.5 text-[11px] text-muted-foreground">
|
| 367 |
+
Lab Manager: {trace.lab_manager_action_type.replace(/_/g, ' ')}
|
| 368 |
+
</span>
|
| 369 |
+
)}
|
| 370 |
+
<span className="ml-auto text-xs font-semibold text-foreground">
|
| 371 |
+
{formatReward(trace.reward)}
|
| 372 |
+
</span>
|
| 373 |
+
</div>
|
| 374 |
+
<p className="mt-2 text-xs text-muted-foreground">{trace.scientist_message}</p>
|
| 375 |
+
{trace.lab_manager_message && (
|
| 376 |
+
<p className="mt-1 text-xs text-muted-foreground">{trace.lab_manager_message}</p>
|
| 377 |
+
)}
|
| 378 |
+
</div>
|
| 379 |
+
))}
|
| 380 |
+
</div>
|
| 381 |
+
</div>
|
| 382 |
+
|
| 383 |
+
<div className="rounded-xl border border-border bg-muted/30 p-4">
|
| 384 |
+
<h3 className="mb-3 text-sm font-semibold">Reward component ledger</h3>
|
| 385 |
+
<div className="space-y-2">
|
| 386 |
+
{rewardComponents.length ? rewardComponents.map((entry) => (
|
| 387 |
+
<div key={entry.key} className="flex items-center justify-between rounded-lg border border-border bg-background px-3 py-2 text-xs">
|
| 388 |
+
<span className="capitalize text-muted-foreground">{entry.key.replace(/_/g, ' ')}</span>
|
| 389 |
+
<span className={cn('font-semibold', entry.value >= 0 ? 'text-lab-manager' : 'text-destructive')}>
|
| 390 |
+
{entry.value >= 0 ? '+' : ''}
|
| 391 |
+
{entry.value.toFixed(3)}
|
| 392 |
+
</span>
|
| 393 |
+
</div>
|
| 394 |
+
)) : (
|
| 395 |
+
<div className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
|
| 396 |
+
No intermediate reward components were emitted in this episode.
|
| 397 |
+
</div>
|
| 398 |
+
)}
|
| 399 |
+
</div>
|
| 400 |
+
|
| 401 |
+
<div className="mt-4 rounded-lg border border-border bg-background p-3">
|
| 402 |
+
<div className="mb-2 text-xs font-semibold">Training interpretation</div>
|
| 403 |
+
<p className="text-xs text-muted-foreground">
|
| 404 |
+
{profile.tone === 'good' &&
|
| 405 |
+
'Use this run as a positive exemplar: the policy found a feasible replication plan with minimal negotiation overhead.'}
|
| 406 |
+
{profile.tone === 'learning' &&
|
| 407 |
+
'This is the richest RL case: multiple disagreement rounds created dense reward signals for the Scientist to learn from.'}
|
| 408 |
+
{profile.tone === 'reject' &&
|
| 409 |
+
'This is a strong negative exemplar: the policy spent all six rounds without resolving the key feasibility blockers.'}
|
| 410 |
+
</p>
|
| 411 |
+
</div>
|
| 412 |
+
</div>
|
| 413 |
+
</div>
|
| 414 |
+
</section>
|
| 415 |
+
);
|
| 416 |
+
}
|
| 417 |
+
|
| 418 |
+
function MetricCard({
|
| 419 |
+
icon,
|
| 420 |
+
label,
|
| 421 |
+
value,
|
| 422 |
+
hint,
|
| 423 |
+
}: {
|
| 424 |
+
icon: ReactNode;
|
| 425 |
+
label: string;
|
| 426 |
+
value: string;
|
| 427 |
+
hint: string;
|
| 428 |
+
}) {
|
| 429 |
+
return (
|
| 430 |
+
<div className="rounded-xl border border-border bg-muted/30 p-3">
|
| 431 |
+
<div className="mb-2 flex items-center gap-2 text-xs text-muted-foreground">
|
| 432 |
+
{icon}
|
| 433 |
+
{label}
|
| 434 |
+
</div>
|
| 435 |
+
<div className="text-lg font-bold">{value}</div>
|
| 436 |
+
<p className="mt-1 text-[11px] text-muted-foreground">{hint}</p>
|
| 437 |
+
</div>
|
| 438 |
+
);
|
| 439 |
+
}
|
| 440 |
+
|
| 441 |
+
function InsightCard({
|
| 442 |
+
title,
|
| 443 |
+
icon,
|
| 444 |
+
items,
|
| 445 |
+
}: {
|
| 446 |
+
title: string;
|
| 447 |
+
icon: ReactNode;
|
| 448 |
+
items: string[];
|
| 449 |
+
}) {
|
| 450 |
+
return (
|
| 451 |
+
<div className="rounded-xl border border-border bg-muted/30 p-4">
|
| 452 |
+
<div className="mb-3 flex items-center gap-2 text-sm font-semibold">
|
| 453 |
+
{icon}
|
| 454 |
+
{title}
|
| 455 |
+
</div>
|
| 456 |
+
<div className="space-y-2">
|
| 457 |
+
{items.map((item) => (
|
| 458 |
+
<div key={item} className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
|
| 459 |
+
{item}
|
| 460 |
+
</div>
|
| 461 |
+
))}
|
| 462 |
+
</div>
|
| 463 |
+
</div>
|
| 464 |
+
);
|
| 465 |
+
}
|
frontend/src/components/Header.tsx
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
import { useState } from 'react';
|
| 2 |
import { Link, useLocation } from 'react-router-dom';
|
| 3 |
-
import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows } from 'lucide-react';
|
| 4 |
import { cn } from '@/lib/utils';
|
| 5 |
import { useTheme } from '@/lib/useTheme';
|
| 6 |
import { toggleMute, isMuted } from '@/lib/audio';
|
|
@@ -8,6 +8,7 @@ import { toggleMute, isMuted } from '@/lib/audio';
|
|
| 8 |
const navItems = [
|
| 9 |
{ to: '/', label: 'Dashboard', icon: LayoutDashboard },
|
| 10 |
{ to: '/episode', label: 'Episode', icon: Play },
|
|
|
|
| 11 |
{ to: '/compare', label: 'Compare', icon: GitCompareArrows },
|
| 12 |
];
|
| 13 |
|
|
|
|
| 1 |
import { useState } from 'react';
|
| 2 |
import { Link, useLocation } from 'react-router-dom';
|
| 3 |
+
import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows, GraduationCap } from 'lucide-react';
|
| 4 |
import { cn } from '@/lib/utils';
|
| 5 |
import { useTheme } from '@/lib/useTheme';
|
| 6 |
import { toggleMute, isMuted } from '@/lib/audio';
|
|
|
|
| 8 |
const navItems = [
|
| 9 |
{ to: '/', label: 'Dashboard', icon: LayoutDashboard },
|
| 10 |
{ to: '/episode', label: 'Episode', icon: Play },
|
| 11 |
+
{ to: '/training', label: 'Training', icon: GraduationCap },
|
| 12 |
{ to: '/compare', label: 'Compare', icon: GitCompareArrows },
|
| 13 |
];
|
| 14 |
|
frontend/src/components/TrainingResults.tsx
CHANGED
|
@@ -20,6 +20,7 @@ import {
|
|
| 20 |
Legend,
|
| 21 |
} from 'recharts';
|
| 22 |
import type { TrainingComparison } from '@/types';
|
|
|
|
| 23 |
import { cn, formatReward, formatScore } from '@/lib/utils';
|
| 24 |
|
| 25 |
interface TrainingResultsProps {
|
|
@@ -27,49 +28,8 @@ interface TrainingResultsProps {
|
|
| 27 |
className?: string;
|
| 28 |
}
|
| 29 |
|
| 30 |
-
const DEMO_COMPARISON: TrainingComparison = {
|
| 31 |
-
baseline: [
|
| 32 |
-
{ episode: 1, reward: 3.8, rigor: 0.58, feasibility: 0.46, fidelity: 0.55, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 33 |
-
{ episode: 2, reward: 4.0, rigor: 0.6, feasibility: 0.48, fidelity: 0.56, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 34 |
-
{ episode: 3, reward: 4.1, rigor: 0.61, feasibility: 0.5, fidelity: 0.57, rounds_used: 5, agreement: true, invalid_actions: 1 },
|
| 35 |
-
{ episode: 4, reward: 4.2, rigor: 0.62, feasibility: 0.5, fidelity: 0.58, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 36 |
-
{ episode: 5, reward: 4.0, rigor: 0.6, feasibility: 0.49, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 37 |
-
{ episode: 6, reward: 4.3, rigor: 0.63, feasibility: 0.52, fidelity: 0.59, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 38 |
-
{ episode: 7, reward: 4.1, rigor: 0.61, feasibility: 0.5, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 39 |
-
{ episode: 8, reward: 4.4, rigor: 0.64, feasibility: 0.53, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 40 |
-
{ episode: 9, reward: 4.2, rigor: 0.62, feasibility: 0.51, fidelity: 0.59, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 41 |
-
{ episode: 10, reward: 4.0, rigor: 0.6, feasibility: 0.48, fidelity: 0.57, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 42 |
-
{ episode: 11, reward: 4.3, rigor: 0.64, feasibility: 0.52, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 43 |
-
{ episode: 12, reward: 4.1, rigor: 0.61, feasibility: 0.49, fidelity: 0.58, rounds_used: 5, agreement: false, invalid_actions: 2 },
|
| 44 |
-
],
|
| 45 |
-
trained: [
|
| 46 |
-
{ episode: 1, reward: 4.5, rigor: 0.64, feasibility: 0.56, fidelity: 0.6, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 47 |
-
{ episode: 2, reward: 4.8, rigor: 0.67, feasibility: 0.58, fidelity: 0.62, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 48 |
-
{ episode: 3, reward: 5.0, rigor: 0.69, feasibility: 0.61, fidelity: 0.64, rounds_used: 4, agreement: true, invalid_actions: 1 },
|
| 49 |
-
{ episode: 4, reward: 5.2, rigor: 0.71, feasibility: 0.63, fidelity: 0.66, rounds_used: 3, agreement: true, invalid_actions: 1 },
|
| 50 |
-
{ episode: 5, reward: 5.5, rigor: 0.73, feasibility: 0.66, fidelity: 0.68, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 51 |
-
{ episode: 6, reward: 5.7, rigor: 0.75, feasibility: 0.68, fidelity: 0.69, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 52 |
-
{ episode: 7, reward: 5.9, rigor: 0.76, feasibility: 0.7, fidelity: 0.7, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 53 |
-
{ episode: 8, reward: 6.1, rigor: 0.78, feasibility: 0.72, fidelity: 0.72, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 54 |
-
{ episode: 9, reward: 6.3, rigor: 0.79, feasibility: 0.74, fidelity: 0.73, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 55 |
-
{ episode: 10, reward: 6.5, rigor: 0.8, feasibility: 0.76, fidelity: 0.75, rounds_used: 3, agreement: true, invalid_actions: 0 },
|
| 56 |
-
{ episode: 11, reward: 6.6, rigor: 0.81, feasibility: 0.77, fidelity: 0.76, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 57 |
-
{ episode: 12, reward: 6.8, rigor: 0.83, feasibility: 0.79, fidelity: 0.78, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 58 |
-
],
|
| 59 |
-
summary: {
|
| 60 |
-
baseline_avg_reward: 4.13,
|
| 61 |
-
trained_avg_reward: 5.74,
|
| 62 |
-
baseline_agreement_rate: 0.5,
|
| 63 |
-
trained_agreement_rate: 0.92,
|
| 64 |
-
baseline_avg_rounds: 4.58,
|
| 65 |
-
trained_avg_rounds: 3.0,
|
| 66 |
-
baseline_invalid_rate: 0.15,
|
| 67 |
-
trained_invalid_rate: 0.03,
|
| 68 |
-
},
|
| 69 |
-
};
|
| 70 |
-
|
| 71 |
export default function TrainingResults({ data, className }: TrainingResultsProps) {
|
| 72 |
-
const comparison = data ??
|
| 73 |
const [showTrained, setShowTrained] = useState(true);
|
| 74 |
const chartData = comparison.baseline.map((baselinePoint, index) => ({
|
| 75 |
episode: baselinePoint.episode,
|
|
@@ -96,18 +56,21 @@ export default function TrainingResults({ data, className }: TrainingResultsProp
|
|
| 96 |
<span className="rounded-full bg-primary/10 px-2 py-1 text-[11px] font-medium text-primary">Minimal Colab notebook</span>
|
| 97 |
<span className="rounded-full bg-scientist/10 px-2 py-1 text-[11px] font-medium text-scientist">Unsloth + HF TRL</span>
|
| 98 |
<span className="rounded-full bg-judge/10 px-2 py-1 text-[11px] font-medium text-judge">Deterministic judge reward</span>
|
|
|
|
|
|
|
|
|
|
| 99 |
</div>
|
| 100 |
|
| 101 |
<p className="mb-4 text-sm text-muted-foreground">
|
| 102 |
-
|
| 103 |
-
|
| 104 |
</p>
|
| 105 |
|
| 106 |
<div className="mb-4 grid gap-2 sm:grid-cols-2 xl:grid-cols-4">
|
| 107 |
<StoryCard icon={<FileText className="h-3.5 w-3.5" />} title="Notebook" detail="`train_minimal_colab.ipynb` is the sponsor-facing minimal GRPO path." />
|
| 108 |
<StoryCard icon={<Cpu className="h-3.5 w-3.5" />} title="Trainer" detail="Unsloth + TRL update the Scientist LoRA adapter on seeded tasks." />
|
| 109 |
<StoryCard icon={<FlaskConical className="h-3.5 w-3.5" />} title="Environment" detail="Episodes come from the same paper-derived ReplicaLab benchmark families." />
|
| 110 |
-
<StoryCard icon={<Scale className="h-3.5 w-3.5" />} title="Reward" detail=
|
| 111 |
</div>
|
| 112 |
|
| 113 |
<div className="mb-4 h-48">
|
|
@@ -155,8 +118,8 @@ export default function TrainingResults({ data, className }: TrainingResultsProp
|
|
| 155 |
</div>
|
| 156 |
|
| 157 |
<p className="mt-4 text-xs text-muted-foreground">
|
| 158 |
-
|
| 159 |
-
the
|
| 160 |
</p>
|
| 161 |
</div>
|
| 162 |
);
|
|
|
|
| 20 |
Legend,
|
| 21 |
} from 'recharts';
|
| 22 |
import type { TrainingComparison } from '@/types';
|
| 23 |
+
import { HOLDOUT_COMPARE, LIVE_CHECKPOINTS, TRAINING_ASSESSMENT } from '@/data/trainingArtifacts';
|
| 24 |
import { cn, formatReward, formatScore } from '@/lib/utils';
|
| 25 |
|
| 26 |
interface TrainingResultsProps {
|
|
|
|
| 28 |
className?: string;
|
| 29 |
}
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
export default function TrainingResults({ data, className }: TrainingResultsProps) {
|
| 32 |
+
const comparison = data ?? HOLDOUT_COMPARE;
|
| 33 |
const [showTrained, setShowTrained] = useState(true);
|
| 34 |
const chartData = comparison.baseline.map((baselinePoint, index) => ({
|
| 35 |
episode: baselinePoint.episode,
|
|
|
|
| 56 |
<span className="rounded-full bg-primary/10 px-2 py-1 text-[11px] font-medium text-primary">Minimal Colab notebook</span>
|
| 57 |
<span className="rounded-full bg-scientist/10 px-2 py-1 text-[11px] font-medium text-scientist">Unsloth + HF TRL</span>
|
| 58 |
<span className="rounded-full bg-judge/10 px-2 py-1 text-[11px] font-medium text-judge">Deterministic judge reward</span>
|
| 59 |
+
<span className="rounded-full bg-destructive/10 px-2 py-1 text-[11px] font-medium text-destructive">
|
| 60 |
+
More training required
|
| 61 |
+
</span>
|
| 62 |
</div>
|
| 63 |
|
| 64 |
<p className="mb-4 text-sm text-muted-foreground">
|
| 65 |
+
This panel now uses the real hold-out compare artifact. The training stack ran successfully, but the current
|
| 66 |
+
trained Scientist still underperforms the deterministic baseline on fixed seeded evaluation.
|
| 67 |
</p>
|
| 68 |
|
| 69 |
<div className="mb-4 grid gap-2 sm:grid-cols-2 xl:grid-cols-4">
|
| 70 |
<StoryCard icon={<FileText className="h-3.5 w-3.5" />} title="Notebook" detail="`train_minimal_colab.ipynb` is the sponsor-facing minimal GRPO path." />
|
| 71 |
<StoryCard icon={<Cpu className="h-3.5 w-3.5" />} title="Trainer" detail="Unsloth + TRL update the Scientist LoRA adapter on seeded tasks." />
|
| 72 |
<StoryCard icon={<FlaskConical className="h-3.5 w-3.5" />} title="Environment" detail="Episodes come from the same paper-derived ReplicaLab benchmark families." />
|
| 73 |
+
<StoryCard icon={<Scale className="h-3.5 w-3.5" />} title="Reward" detail={`Latest live checkpoint reached ${formatReward(LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1].averageReward)} average reward at ${(LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1].agreementRate * 100).toFixed(0)}% agreement.`} />
|
| 74 |
</div>
|
| 75 |
|
| 76 |
<div className="mb-4 h-48">
|
|
|
|
| 118 |
</div>
|
| 119 |
|
| 120 |
<p className="mt-4 text-xs text-muted-foreground">
|
| 121 |
+
Current status: {TRAINING_ASSESSMENT.achieved[2]} The next gate is reducing invalid actions and rerunning
|
| 122 |
+
the hold-out compare until trained performance overtakes baseline.
|
| 123 |
</p>
|
| 124 |
</div>
|
| 125 |
);
|
frontend/src/data/trainingArtifacts.ts
ADDED
|
@@ -0,0 +1,210 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import type { TrainingComparison } from '@/types';
|
| 2 |
+
|
| 3 |
+
export interface TrainingCheckpoint {
|
| 4 |
+
label: string;
|
| 5 |
+
artifactStep: number;
|
| 6 |
+
averageReward: number;
|
| 7 |
+
agreementRate: number;
|
| 8 |
+
averageRounds: number;
|
| 9 |
+
rolloutsPerGroup: number;
|
| 10 |
+
scenarioCount: number;
|
| 11 |
+
finishedAt: string;
|
| 12 |
+
}
|
| 13 |
+
|
| 14 |
+
export interface TrainingLogRow {
|
| 15 |
+
label: string;
|
| 16 |
+
trainingStep: number;
|
| 17 |
+
seed: number;
|
| 18 |
+
paperTitle: string;
|
| 19 |
+
reward: number;
|
| 20 |
+
roundsUsed: number;
|
| 21 |
+
invalidActionCount: number;
|
| 22 |
+
parseErrorCount: number;
|
| 23 |
+
verdict: string | null;
|
| 24 |
+
note: string;
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
export interface PreviewArtifact {
|
| 28 |
+
runName: string;
|
| 29 |
+
modelName: string;
|
| 30 |
+
datasetSize: number;
|
| 31 |
+
evidencePackVersion: string;
|
| 32 |
+
config: {
|
| 33 |
+
maxSteps?: number;
|
| 34 |
+
numTrainEpochs?: number;
|
| 35 |
+
learningRate: number;
|
| 36 |
+
loraRank: number;
|
| 37 |
+
maxSeqLength: number;
|
| 38 |
+
templates: string[];
|
| 39 |
+
difficulties: string[];
|
| 40 |
+
};
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
export const HOLDOUT_COMPARE: TrainingComparison = {
|
| 44 |
+
baseline: [
|
| 45 |
+
{ episode: 1, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 46 |
+
{ episode: 2, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 47 |
+
{ episode: 3, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 48 |
+
{ episode: 4, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
|
| 49 |
+
],
|
| 50 |
+
trained: [
|
| 51 |
+
{ episode: 1, reward: -5.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
|
| 52 |
+
{ episode: 2, reward: -4.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
|
| 53 |
+
{ episode: 3, reward: -5.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
|
| 54 |
+
{ episode: 4, reward: -4.0, rigor: 0.0, feasibility: 0.0, fidelity: 0.0, rounds_used: 20, agreement: false, invalid_actions: 20 },
|
| 55 |
+
],
|
| 56 |
+
summary: {
|
| 57 |
+
baseline_avg_reward: 4.925,
|
| 58 |
+
trained_avg_reward: -4.5,
|
| 59 |
+
baseline_agreement_rate: 1.0,
|
| 60 |
+
trained_agreement_rate: 0.0,
|
| 61 |
+
baseline_avg_rounds: 2.0,
|
| 62 |
+
trained_avg_rounds: 20.0,
|
| 63 |
+
baseline_invalid_rate: 0.0,
|
| 64 |
+
trained_invalid_rate: 1.0,
|
| 65 |
+
},
|
| 66 |
+
};
|
| 67 |
+
|
| 68 |
+
export const LIVE_CHECKPOINTS: TrainingCheckpoint[] = [
|
| 69 |
+
{
|
| 70 |
+
label: 'Checkpoint 1',
|
| 71 |
+
artifactStep: 1,
|
| 72 |
+
averageReward: -1.0,
|
| 73 |
+
agreementRate: 0.0,
|
| 74 |
+
averageRounds: 1.75,
|
| 75 |
+
rolloutsPerGroup: 2,
|
| 76 |
+
scenarioCount: 2,
|
| 77 |
+
finishedAt: '2026-03-08T17:55:41.971589+00:00',
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
label: 'Checkpoint 2',
|
| 81 |
+
artifactStep: 2,
|
| 82 |
+
averageReward: 0.387202,
|
| 83 |
+
agreementRate: 0.25,
|
| 84 |
+
averageRounds: 3.0,
|
| 85 |
+
rolloutsPerGroup: 2,
|
| 86 |
+
scenarioCount: 2,
|
| 87 |
+
finishedAt: '2026-03-08T17:59:47.820056+00:00',
|
| 88 |
+
},
|
| 89 |
+
{
|
| 90 |
+
label: 'Checkpoint 5',
|
| 91 |
+
artifactStep: 5,
|
| 92 |
+
averageReward: 0.596966,
|
| 93 |
+
agreementRate: 0.305556,
|
| 94 |
+
averageRounds: 3.055556,
|
| 95 |
+
rolloutsPerGroup: 3,
|
| 96 |
+
scenarioCount: 4,
|
| 97 |
+
finishedAt: '2026-03-08T18:12:40.950355+00:00',
|
| 98 |
+
},
|
| 99 |
+
];
|
| 100 |
+
|
| 101 |
+
export const TRAINING_LOG_ROWS: TrainingLogRow[] = [
|
| 102 |
+
{
|
| 103 |
+
label: 'clean_accept_seed0_step1',
|
| 104 |
+
trainingStep: 1,
|
| 105 |
+
seed: 0,
|
| 106 |
+
paperTitle: 'Reproducing a CIFAR-10 ResNet-18 baseline',
|
| 107 |
+
reward: 4.54881,
|
| 108 |
+
roundsUsed: 3,
|
| 109 |
+
invalidActionCount: 0,
|
| 110 |
+
parseErrorCount: 0,
|
| 111 |
+
verdict: 'accept',
|
| 112 |
+
note: 'The policy produced a valid protocol and reached a clean judged acceptance.',
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
label: 'ag_news_invalids_step1',
|
| 116 |
+
trainingStep: 1,
|
| 117 |
+
seed: 1,
|
| 118 |
+
paperTitle: 'Reproducing an AG News TinyBERT baseline',
|
| 119 |
+
reward: -0.25,
|
| 120 |
+
roundsUsed: 4,
|
| 121 |
+
invalidActionCount: 4,
|
| 122 |
+
parseErrorCount: 0,
|
| 123 |
+
verdict: null,
|
| 124 |
+
note: 'The main failure mode was repeated invalid actions on the medium AG News case.',
|
| 125 |
+
},
|
| 126 |
+
{
|
| 127 |
+
label: 'diffusion_parse_failure_step2',
|
| 128 |
+
trainingStep: 2,
|
| 129 |
+
seed: 2,
|
| 130 |
+
paperTitle: 'Reproducing an AG News TinyBERT baseline',
|
| 131 |
+
reward: -1.0,
|
| 132 |
+
roundsUsed: 2,
|
| 133 |
+
invalidActionCount: 0,
|
| 134 |
+
parseErrorCount: 1,
|
| 135 |
+
verdict: null,
|
| 136 |
+
note: 'Parser instability still caused zero-score rollouts after training had already begun.',
|
| 137 |
+
},
|
| 138 |
+
{
|
| 139 |
+
label: 'checkpoint5_best_snapshot',
|
| 140 |
+
trainingStep: 5,
|
| 141 |
+
seed: 0,
|
| 142 |
+
paperTitle: 'replicalab-scientist-art-live:step5',
|
| 143 |
+
reward: 0.596966,
|
| 144 |
+
roundsUsed: 3,
|
| 145 |
+
invalidActionCount: 0,
|
| 146 |
+
parseErrorCount: 0,
|
| 147 |
+
verdict: 'mixed',
|
| 148 |
+
note: 'By checkpoint 5 the live run had moved into positive average reward, but not enough to beat baseline on hold-out compare.',
|
| 149 |
+
},
|
| 150 |
+
];
|
| 151 |
+
|
| 152 |
+
export const SCIENTIST_PREVIEW_ARTIFACT: PreviewArtifact = {
|
| 153 |
+
runName: 'scientist-preview-smoke-20260308b',
|
| 154 |
+
modelName: 'Qwen/Qwen3-8B',
|
| 155 |
+
datasetSize: 18,
|
| 156 |
+
evidencePackVersion: '6a0802447dc4',
|
| 157 |
+
config: {
|
| 158 |
+
maxSteps: 12,
|
| 159 |
+
learningRate: 5e-6,
|
| 160 |
+
loraRank: 32,
|
| 161 |
+
maxSeqLength: 4096,
|
| 162 |
+
templates: ['math_reasoning', 'ml_benchmark', 'finance_trading'],
|
| 163 |
+
difficulties: ['easy', 'medium', 'hard'],
|
| 164 |
+
},
|
| 165 |
+
};
|
| 166 |
+
|
| 167 |
+
export const LAB_MANAGER_PREVIEW_ARTIFACT: PreviewArtifact = {
|
| 168 |
+
runName: 'lab-manager-preview-smoke-20260308b',
|
| 169 |
+
modelName: 'Qwen/Qwen3-8B',
|
| 170 |
+
datasetSize: 54,
|
| 171 |
+
evidencePackVersion: '6a0802447dc4',
|
| 172 |
+
config: {
|
| 173 |
+
numTrainEpochs: 1.0,
|
| 174 |
+
learningRate: 2e-5,
|
| 175 |
+
loraRank: 32,
|
| 176 |
+
maxSeqLength: 3072,
|
| 177 |
+
templates: ['math_reasoning', 'ml_benchmark', 'finance_trading'],
|
| 178 |
+
difficulties: ['easy', 'medium', 'hard'],
|
| 179 |
+
},
|
| 180 |
+
};
|
| 181 |
+
|
| 182 |
+
export const LOCAL_BASELINE_SUMMARY = {
|
| 183 |
+
averageReward: 4.600926,
|
| 184 |
+
agreementRate: 1.0,
|
| 185 |
+
averageRounds: 2.0,
|
| 186 |
+
averageRigor: 0.805556,
|
| 187 |
+
averageFeasibility: 1.0,
|
| 188 |
+
averageFidelity: 0.438889,
|
| 189 |
+
episodeCount: 3,
|
| 190 |
+
};
|
| 191 |
+
|
| 192 |
+
export const TRAINING_ASSESSMENT = {
|
| 193 |
+
needsMoreTraining: true,
|
| 194 |
+
achieved: [
|
| 195 |
+
'The minimal Colab path and the reusable training modules are both in place.',
|
| 196 |
+
'Scientist preview data and Lab Manager preview data were generated successfully on the frozen evidence packs.',
|
| 197 |
+
'The live ART/OpenEnv Scientist run reached positive average reward by checkpoint 5.',
|
| 198 |
+
],
|
| 199 |
+
gaps: [
|
| 200 |
+
'Hold-out compare still strongly favors the deterministic baseline over the trained scientist policy.',
|
| 201 |
+
'Invalid-action rate on the hold-out compare is still 1.0 for the trained policy.',
|
| 202 |
+
'Lab Manager has preview artifacts but does not yet have a live trained-and-evaluated adapter in the demo.',
|
| 203 |
+
],
|
| 204 |
+
improvements: [
|
| 205 |
+
'Reduce invalid JSON and invalid action rate before extending training length.',
|
| 206 |
+
'Run more train steps on the same frozen evidence version and compare on fixed held-out seeds after each checkpoint.',
|
| 207 |
+
'Add curriculum or parser-focused reward shaping for the medium ML benchmark cases.',
|
| 208 |
+
'Finish the Lab Manager SFT run and evaluate Scientist-plus-Lab-Manager together instead of only Scientist RL.',
|
| 209 |
+
],
|
| 210 |
+
};
|
frontend/src/lib/api.ts
CHANGED
|
@@ -15,6 +15,7 @@ import type {
|
|
| 15 |
BackendScenarioFamily,
|
| 16 |
ScenarioTemplate,
|
| 17 |
Difficulty,
|
|
|
|
| 18 |
} from '@/types';
|
| 19 |
|
| 20 |
const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? '/api';
|
|
@@ -121,6 +122,34 @@ function observationToEpisodeState(
|
|
| 121 |
conversation: adaptConversation(sci.conversation_history),
|
| 122 |
scores: null,
|
| 123 |
judge_audit: null,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
};
|
| 125 |
}
|
| 126 |
|
|
@@ -222,9 +251,11 @@ export async function stepEpisode(
|
|
| 222 |
: prevState.conversation;
|
| 223 |
const protocol = obs?.scientist?.current_protocol ?? prevState.protocol;
|
| 224 |
const round = obs?.scientist?.round_number ?? prevState.round + 1;
|
|
|
|
| 225 |
|
| 226 |
// Update lab constraints if available
|
| 227 |
const labConstraints = obs ? adaptLabConstraints(obs) : prevState.lab_constraints;
|
|
|
|
| 228 |
|
| 229 |
return {
|
| 230 |
...prevState,
|
|
@@ -235,6 +266,8 @@ export async function stepEpisode(
|
|
| 235 |
lab_constraints: labConstraints,
|
| 236 |
scores,
|
| 237 |
judge_audit: judgeAudit,
|
|
|
|
|
|
|
| 238 |
};
|
| 239 |
}
|
| 240 |
|
|
@@ -297,41 +330,61 @@ export function sendWsMessage(ws: WebSocket, msg: WebSocketMessage) {
|
|
| 297 |
// ---------------------------------------------------------------------------
|
| 298 |
|
| 299 |
export function buildDefaultScientistAction(state?: EpisodeState): ScientistAction {
|
| 300 |
-
if (state?.protocol) {
|
| 301 |
-
return buildAcceptAction();
|
| 302 |
-
}
|
| 303 |
-
|
| 304 |
const durationLimit = Math.max(1, state?.lab_constraints.time_limit_days ?? 3);
|
| 305 |
-
const originalDuration = state?.paper.original_duration_days ?? 0;
|
| 306 |
-
const durationDays = Math.max(1, Math.min(durationLimit, originalDuration || durationLimit));
|
| 307 |
const template = state?.template;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 308 |
|
| 309 |
const technique =
|
| 310 |
-
|
| 311 |
-
? state.paper.original_technique
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
: template === 'math_reasoning'
|
| 313 |
-
?
|
| 314 |
-
:
|
| 315 |
-
? 'offline_backtest'
|
| 316 |
-
: 'published_training_recipe';
|
| 317 |
-
|
| 318 |
-
const controls = state?.paper.original_controls.length
|
| 319 |
-
? state.paper.original_controls
|
| 320 |
-
: ['baseline'];
|
| 321 |
|
| 322 |
-
const requiredEquipment =
|
| 323 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 324 |
|
| 325 |
return {
|
| 326 |
-
action_type: 'propose_protocol',
|
| 327 |
-
sample_size:
|
| 328 |
controls,
|
| 329 |
technique,
|
| 330 |
duration_days: durationDays,
|
| 331 |
required_equipment: requiredEquipment,
|
| 332 |
required_reagents: requiredReagents,
|
| 333 |
questions: [],
|
| 334 |
-
rationale:
|
|
|
|
|
|
|
| 335 |
};
|
| 336 |
}
|
| 337 |
|
|
|
|
| 15 |
BackendScenarioFamily,
|
| 16 |
ScenarioTemplate,
|
| 17 |
Difficulty,
|
| 18 |
+
EpisodeStepTrace,
|
| 19 |
} from '@/types';
|
| 20 |
|
| 21 |
const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? '/api';
|
|
|
|
| 122 |
conversation: adaptConversation(sci.conversation_history),
|
| 123 |
scores: null,
|
| 124 |
judge_audit: null,
|
| 125 |
+
cumulative_reward: 0,
|
| 126 |
+
step_history: [],
|
| 127 |
+
};
|
| 128 |
+
}
|
| 129 |
+
|
| 130 |
+
function buildRoundTrace(
|
| 131 |
+
prevState: EpisodeState,
|
| 132 |
+
data: BackendStepResult,
|
| 133 |
+
): EpisodeStepTrace {
|
| 134 |
+
const round = data.info.round;
|
| 135 |
+
const rawHistory = data.observation?.scientist?.conversation_history ?? [];
|
| 136 |
+
const roundEntries = rawHistory.filter((entry) => entry.round_number === round);
|
| 137 |
+
const scientistEntry = roundEntries.find((entry) => entry.role === 'scientist');
|
| 138 |
+
const labManagerEntry = roundEntries.find((entry) => entry.role === 'lab_manager');
|
| 139 |
+
|
| 140 |
+
return {
|
| 141 |
+
round,
|
| 142 |
+
reward: data.reward,
|
| 143 |
+
cumulative_reward: data.info.cumulative_reward ?? prevState.cumulative_reward + data.reward,
|
| 144 |
+
action_type: scientistEntry?.action_type ?? 'unknown',
|
| 145 |
+
scientist_message: scientistEntry?.message ?? '',
|
| 146 |
+
lab_manager_action_type: labManagerEntry?.action_type ?? undefined,
|
| 147 |
+
lab_manager_message: labManagerEntry?.message ?? undefined,
|
| 148 |
+
step_reward_components: data.info.step_reward_components ?? {},
|
| 149 |
+
protocol: data.observation?.scientist?.current_protocol ?? prevState.protocol,
|
| 150 |
+
oracle_round_score: data.info.oracle_round_score ?? null,
|
| 151 |
+
oracle_post_mortem: data.info.oracle_post_mortem ?? null,
|
| 152 |
+
oracle_event: data.info.oracle_event ?? null,
|
| 153 |
};
|
| 154 |
}
|
| 155 |
|
|
|
|
| 251 |
: prevState.conversation;
|
| 252 |
const protocol = obs?.scientist?.current_protocol ?? prevState.protocol;
|
| 253 |
const round = obs?.scientist?.round_number ?? prevState.round + 1;
|
| 254 |
+
const cumulativeReward = data.info.cumulative_reward ?? prevState.cumulative_reward + data.reward;
|
| 255 |
|
| 256 |
// Update lab constraints if available
|
| 257 |
const labConstraints = obs ? adaptLabConstraints(obs) : prevState.lab_constraints;
|
| 258 |
+
const roundTrace = buildRoundTrace(prevState, data);
|
| 259 |
|
| 260 |
return {
|
| 261 |
...prevState,
|
|
|
|
| 266 |
lab_constraints: labConstraints,
|
| 267 |
scores,
|
| 268 |
judge_audit: judgeAudit,
|
| 269 |
+
cumulative_reward: cumulativeReward,
|
| 270 |
+
step_history: [...prevState.step_history, roundTrace],
|
| 271 |
};
|
| 272 |
}
|
| 273 |
|
|
|
|
| 330 |
// ---------------------------------------------------------------------------
|
| 331 |
|
| 332 |
export function buildDefaultScientistAction(state?: EpisodeState): ScientistAction {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 333 |
const durationLimit = Math.max(1, state?.lab_constraints.time_limit_days ?? 3);
|
|
|
|
|
|
|
| 334 |
const template = state?.template;
|
| 335 |
+
const currentProtocol = state?.protocol;
|
| 336 |
+
|
| 337 |
+
const originalDuration = state?.paper.original_duration_days ?? 0;
|
| 338 |
+
const preferredDuration = currentProtocol?.duration_days ?? (originalDuration || durationLimit);
|
| 339 |
+
const durationDays = Math.max(
|
| 340 |
+
1,
|
| 341 |
+
Math.min(durationLimit, preferredDuration),
|
| 342 |
+
);
|
| 343 |
|
| 344 |
const technique =
|
| 345 |
+
currentProtocol?.technique
|
| 346 |
+
?? (state?.paper.original_technique && state.paper.original_technique !== 'N/A'
|
| 347 |
+
? state.paper.original_technique
|
| 348 |
+
: template === 'math_reasoning'
|
| 349 |
+
? 'structured_proof_check'
|
| 350 |
+
: template === 'finance_trading'
|
| 351 |
+
? 'offline_backtest'
|
| 352 |
+
: 'published_training_recipe');
|
| 353 |
+
|
| 354 |
+
const controls =
|
| 355 |
+
currentProtocol?.controls.length
|
| 356 |
+
? currentProtocol.controls
|
| 357 |
+
: state?.paper.original_controls.length
|
| 358 |
+
? state.paper.original_controls
|
| 359 |
+
: ['baseline'];
|
| 360 |
+
|
| 361 |
+
const baseSampleSize = currentProtocol?.sample_size ?? 3;
|
| 362 |
+
const sampleSize =
|
| 363 |
+
state?.round && state.round > 0
|
| 364 |
+
? Math.max(3, Math.min(baseSampleSize + (state.round % 2 === 0 ? 1 : -1), 12))
|
| 365 |
: template === 'math_reasoning'
|
| 366 |
+
? 4
|
| 367 |
+
: 3;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 368 |
|
| 369 |
+
const requiredEquipment = currentProtocol?.required_equipment.length
|
| 370 |
+
? currentProtocol.required_equipment
|
| 371 |
+
: state?.lab_constraints.equipment_available.slice(0, 1) ?? [];
|
| 372 |
+
const requiredReagents = currentProtocol?.required_reagents.length
|
| 373 |
+
? currentProtocol.required_reagents
|
| 374 |
+
: state?.lab_constraints.reagents_available.slice(0, 1) ?? [];
|
| 375 |
|
| 376 |
return {
|
| 377 |
+
action_type: currentProtocol ? 'revise_protocol' : 'propose_protocol',
|
| 378 |
+
sample_size: sampleSize,
|
| 379 |
controls,
|
| 380 |
technique,
|
| 381 |
duration_days: durationDays,
|
| 382 |
required_equipment: requiredEquipment,
|
| 383 |
required_reagents: requiredReagents,
|
| 384 |
questions: [],
|
| 385 |
+
rationale: currentProtocol
|
| 386 |
+
? `Refine the existing protocol for round ${state?.round ?? 0} while staying inside the ${durationLimit}-day lab window.`
|
| 387 |
+
: `Replicate the source result within the available lab window of ${durationLimit} days using currently available resources.`,
|
| 388 |
};
|
| 389 |
}
|
| 390 |
|
frontend/src/lib/demo.ts
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import type { DemoCase, EpisodeState, ScientistAction } from '@/types';
|
| 2 |
+
import { buildAcceptAction, buildDefaultScientistAction } from '@/lib/api';
|
| 3 |
+
|
| 4 |
+
export interface DemoCaseMeta {
|
| 5 |
+
id: DemoCase;
|
| 6 |
+
title: string;
|
| 7 |
+
subtitle: string;
|
| 8 |
+
summary: string;
|
| 9 |
+
seed: number;
|
| 10 |
+
}
|
| 11 |
+
|
| 12 |
+
export const DEMO_CASES: DemoCaseMeta[] = [
|
| 13 |
+
{
|
| 14 |
+
id: 'fast-agreement',
|
| 15 |
+
title: 'Scenario 1: First-Round Agreement',
|
| 16 |
+
subtitle: 'Good paper, fast agreement',
|
| 17 |
+
summary: 'A strong paper-derived benchmark where the first protocol is feasible and the agents converge immediately.',
|
| 18 |
+
seed: 101,
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
id: 'learning-opportunity',
|
| 22 |
+
title: 'Scenario 2: Multi-Round Learning',
|
| 23 |
+
subtitle: 'Heavy RL learning opportunity',
|
| 24 |
+
summary: 'The Scientist starts overly ambitious, absorbs constraint feedback across several rounds, then lands on a feasible protocol.',
|
| 25 |
+
seed: 202,
|
| 26 |
+
},
|
| 27 |
+
{
|
| 28 |
+
id: 'no-agreement',
|
| 29 |
+
title: 'Scenario 3: No Agreement',
|
| 30 |
+
subtitle: 'Reject, not replicable under current setup',
|
| 31 |
+
summary: 'The protocol never becomes acceptable within the six-round budget, producing a clean rejection case for the demo.',
|
| 32 |
+
seed: 303,
|
| 33 |
+
},
|
| 34 |
+
];
|
| 35 |
+
|
| 36 |
+
export function parseDemoCase(value: string | null): DemoCase | undefined {
|
| 37 |
+
return DEMO_CASES.find((item) => item.id === value)?.id;
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
function buildScenarioAwareProposal(
|
| 41 |
+
state: EpisodeState,
|
| 42 |
+
overrides: Partial<ScientistAction>,
|
| 43 |
+
): ScientistAction {
|
| 44 |
+
const baseAction = buildDefaultScientistAction(state);
|
| 45 |
+
const actionType = state.protocol ? 'revise_protocol' : 'propose_protocol';
|
| 46 |
+
|
| 47 |
+
return {
|
| 48 |
+
...baseAction,
|
| 49 |
+
action_type: actionType,
|
| 50 |
+
...overrides,
|
| 51 |
+
};
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
export function buildDemoScientistAction(
|
| 55 |
+
state: EpisodeState,
|
| 56 |
+
demoCase?: DemoCase,
|
| 57 |
+
): ScientistAction {
|
| 58 |
+
if (!demoCase) {
|
| 59 |
+
return state.round >= state.max_rounds - 1
|
| 60 |
+
? buildAcceptAction()
|
| 61 |
+
: buildDefaultScientistAction(state);
|
| 62 |
+
}
|
| 63 |
+
|
| 64 |
+
const defaultEquipment = state.lab_constraints.equipment_available.slice(0, 1);
|
| 65 |
+
const defaultReagents = state.lab_constraints.reagents_available.slice(0, 1);
|
| 66 |
+
const defaultTechnique =
|
| 67 |
+
state.protocol?.technique ||
|
| 68 |
+
(state.paper.original_technique !== 'N/A'
|
| 69 |
+
? state.paper.original_technique
|
| 70 |
+
: 'published_training_recipe');
|
| 71 |
+
|
| 72 |
+
if (demoCase === 'fast-agreement') {
|
| 73 |
+
if (state.round >= 1) {
|
| 74 |
+
return buildAcceptAction();
|
| 75 |
+
}
|
| 76 |
+
return buildScenarioAwareProposal(state, {
|
| 77 |
+
sample_size: 3,
|
| 78 |
+
controls: ['baseline'],
|
| 79 |
+
technique: defaultTechnique,
|
| 80 |
+
duration_days: Math.min(4, state.lab_constraints.time_limit_days),
|
| 81 |
+
required_equipment: defaultEquipment,
|
| 82 |
+
required_reagents: defaultReagents,
|
| 83 |
+
rationale: 'Reproduce the published baseline exactly once with the available reviewer pass and hardware.',
|
| 84 |
+
});
|
| 85 |
+
}
|
| 86 |
+
|
| 87 |
+
if (demoCase === 'learning-opportunity') {
|
| 88 |
+
const learningPlan: Record<number, Partial<ScientistAction>> = {
|
| 89 |
+
0: {
|
| 90 |
+
sample_size: 96,
|
| 91 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 92 |
+
rationale: 'Start with an expansive benchmark reproduction to maximize coverage before trimming.',
|
| 93 |
+
},
|
| 94 |
+
1: {
|
| 95 |
+
sample_size: 72,
|
| 96 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 97 |
+
rationale: 'Reduce the experimental load, but keep the extra controls while the constraints are still being learned.',
|
| 98 |
+
},
|
| 99 |
+
2: {
|
| 100 |
+
sample_size: 48,
|
| 101 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 102 |
+
rationale: 'Continue lowering the sample size while preserving the full validation set of controls.',
|
| 103 |
+
},
|
| 104 |
+
3: {
|
| 105 |
+
sample_size: 32,
|
| 106 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 107 |
+
rationale: 'The Scientist now converges toward feasibility by balancing budget with stronger controls.',
|
| 108 |
+
},
|
| 109 |
+
4: {
|
| 110 |
+
sample_size: 24,
|
| 111 |
+
controls: ['baseline', 'ablation'],
|
| 112 |
+
rationale: 'Final revision: keep the baseline plus one ablation and fit the protocol cleanly inside the lab capacity.',
|
| 113 |
+
},
|
| 114 |
+
};
|
| 115 |
+
|
| 116 |
+
if (state.round >= 5) {
|
| 117 |
+
return buildAcceptAction();
|
| 118 |
+
}
|
| 119 |
+
return buildScenarioAwareProposal(state, {
|
| 120 |
+
technique: defaultTechnique,
|
| 121 |
+
duration_days: Math.min(4, state.lab_constraints.time_limit_days),
|
| 122 |
+
required_equipment: defaultEquipment,
|
| 123 |
+
required_reagents: defaultReagents,
|
| 124 |
+
...learningPlan[state.round],
|
| 125 |
+
});
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
const rejectionPlan: Record<number, Partial<ScientistAction>> = {
|
| 129 |
+
0: {
|
| 130 |
+
sample_size: 96,
|
| 131 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 132 |
+
rationale: 'Push for the full benchmark replication despite the current resource limits.',
|
| 133 |
+
},
|
| 134 |
+
1: {
|
| 135 |
+
sample_size: 88,
|
| 136 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 137 |
+
rationale: 'Trim slightly, but keep the high-coverage plan intact.',
|
| 138 |
+
},
|
| 139 |
+
2: {
|
| 140 |
+
sample_size: 80,
|
| 141 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 142 |
+
rationale: 'Still prioritize coverage over the budget warnings.',
|
| 143 |
+
},
|
| 144 |
+
3: {
|
| 145 |
+
sample_size: 76,
|
| 146 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 147 |
+
rationale: 'The Scientist refuses to reduce scope enough to become feasible.',
|
| 148 |
+
},
|
| 149 |
+
4: {
|
| 150 |
+
sample_size: 72,
|
| 151 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 152 |
+
rationale: 'The protocol remains too expensive and staff-heavy for this lab.',
|
| 153 |
+
},
|
| 154 |
+
5: {
|
| 155 |
+
sample_size: 68,
|
| 156 |
+
controls: ['baseline', 'ablation', 'sanity_check'],
|
| 157 |
+
rationale: 'Final round: still no viable path to agreement under the current constraints.',
|
| 158 |
+
},
|
| 159 |
+
};
|
| 160 |
+
|
| 161 |
+
return buildScenarioAwareProposal(state, {
|
| 162 |
+
technique: defaultTechnique,
|
| 163 |
+
duration_days: Math.min(4, state.lab_constraints.time_limit_days),
|
| 164 |
+
required_equipment: defaultEquipment,
|
| 165 |
+
required_reagents: defaultReagents,
|
| 166 |
+
...(rejectionPlan[state.round] ?? rejectionPlan[5]),
|
| 167 |
+
});
|
| 168 |
+
}
|
frontend/src/pages/ComparePage.tsx
CHANGED
|
@@ -17,7 +17,8 @@ export default function ComparePage() {
|
|
| 17 |
<h1 className="mb-2 text-3xl font-bold tracking-tight">Seeded Benchmark Comparison</h1>
|
| 18 |
<p className="mx-auto max-w-lg text-muted-foreground">
|
| 19 |
Use the same frontend to replay fixed benchmark cases across domains or difficulty levels.
|
| 20 |
-
This page is the evaluation bench for demo runs, while the
|
|
|
|
| 21 |
</p>
|
| 22 |
</motion.div>
|
| 23 |
|
|
|
|
| 17 |
<h1 className="mb-2 text-3xl font-bold tracking-tight">Seeded Benchmark Comparison</h1>
|
| 18 |
<p className="mx-auto max-w-lg text-muted-foreground">
|
| 19 |
Use the same frontend to replay fixed benchmark cases across domains or difficulty levels.
|
| 20 |
+
This page is the evaluation bench for demo runs, while the dedicated Training page holds the artifact-backed
|
| 21 |
+
training logs and improvement analysis.
|
| 22 |
</p>
|
| 23 |
</motion.div>
|
| 24 |
|
frontend/src/pages/DashboardPage.tsx
CHANGED
|
@@ -10,12 +10,16 @@ import {
|
|
| 10 |
MessageSquare,
|
| 11 |
Scale,
|
| 12 |
TrendingUp,
|
|
|
|
|
|
|
|
|
|
| 13 |
} from 'lucide-react';
|
| 14 |
import TrainingResults from '@/components/TrainingResults';
|
| 15 |
import AnimatedCharacter from '@/components/AnimatedCharacter';
|
| 16 |
import MoleculeScene from '@/components/MoleculeScene';
|
| 17 |
import TiltCard from '@/components/TiltCard';
|
| 18 |
import { cn } from '@/lib/utils';
|
|
|
|
| 19 |
|
| 20 |
const SCENARIOS = [
|
| 21 |
{
|
|
@@ -50,7 +54,7 @@ const SCENARIOS = [
|
|
| 50 |
const ROLES = [
|
| 51 |
{
|
| 52 |
key: 'scientist',
|
| 53 |
-
name: 'Dr. Elara
|
| 54 |
tagline: 'Protects the Science',
|
| 55 |
description: 'A brilliant researcher who proposes, revises, and asks questions to build the best replication plan without compromising scientific validity.',
|
| 56 |
color: 'text-scientist',
|
|
@@ -59,7 +63,7 @@ const ROLES = [
|
|
| 59 |
},
|
| 60 |
{
|
| 61 |
key: 'lab_manager',
|
| 62 |
-
name: 'Takuma
|
| 63 |
tagline: 'Guards Feasibility',
|
| 64 |
description: 'A pragmatic organizer who reports budget, equipment, scheduling, and staffing constraints. He ensures plans are actually executable.',
|
| 65 |
color: 'text-lab-manager',
|
|
@@ -68,7 +72,7 @@ const ROLES = [
|
|
| 68 |
},
|
| 69 |
{
|
| 70 |
key: 'judge',
|
| 71 |
-
name: 'Aldric
|
| 72 |
tagline: 'Delivers the Verdict',
|
| 73 |
description: 'An impartial arbiter who scores the final protocol on rigor, feasibility, and fidelity using a deterministic rubric engine.',
|
| 74 |
color: 'text-judge',
|
|
@@ -176,19 +180,19 @@ export default function DashboardPage() {
|
|
| 176 |
|
| 177 |
<div className="mt-8 flex items-center justify-center gap-3">
|
| 178 |
<Link
|
| 179 |
-
to="/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1"
|
| 180 |
className="inline-flex items-center gap-2 rounded-lg bg-primary px-5 py-2.5 text-sm font-medium text-primary-foreground transition-colors hover:bg-primary/90 shadow-lg shadow-primary/25"
|
| 181 |
>
|
| 182 |
<Play className="h-4 w-4" />
|
| 183 |
Replicate a Paper
|
| 184 |
</Link>
|
| 185 |
-
<
|
| 186 |
-
|
| 187 |
className="inline-flex items-center gap-2 rounded-lg border border-border bg-background/80 backdrop-blur-sm px-5 py-2.5 text-sm font-medium text-foreground transition-colors hover:bg-muted"
|
| 188 |
>
|
| 189 |
<BarChart3 className="h-4 w-4" />
|
| 190 |
-
See Training
|
| 191 |
-
</
|
| 192 |
</div>
|
| 193 |
|
| 194 |
<div className="mx-auto mt-8 grid max-w-4xl gap-3 text-left md:grid-cols-4">
|
|
@@ -296,6 +300,41 @@ export default function DashboardPage() {
|
|
| 296 |
</div>
|
| 297 |
</section>
|
| 298 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 299 |
{/* Training Results */}
|
| 300 |
<section id="training" className="mb-12">
|
| 301 |
<h2 className="mb-2 text-center text-xl font-semibold">Training Results</h2>
|
|
@@ -307,10 +346,10 @@ export default function DashboardPage() {
|
|
| 307 |
</div>
|
| 308 |
<div className="mt-4 text-center">
|
| 309 |
<Link
|
| 310 |
-
to="/
|
| 311 |
className="inline-flex items-center gap-2 text-sm font-medium text-primary transition-colors hover:text-primary/80"
|
| 312 |
>
|
| 313 |
-
Open the
|
| 314 |
<ArrowRight className="h-4 w-4" />
|
| 315 |
</Link>
|
| 316 |
</div>
|
|
|
|
| 10 |
MessageSquare,
|
| 11 |
Scale,
|
| 12 |
TrendingUp,
|
| 13 |
+
CheckCircle2,
|
| 14 |
+
AlertTriangle,
|
| 15 |
+
XCircle,
|
| 16 |
} from 'lucide-react';
|
| 17 |
import TrainingResults from '@/components/TrainingResults';
|
| 18 |
import AnimatedCharacter from '@/components/AnimatedCharacter';
|
| 19 |
import MoleculeScene from '@/components/MoleculeScene';
|
| 20 |
import TiltCard from '@/components/TiltCard';
|
| 21 |
import { cn } from '@/lib/utils';
|
| 22 |
+
import { DEMO_CASES } from '@/lib/demo';
|
| 23 |
|
| 24 |
const SCENARIOS = [
|
| 25 |
{
|
|
|
|
| 54 |
const ROLES = [
|
| 55 |
{
|
| 56 |
key: 'scientist',
|
| 57 |
+
name: 'Dr. Elara - The Scientist',
|
| 58 |
tagline: 'Protects the Science',
|
| 59 |
description: 'A brilliant researcher who proposes, revises, and asks questions to build the best replication plan without compromising scientific validity.',
|
| 60 |
color: 'text-scientist',
|
|
|
|
| 63 |
},
|
| 64 |
{
|
| 65 |
key: 'lab_manager',
|
| 66 |
+
name: 'Takuma - The Lab Manager',
|
| 67 |
tagline: 'Guards Feasibility',
|
| 68 |
description: 'A pragmatic organizer who reports budget, equipment, scheduling, and staffing constraints. He ensures plans are actually executable.',
|
| 69 |
color: 'text-lab-manager',
|
|
|
|
| 72 |
},
|
| 73 |
{
|
| 74 |
key: 'judge',
|
| 75 |
+
name: 'Aldric - The Judge',
|
| 76 |
tagline: 'Delivers the Verdict',
|
| 77 |
description: 'An impartial arbiter who scores the final protocol on rigor, feasibility, and fidelity using a deterministic rubric engine.',
|
| 78 |
color: 'text-judge',
|
|
|
|
| 180 |
|
| 181 |
<div className="mt-8 flex items-center justify-center gap-3">
|
| 182 |
<Link
|
| 183 |
+
to="/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1&demoCase=fast-agreement"
|
| 184 |
className="inline-flex items-center gap-2 rounded-lg bg-primary px-5 py-2.5 text-sm font-medium text-primary-foreground transition-colors hover:bg-primary/90 shadow-lg shadow-primary/25"
|
| 185 |
>
|
| 186 |
<Play className="h-4 w-4" />
|
| 187 |
Replicate a Paper
|
| 188 |
</Link>
|
| 189 |
+
<Link
|
| 190 |
+
to="/training"
|
| 191 |
className="inline-flex items-center gap-2 rounded-lg border border-border bg-background/80 backdrop-blur-sm px-5 py-2.5 text-sm font-medium text-foreground transition-colors hover:bg-muted"
|
| 192 |
>
|
| 193 |
<BarChart3 className="h-4 w-4" />
|
| 194 |
+
See Training Logs
|
| 195 |
+
</Link>
|
| 196 |
</div>
|
| 197 |
|
| 198 |
<div className="mx-auto mt-8 grid max-w-4xl gap-3 text-left md:grid-cols-4">
|
|
|
|
| 300 |
</div>
|
| 301 |
</section>
|
| 302 |
|
| 303 |
+
<section className="mb-16">
|
| 304 |
+
<h2 className="mb-2 text-center text-xl font-semibold">Live Demo Outcomes</h2>
|
| 305 |
+
<p className="mb-8 text-center text-sm text-muted-foreground">
|
| 306 |
+
Pick the exact story you want to show: immediate agreement, multi-round learning, or clear rejection.
|
| 307 |
+
</p>
|
| 308 |
+
<div className="grid gap-4 md:grid-cols-3">
|
| 309 |
+
{DEMO_CASES.map((demo) => {
|
| 310 |
+
const Icon =
|
| 311 |
+
demo.id === 'fast-agreement'
|
| 312 |
+
? CheckCircle2
|
| 313 |
+
: demo.id === 'learning-opportunity'
|
| 314 |
+
? AlertTriangle
|
| 315 |
+
: XCircle;
|
| 316 |
+
|
| 317 |
+
return (
|
| 318 |
+
<Link
|
| 319 |
+
key={demo.id}
|
| 320 |
+
to={`/episode?template=ml_benchmark&difficulty=medium&seed=${demo.seed}&demo=1&autoplay=1&demoCase=${demo.id}`}
|
| 321 |
+
className="rounded-xl border border-border bg-card p-5 transition-colors hover:border-primary/40 hover:bg-muted/40"
|
| 322 |
+
>
|
| 323 |
+
<div className="mb-3 inline-flex rounded-full bg-primary/10 p-2 text-primary">
|
| 324 |
+
<Icon className="h-4 w-4" />
|
| 325 |
+
</div>
|
| 326 |
+
<h3 className="text-sm font-semibold">{demo.title}</h3>
|
| 327 |
+
<p className="mt-1 text-xs font-medium text-primary">{demo.subtitle}</p>
|
| 328 |
+
<p className="mt-3 text-sm text-muted-foreground">{demo.summary}</p>
|
| 329 |
+
<div className="mt-4 inline-flex items-center gap-1 text-xs font-medium text-primary">
|
| 330 |
+
Launch demo <ArrowRight className="h-3 w-3" />
|
| 331 |
+
</div>
|
| 332 |
+
</Link>
|
| 333 |
+
);
|
| 334 |
+
})}
|
| 335 |
+
</div>
|
| 336 |
+
</section>
|
| 337 |
+
|
| 338 |
{/* Training Results */}
|
| 339 |
<section id="training" className="mb-12">
|
| 340 |
<h2 className="mb-2 text-center text-xl font-semibold">Training Results</h2>
|
|
|
|
| 346 |
</div>
|
| 347 |
<div className="mt-4 text-center">
|
| 348 |
<Link
|
| 349 |
+
to="/training"
|
| 350 |
className="inline-flex items-center gap-2 text-sm font-medium text-primary transition-colors hover:text-primary/80"
|
| 351 |
>
|
| 352 |
+
Open the detailed training page
|
| 353 |
<ArrowRight className="h-4 w-4" />
|
| 354 |
</Link>
|
| 355 |
</div>
|
frontend/src/pages/EpisodePage.tsx
CHANGED
|
@@ -3,9 +3,10 @@ import { useNavigate, useSearchParams } from 'react-router-dom';
|
|
| 3 |
import { motion, AnimatePresence } from 'framer-motion';
|
| 4 |
import { FileText, FlaskConical, Scale, TrendingUp, Play, Pencil } from 'lucide-react';
|
| 5 |
import type { EpisodeState, ResetParams, ScientistAction } from '@/types';
|
| 6 |
-
import { resetEpisode, stepEpisode
|
| 7 |
import { sfx, startAmbient, stopAmbient } from '@/lib/audio';
|
| 8 |
import { fireSuccessConfetti, fireGavelConfetti } from '@/lib/confetti';
|
|
|
|
| 9 |
import { useToast } from '@/components/Toast';
|
| 10 |
import { useKeyboardShortcuts, ShortcutsOverlay, ShortcutHint } from '@/components/KeyboardShortcuts';
|
| 11 |
import AutoPlayControls, { useAutoPlay } from '@/components/AutoPlayControls';
|
|
@@ -24,6 +25,7 @@ import ProtocolTimeline from '@/components/ProtocolTimeline';
|
|
| 24 |
import AgentThoughts from '@/components/AgentThoughts';
|
| 25 |
import ProtocolEditor from '@/components/ProtocolEditor';
|
| 26 |
import LabScene3D from '@/components/LabScene3D';
|
|
|
|
| 27 |
|
| 28 |
const TEMPLATE_OPTIONS = ['math_reasoning', 'ml_benchmark', 'finance_trading'] as const;
|
| 29 |
const DIFFICULTY_OPTIONS = ['easy', 'medium', 'hard'] as const;
|
|
@@ -66,6 +68,11 @@ export default function EpisodePage() {
|
|
| 66 |
() => searchParams.get('demo') === '1' || searchParams.get('autostart') === '1',
|
| 67 |
[searchParams],
|
| 68 |
);
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
const autoPlayRequested = useMemo(
|
| 70 |
() => demoRequested || searchParams.get('autoplay') === '1',
|
| 71 |
[demoRequested, searchParams],
|
|
@@ -86,6 +93,7 @@ export default function EpisodePage() {
|
|
| 86 |
// Sound effects for phase transitions
|
| 87 |
const prevPhaseRef = useRef(phase);
|
| 88 |
const prevMsgCountRef = useRef(0);
|
|
|
|
| 89 |
|
| 90 |
useEffect(() => {
|
| 91 |
const prev = prevPhaseRef.current;
|
|
@@ -109,10 +117,10 @@ export default function EpisodePage() {
|
|
| 109 |
sfx.success();
|
| 110 |
fireSuccessConfetti();
|
| 111 |
}, 400);
|
| 112 |
-
toast('Episode complete
|
| 113 |
} else if (verdict) {
|
| 114 |
setTimeout(() => sfx.failure(), 400);
|
| 115 |
-
toast(`Episode complete
|
| 116 |
}
|
| 117 |
}
|
| 118 |
}, [phase, episode?.judge_audit?.verdict, toast]);
|
|
@@ -134,6 +142,12 @@ export default function EpisodePage() {
|
|
| 134 |
return () => stopAmbient();
|
| 135 |
}, []);
|
| 136 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
const handleStart = useCallback(async (params: ResetParams) => {
|
| 138 |
setLoading(true);
|
| 139 |
setIsJudging(false);
|
|
@@ -142,7 +156,8 @@ export default function EpisodePage() {
|
|
| 142 |
sfx.click();
|
| 143 |
try {
|
| 144 |
const state = await resetEpisode(params);
|
| 145 |
-
|
|
|
|
| 146 |
prevMsgCountRef.current = 0;
|
| 147 |
toast('Episode started!', 'info');
|
| 148 |
// Feature 9: Update URL for shareable link
|
|
@@ -156,6 +171,9 @@ export default function EpisodePage() {
|
|
| 156 |
if (autoPlayRequested) {
|
| 157 |
nextSearch.set('autoplay', '1');
|
| 158 |
}
|
|
|
|
|
|
|
|
|
|
| 159 |
navigate(`/episode/${state.episode_id}?${nextSearch.toString()}`, { replace: true });
|
| 160 |
} catch (err) {
|
| 161 |
console.error('Failed to start episode:', err);
|
|
@@ -165,7 +183,7 @@ export default function EpisodePage() {
|
|
| 165 |
} finally {
|
| 166 |
setLoading(false);
|
| 167 |
}
|
| 168 |
-
}, [autoPlayRequested, demoRequested, navigate, toast]);
|
| 169 |
|
| 170 |
const handleStepWithAction = useCallback(async (action?: ScientistAction) => {
|
| 171 |
if (!episode || episode.done) return;
|
|
@@ -178,7 +196,7 @@ export default function EpisodePage() {
|
|
| 178 |
|
| 179 |
try {
|
| 180 |
const isLastRound = episode.round >= episode.max_rounds - 1;
|
| 181 |
-
const finalAction = action ?? (
|
| 182 |
|
| 183 |
if (isLastRound && !action) {
|
| 184 |
setIsJudging(true);
|
|
@@ -205,7 +223,7 @@ export default function EpisodePage() {
|
|
| 205 |
} finally {
|
| 206 |
setLoading(false);
|
| 207 |
}
|
| 208 |
-
}, [episode, toast]);
|
| 209 |
|
| 210 |
useEffect(() => {
|
| 211 |
if (!demoRequested || autoStartTriggered || episode || loading) {
|
|
@@ -395,6 +413,28 @@ export default function EpisodePage() {
|
|
| 395 |
))}
|
| 396 |
</motion.div>
|
| 397 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 398 |
{!episode.done && episode.conversation.length === 0 && (
|
| 399 |
<motion.div
|
| 400 |
className="mb-4 rounded-lg border border-primary/20 bg-primary/5 p-4"
|
|
@@ -578,6 +618,18 @@ export default function EpisodePage() {
|
|
| 578 |
{episode.done && <ReplayViewer messages={episode.conversation} />}
|
| 579 |
</motion.div>
|
| 580 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 581 |
</div>
|
| 582 |
);
|
| 583 |
}
|
|
|
|
| 3 |
import { motion, AnimatePresence } from 'framer-motion';
|
| 4 |
import { FileText, FlaskConical, Scale, TrendingUp, Play, Pencil } from 'lucide-react';
|
| 5 |
import type { EpisodeState, ResetParams, ScientistAction } from '@/types';
|
| 6 |
+
import { resetEpisode, stepEpisode } from '@/lib/api';
|
| 7 |
import { sfx, startAmbient, stopAmbient } from '@/lib/audio';
|
| 8 |
import { fireSuccessConfetti, fireGavelConfetti } from '@/lib/confetti';
|
| 9 |
+
import { buildDemoScientistAction, DEMO_CASES, parseDemoCase } from '@/lib/demo';
|
| 10 |
import { useToast } from '@/components/Toast';
|
| 11 |
import { useKeyboardShortcuts, ShortcutsOverlay, ShortcutHint } from '@/components/KeyboardShortcuts';
|
| 12 |
import AutoPlayControls, { useAutoPlay } from '@/components/AutoPlayControls';
|
|
|
|
| 25 |
import AgentThoughts from '@/components/AgentThoughts';
|
| 26 |
import ProtocolEditor from '@/components/ProtocolEditor';
|
| 27 |
import LabScene3D from '@/components/LabScene3D';
|
| 28 |
+
import EpisodeResultsReport from '@/components/EpisodeResultsReport';
|
| 29 |
|
| 30 |
const TEMPLATE_OPTIONS = ['math_reasoning', 'ml_benchmark', 'finance_trading'] as const;
|
| 31 |
const DIFFICULTY_OPTIONS = ['easy', 'medium', 'hard'] as const;
|
|
|
|
| 68 |
() => searchParams.get('demo') === '1' || searchParams.get('autostart') === '1',
|
| 69 |
[searchParams],
|
| 70 |
);
|
| 71 |
+
const demoCase = useMemo(() => parseDemoCase(searchParams.get('demoCase')), [searchParams]);
|
| 72 |
+
const demoMeta = useMemo(
|
| 73 |
+
() => DEMO_CASES.find((item) => item.id === demoCase),
|
| 74 |
+
[demoCase],
|
| 75 |
+
);
|
| 76 |
const autoPlayRequested = useMemo(
|
| 77 |
() => demoRequested || searchParams.get('autoplay') === '1',
|
| 78 |
[demoRequested, searchParams],
|
|
|
|
| 93 |
// Sound effects for phase transitions
|
| 94 |
const prevPhaseRef = useRef(phase);
|
| 95 |
const prevMsgCountRef = useRef(0);
|
| 96 |
+
const resultsRef = useRef<HTMLDivElement | null>(null);
|
| 97 |
|
| 98 |
useEffect(() => {
|
| 99 |
const prev = prevPhaseRef.current;
|
|
|
|
| 117 |
sfx.success();
|
| 118 |
fireSuccessConfetti();
|
| 119 |
}, 400);
|
| 120 |
+
toast('Episode complete - Agreement reached!', 'success');
|
| 121 |
} else if (verdict) {
|
| 122 |
setTimeout(() => sfx.failure(), 400);
|
| 123 |
+
toast(`Episode complete - Verdict: ${verdict}`, 'warning');
|
| 124 |
}
|
| 125 |
}
|
| 126 |
}, [phase, episode?.judge_audit?.verdict, toast]);
|
|
|
|
| 142 |
return () => stopAmbient();
|
| 143 |
}, []);
|
| 144 |
|
| 145 |
+
useEffect(() => {
|
| 146 |
+
if (episode?.done) {
|
| 147 |
+
resultsRef.current?.scrollIntoView({ behavior: 'smooth', block: 'start' });
|
| 148 |
+
}
|
| 149 |
+
}, [episode?.done]);
|
| 150 |
+
|
| 151 |
const handleStart = useCallback(async (params: ResetParams) => {
|
| 152 |
setLoading(true);
|
| 153 |
setIsJudging(false);
|
|
|
|
| 156 |
sfx.click();
|
| 157 |
try {
|
| 158 |
const state = await resetEpisode(params);
|
| 159 |
+
const nextState = demoCase ? { ...state, demo_case: demoCase } : state;
|
| 160 |
+
setEpisode(nextState);
|
| 161 |
prevMsgCountRef.current = 0;
|
| 162 |
toast('Episode started!', 'info');
|
| 163 |
// Feature 9: Update URL for shareable link
|
|
|
|
| 171 |
if (autoPlayRequested) {
|
| 172 |
nextSearch.set('autoplay', '1');
|
| 173 |
}
|
| 174 |
+
if (demoCase) {
|
| 175 |
+
nextSearch.set('demoCase', demoCase);
|
| 176 |
+
}
|
| 177 |
navigate(`/episode/${state.episode_id}?${nextSearch.toString()}`, { replace: true });
|
| 178 |
} catch (err) {
|
| 179 |
console.error('Failed to start episode:', err);
|
|
|
|
| 183 |
} finally {
|
| 184 |
setLoading(false);
|
| 185 |
}
|
| 186 |
+
}, [autoPlayRequested, demoCase, demoRequested, navigate, toast]);
|
| 187 |
|
| 188 |
const handleStepWithAction = useCallback(async (action?: ScientistAction) => {
|
| 189 |
if (!episode || episode.done) return;
|
|
|
|
| 196 |
|
| 197 |
try {
|
| 198 |
const isLastRound = episode.round >= episode.max_rounds - 1;
|
| 199 |
+
const finalAction = action ?? buildDemoScientistAction(episode, demoCase);
|
| 200 |
|
| 201 |
if (isLastRound && !action) {
|
| 202 |
setIsJudging(true);
|
|
|
|
| 223 |
} finally {
|
| 224 |
setLoading(false);
|
| 225 |
}
|
| 226 |
+
}, [demoCase, episode, toast]);
|
| 227 |
|
| 228 |
useEffect(() => {
|
| 229 |
if (!demoRequested || autoStartTriggered || episode || loading) {
|
|
|
|
| 413 |
))}
|
| 414 |
</motion.div>
|
| 415 |
|
| 416 |
+
{demoMeta && (
|
| 417 |
+
<motion.div
|
| 418 |
+
className="mb-4 rounded-xl border border-primary/20 bg-primary/5 p-4"
|
| 419 |
+
initial={{ opacity: 0, y: 8 }}
|
| 420 |
+
animate={{ opacity: 1, y: 0 }}
|
| 421 |
+
transition={{ delay: 0.14 }}
|
| 422 |
+
>
|
| 423 |
+
<div className="flex flex-col gap-2 md:flex-row md:items-center md:justify-between">
|
| 424 |
+
<div>
|
| 425 |
+
<div className="text-xs font-semibold uppercase tracking-[0.18em] text-primary">
|
| 426 |
+
{demoMeta.title}
|
| 427 |
+
</div>
|
| 428 |
+
<h2 className="mt-1 text-base font-semibold">{demoMeta.subtitle}</h2>
|
| 429 |
+
<p className="mt-1 text-sm text-muted-foreground">{demoMeta.summary}</p>
|
| 430 |
+
</div>
|
| 431 |
+
<div className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
|
| 432 |
+
Seed {episode.seed} · {episode.template.replace(/_/g, ' ')} · {episode.difficulty}
|
| 433 |
+
</div>
|
| 434 |
+
</div>
|
| 435 |
+
</motion.div>
|
| 436 |
+
)}
|
| 437 |
+
|
| 438 |
{!episode.done && episode.conversation.length === 0 && (
|
| 439 |
<motion.div
|
| 440 |
className="mb-4 rounded-lg border border-primary/20 bg-primary/5 p-4"
|
|
|
|
| 618 |
{episode.done && <ReplayViewer messages={episode.conversation} />}
|
| 619 |
</motion.div>
|
| 620 |
</div>
|
| 621 |
+
|
| 622 |
+
{episode.done && (
|
| 623 |
+
<motion.div
|
| 624 |
+
ref={resultsRef}
|
| 625 |
+
className="mt-6"
|
| 626 |
+
initial={{ opacity: 0, y: 12 }}
|
| 627 |
+
animate={{ opacity: 1, y: 0 }}
|
| 628 |
+
transition={{ delay: 0.35 }}
|
| 629 |
+
>
|
| 630 |
+
<EpisodeResultsReport episode={episode} />
|
| 631 |
+
</motion.div>
|
| 632 |
+
)}
|
| 633 |
</div>
|
| 634 |
);
|
| 635 |
}
|
frontend/src/pages/TrainingPage.tsx
ADDED
|
@@ -0,0 +1,365 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import type { ReactNode } from 'react';
|
| 2 |
+
import { motion } from 'framer-motion';
|
| 3 |
+
import {
|
| 4 |
+
BrainCircuit,
|
| 5 |
+
CheckCircle2,
|
| 6 |
+
FileCode2,
|
| 7 |
+
FlaskConical,
|
| 8 |
+
Gauge,
|
| 9 |
+
GraduationCap,
|
| 10 |
+
AlertTriangle,
|
| 11 |
+
Wrench,
|
| 12 |
+
TerminalSquare,
|
| 13 |
+
} from 'lucide-react';
|
| 14 |
+
import {
|
| 15 |
+
ResponsiveContainer,
|
| 16 |
+
LineChart,
|
| 17 |
+
Line,
|
| 18 |
+
CartesianGrid,
|
| 19 |
+
XAxis,
|
| 20 |
+
YAxis,
|
| 21 |
+
Tooltip,
|
| 22 |
+
Legend,
|
| 23 |
+
BarChart,
|
| 24 |
+
Bar,
|
| 25 |
+
} from 'recharts';
|
| 26 |
+
import {
|
| 27 |
+
HOLDOUT_COMPARE,
|
| 28 |
+
LAB_MANAGER_PREVIEW_ARTIFACT,
|
| 29 |
+
LIVE_CHECKPOINTS,
|
| 30 |
+
LOCAL_BASELINE_SUMMARY,
|
| 31 |
+
SCIENTIST_PREVIEW_ARTIFACT,
|
| 32 |
+
TRAINING_ASSESSMENT,
|
| 33 |
+
TRAINING_LOG_ROWS,
|
| 34 |
+
} from '@/data/trainingArtifacts';
|
| 35 |
+
import { cn, formatReward, formatScore } from '@/lib/utils';
|
| 36 |
+
|
| 37 |
+
const compareBars = [
|
| 38 |
+
{
|
| 39 |
+
metric: 'Avg reward',
|
| 40 |
+
baseline: HOLDOUT_COMPARE.summary.baseline_avg_reward,
|
| 41 |
+
trained: HOLDOUT_COMPARE.summary.trained_avg_reward,
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
metric: 'Agreement %',
|
| 45 |
+
baseline: HOLDOUT_COMPARE.summary.baseline_agreement_rate * 100,
|
| 46 |
+
trained: HOLDOUT_COMPARE.summary.trained_agreement_rate * 100,
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
metric: 'Invalid %',
|
| 50 |
+
baseline: HOLDOUT_COMPARE.summary.baseline_invalid_rate * 100,
|
| 51 |
+
trained: HOLDOUT_COMPARE.summary.trained_invalid_rate * 100,
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
metric: 'Avg rounds',
|
| 55 |
+
baseline: HOLDOUT_COMPARE.summary.baseline_avg_rounds,
|
| 56 |
+
trained: HOLDOUT_COMPARE.summary.trained_avg_rounds,
|
| 57 |
+
},
|
| 58 |
+
];
|
| 59 |
+
|
| 60 |
+
const checkpointChart = LIVE_CHECKPOINTS.map((checkpoint) => ({
|
| 61 |
+
label: checkpoint.label,
|
| 62 |
+
reward: Number(checkpoint.averageReward.toFixed(3)),
|
| 63 |
+
agreementPercent: Number((checkpoint.agreementRate * 100).toFixed(1)),
|
| 64 |
+
rounds: Number(checkpoint.averageRounds.toFixed(2)),
|
| 65 |
+
}));
|
| 66 |
+
|
| 67 |
+
export default function TrainingPage() {
|
| 68 |
+
const latestCheckpoint = LIVE_CHECKPOINTS[LIVE_CHECKPOINTS.length - 1];
|
| 69 |
+
|
| 70 |
+
return (
|
| 71 |
+
<div className="mx-auto max-w-screen-xl px-4 py-8">
|
| 72 |
+
<motion.div
|
| 73 |
+
className="mb-8 text-center"
|
| 74 |
+
initial={{ opacity: 0, y: -10 }}
|
| 75 |
+
animate={{ opacity: 1, y: 0 }}
|
| 76 |
+
>
|
| 77 |
+
<div className="mb-3 inline-flex items-center gap-2 rounded-full bg-primary/10 px-4 py-1.5 text-sm font-medium text-primary">
|
| 78 |
+
<GraduationCap className="h-4 w-4" />
|
| 79 |
+
Training Logs And Analysis
|
| 80 |
+
</div>
|
| 81 |
+
<h1 className="mb-2 text-3xl font-bold tracking-tight">Scientist And Lab Manager Training Status</h1>
|
| 82 |
+
<p className="mx-auto max-w-3xl text-muted-foreground">
|
| 83 |
+
This page is backed by the real ReplicaLab training artifacts already generated in the repo. It shows what
|
| 84 |
+
the training stack achieved, where it is still failing, and whether more training is needed before claiming
|
| 85 |
+
the agents are actually better.
|
| 86 |
+
</p>
|
| 87 |
+
</motion.div>
|
| 88 |
+
|
| 89 |
+
<motion.div
|
| 90 |
+
className="mb-6 grid gap-4 lg:grid-cols-4"
|
| 91 |
+
initial={{ opacity: 0, y: 10 }}
|
| 92 |
+
animate={{ opacity: 1, y: 0 }}
|
| 93 |
+
transition={{ delay: 0.05 }}
|
| 94 |
+
>
|
| 95 |
+
<StatusCard
|
| 96 |
+
icon={<FileCode2 className="h-4 w-4" />}
|
| 97 |
+
title="Scientist preview"
|
| 98 |
+
value={`${SCIENTIST_PREVIEW_ARTIFACT.datasetSize} prompts`}
|
| 99 |
+
detail={`${SCIENTIST_PREVIEW_ARTIFACT.modelName} · max steps ${SCIENTIST_PREVIEW_ARTIFACT.config.maxSteps}`}
|
| 100 |
+
/>
|
| 101 |
+
<StatusCard
|
| 102 |
+
icon={<FlaskConical className="h-4 w-4" />}
|
| 103 |
+
title="Lab Manager preview"
|
| 104 |
+
value={`${LAB_MANAGER_PREVIEW_ARTIFACT.datasetSize} samples`}
|
| 105 |
+
detail={`${LAB_MANAGER_PREVIEW_ARTIFACT.modelName} · ${LAB_MANAGER_PREVIEW_ARTIFACT.config.numTrainEpochs} epoch preview`}
|
| 106 |
+
/>
|
| 107 |
+
<StatusCard
|
| 108 |
+
icon={<Gauge className="h-4 w-4" />}
|
| 109 |
+
title="Latest live RL checkpoint"
|
| 110 |
+
value={formatReward(latestCheckpoint.averageReward)}
|
| 111 |
+
detail={`${formatScore(latestCheckpoint.agreementRate)} agreement at artifact step ${latestCheckpoint.artifactStep}`}
|
| 112 |
+
/>
|
| 113 |
+
<StatusCard
|
| 114 |
+
icon={<AlertTriangle className="h-4 w-4" />}
|
| 115 |
+
title="Do we need more training?"
|
| 116 |
+
value="Yes"
|
| 117 |
+
detail="Hold-out compare still favors baseline."
|
| 118 |
+
tone="warn"
|
| 119 |
+
/>
|
| 120 |
+
</motion.div>
|
| 121 |
+
|
| 122 |
+
<motion.div
|
| 123 |
+
className="mb-6 grid gap-4 lg:grid-cols-2"
|
| 124 |
+
initial={{ opacity: 0, y: 12 }}
|
| 125 |
+
animate={{ opacity: 1, y: 0 }}
|
| 126 |
+
transition={{ delay: 0.1 }}
|
| 127 |
+
>
|
| 128 |
+
<ChartCard
|
| 129 |
+
title="Live RL checkpoint progression"
|
| 130 |
+
subtitle="Real checkpoint summaries from the ART/OpenEnv scientist run."
|
| 131 |
+
>
|
| 132 |
+
<div className="h-72">
|
| 133 |
+
<ResponsiveContainer width="100%" height="100%">
|
| 134 |
+
<LineChart data={checkpointChart}>
|
| 135 |
+
<CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
|
| 136 |
+
<XAxis dataKey="label" tick={{ fontSize: 11 }} />
|
| 137 |
+
<YAxis yAxisId="left" tick={{ fontSize: 11 }} />
|
| 138 |
+
<YAxis
|
| 139 |
+
yAxisId="right"
|
| 140 |
+
orientation="right"
|
| 141 |
+
tick={{ fontSize: 11 }}
|
| 142 |
+
tickFormatter={(value) => `${value}%`}
|
| 143 |
+
/>
|
| 144 |
+
<Tooltip />
|
| 145 |
+
<Legend wrapperStyle={{ fontSize: '12px' }} />
|
| 146 |
+
<Line yAxisId="left" type="monotone" dataKey="reward" stroke="var(--color-primary)" strokeWidth={2.5} name="Avg reward" />
|
| 147 |
+
<Line yAxisId="right" type="monotone" dataKey="agreementPercent" stroke="var(--color-lab-manager)" strokeWidth={2.5} name="Agreement %" />
|
| 148 |
+
<Line yAxisId="left" type="monotone" dataKey="rounds" stroke="var(--color-judge)" strokeWidth={2} name="Avg rounds" />
|
| 149 |
+
</LineChart>
|
| 150 |
+
</ResponsiveContainer>
|
| 151 |
+
</div>
|
| 152 |
+
</ChartCard>
|
| 153 |
+
|
| 154 |
+
<ChartCard
|
| 155 |
+
title="Hold-out compare"
|
| 156 |
+
subtitle="Real seeded comparison between the deterministic baseline and the current trained scientist."
|
| 157 |
+
>
|
| 158 |
+
<div className="h-72">
|
| 159 |
+
<ResponsiveContainer width="100%" height="100%">
|
| 160 |
+
<BarChart data={compareBars}>
|
| 161 |
+
<CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
|
| 162 |
+
<XAxis dataKey="metric" tick={{ fontSize: 11 }} />
|
| 163 |
+
<YAxis tick={{ fontSize: 11 }} />
|
| 164 |
+
<Tooltip />
|
| 165 |
+
<Legend wrapperStyle={{ fontSize: '12px' }} />
|
| 166 |
+
<Bar dataKey="baseline" fill="var(--color-lab-manager)" radius={[6, 6, 0, 0]} name="Baseline" />
|
| 167 |
+
<Bar dataKey="trained" fill="var(--color-destructive)" radius={[6, 6, 0, 0]} name="Trained" />
|
| 168 |
+
</BarChart>
|
| 169 |
+
</ResponsiveContainer>
|
| 170 |
+
</div>
|
| 171 |
+
</ChartCard>
|
| 172 |
+
</motion.div>
|
| 173 |
+
|
| 174 |
+
<motion.div
|
| 175 |
+
className="mb-6 grid gap-4 lg:grid-cols-[1.35fr_1fr]"
|
| 176 |
+
initial={{ opacity: 0, y: 12 }}
|
| 177 |
+
animate={{ opacity: 1, y: 0 }}
|
| 178 |
+
transition={{ delay: 0.16 }}
|
| 179 |
+
>
|
| 180 |
+
<div className="rounded-xl border border-border bg-card p-5">
|
| 181 |
+
<div className="mb-4 flex items-center gap-2">
|
| 182 |
+
<TerminalSquare className="h-4 w-4 text-primary" />
|
| 183 |
+
<h2 className="text-base font-semibold">Training log highlights</h2>
|
| 184 |
+
</div>
|
| 185 |
+
<div className="space-y-3">
|
| 186 |
+
{TRAINING_LOG_ROWS.map((row) => (
|
| 187 |
+
<div key={row.label} className="rounded-lg border border-border bg-muted/30 p-3">
|
| 188 |
+
<div className="flex flex-wrap items-center gap-2 text-[11px] font-medium">
|
| 189 |
+
<span className="rounded-full bg-background px-2 py-1 text-primary">step {row.trainingStep}</span>
|
| 190 |
+
<span className="rounded-full bg-background px-2 py-1 text-primary">seed {row.seed}</span>
|
| 191 |
+
<span className={cn(
|
| 192 |
+
'rounded-full px-2 py-1',
|
| 193 |
+
row.verdict === 'accept' ? 'bg-lab-manager/10 text-lab-manager' : 'bg-destructive/10 text-destructive',
|
| 194 |
+
)}>
|
| 195 |
+
{row.verdict ?? 'no verdict'}
|
| 196 |
+
</span>
|
| 197 |
+
</div>
|
| 198 |
+
<div className="mt-2 font-medium">{row.paperTitle}</div>
|
| 199 |
+
<div className="mt-2 grid gap-2 text-xs text-muted-foreground sm:grid-cols-4">
|
| 200 |
+
<span>reward {formatReward(row.reward)}</span>
|
| 201 |
+
<span>rounds {row.roundsUsed}</span>
|
| 202 |
+
<span>invalid {row.invalidActionCount}</span>
|
| 203 |
+
<span>parse {row.parseErrorCount}</span>
|
| 204 |
+
</div>
|
| 205 |
+
<p className="mt-2 text-xs text-muted-foreground">{row.note}</p>
|
| 206 |
+
</div>
|
| 207 |
+
))}
|
| 208 |
+
</div>
|
| 209 |
+
</div>
|
| 210 |
+
|
| 211 |
+
<div className="space-y-4">
|
| 212 |
+
<SummaryCard
|
| 213 |
+
title="What we achieved"
|
| 214 |
+
icon={<CheckCircle2 className="h-4 w-4" />}
|
| 215 |
+
items={TRAINING_ASSESSMENT.achieved}
|
| 216 |
+
tone="good"
|
| 217 |
+
/>
|
| 218 |
+
<SummaryCard
|
| 219 |
+
title="Why the current model still fails"
|
| 220 |
+
icon={<BrainCircuit className="h-4 w-4" />}
|
| 221 |
+
items={TRAINING_ASSESSMENT.gaps}
|
| 222 |
+
tone="warn"
|
| 223 |
+
/>
|
| 224 |
+
<SummaryCard
|
| 225 |
+
title="What to improve next"
|
| 226 |
+
icon={<Wrench className="h-4 w-4" />}
|
| 227 |
+
items={TRAINING_ASSESSMENT.improvements}
|
| 228 |
+
tone="neutral"
|
| 229 |
+
/>
|
| 230 |
+
</div>
|
| 231 |
+
</motion.div>
|
| 232 |
+
|
| 233 |
+
<motion.div
|
| 234 |
+
className="grid gap-4 lg:grid-cols-3"
|
| 235 |
+
initial={{ opacity: 0, y: 12 }}
|
| 236 |
+
animate={{ opacity: 1, y: 0 }}
|
| 237 |
+
transition={{ delay: 0.22 }}
|
| 238 |
+
>
|
| 239 |
+
<InfoPanel
|
| 240 |
+
title="Local deterministic baseline"
|
| 241 |
+
lines={[
|
| 242 |
+
`${LOCAL_BASELINE_SUMMARY.episodeCount} episodes across all three scenario families`,
|
| 243 |
+
`Average reward ${formatReward(LOCAL_BASELINE_SUMMARY.averageReward)}`,
|
| 244 |
+
`Agreement ${formatScore(LOCAL_BASELINE_SUMMARY.agreementRate)}`,
|
| 245 |
+
`Average rounds ${LOCAL_BASELINE_SUMMARY.averageRounds.toFixed(1)}`,
|
| 246 |
+
]}
|
| 247 |
+
/>
|
| 248 |
+
<InfoPanel
|
| 249 |
+
title="Scientist run config"
|
| 250 |
+
lines={[
|
| 251 |
+
SCIENTIST_PREVIEW_ARTIFACT.modelName,
|
| 252 |
+
`LoRA rank ${SCIENTIST_PREVIEW_ARTIFACT.config.loraRank}`,
|
| 253 |
+
`Max seq length ${SCIENTIST_PREVIEW_ARTIFACT.config.maxSeqLength}`,
|
| 254 |
+
`Learning rate ${SCIENTIST_PREVIEW_ARTIFACT.config.learningRate}`,
|
| 255 |
+
]}
|
| 256 |
+
/>
|
| 257 |
+
<InfoPanel
|
| 258 |
+
title="Lab Manager status"
|
| 259 |
+
lines={[
|
| 260 |
+
`Preview dataset ${LAB_MANAGER_PREVIEW_ARTIFACT.datasetSize} samples`,
|
| 261 |
+
`Evidence pack ${LAB_MANAGER_PREVIEW_ARTIFACT.evidencePackVersion}`,
|
| 262 |
+
'Preview artifact exists, but no live evaluated SFT adapter is shown in the demo yet',
|
| 263 |
+
'Next step is a real trained-plus-evaluated Lab Manager run',
|
| 264 |
+
]}
|
| 265 |
+
/>
|
| 266 |
+
</motion.div>
|
| 267 |
+
</div>
|
| 268 |
+
);
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
function StatusCard({
|
| 272 |
+
icon,
|
| 273 |
+
title,
|
| 274 |
+
value,
|
| 275 |
+
detail,
|
| 276 |
+
tone = 'neutral',
|
| 277 |
+
}: {
|
| 278 |
+
icon: ReactNode;
|
| 279 |
+
title: string;
|
| 280 |
+
value: string;
|
| 281 |
+
detail: string;
|
| 282 |
+
tone?: 'neutral' | 'warn';
|
| 283 |
+
}) {
|
| 284 |
+
return (
|
| 285 |
+
<div className={cn(
|
| 286 |
+
'rounded-xl border bg-card p-4',
|
| 287 |
+
tone === 'warn' ? 'border-judge/30' : 'border-border',
|
| 288 |
+
)}>
|
| 289 |
+
<div className="mb-2 flex items-center gap-2 text-sm font-medium text-muted-foreground">
|
| 290 |
+
{icon}
|
| 291 |
+
{title}
|
| 292 |
+
</div>
|
| 293 |
+
<div className={cn('text-2xl font-bold', tone === 'warn' ? 'text-judge' : 'text-foreground')}>
|
| 294 |
+
{value}
|
| 295 |
+
</div>
|
| 296 |
+
<p className="mt-1 text-sm text-muted-foreground">{detail}</p>
|
| 297 |
+
</div>
|
| 298 |
+
);
|
| 299 |
+
}
|
| 300 |
+
|
| 301 |
+
function ChartCard({
|
| 302 |
+
title,
|
| 303 |
+
subtitle,
|
| 304 |
+
children,
|
| 305 |
+
}: {
|
| 306 |
+
title: string;
|
| 307 |
+
subtitle: string;
|
| 308 |
+
children: ReactNode;
|
| 309 |
+
}) {
|
| 310 |
+
return (
|
| 311 |
+
<div className="rounded-xl border border-border bg-card p-5">
|
| 312 |
+
<h2 className="text-base font-semibold">{title}</h2>
|
| 313 |
+
<p className="mt-1 text-sm text-muted-foreground">{subtitle}</p>
|
| 314 |
+
<div className="mt-4">{children}</div>
|
| 315 |
+
</div>
|
| 316 |
+
);
|
| 317 |
+
}
|
| 318 |
+
|
| 319 |
+
function SummaryCard({
|
| 320 |
+
title,
|
| 321 |
+
icon,
|
| 322 |
+
items,
|
| 323 |
+
tone,
|
| 324 |
+
}: {
|
| 325 |
+
title: string;
|
| 326 |
+
icon: ReactNode;
|
| 327 |
+
items: string[];
|
| 328 |
+
tone: 'good' | 'warn' | 'neutral';
|
| 329 |
+
}) {
|
| 330 |
+
return (
|
| 331 |
+
<div className={cn(
|
| 332 |
+
'rounded-xl border bg-card p-4',
|
| 333 |
+
tone === 'good' && 'border-lab-manager/30',
|
| 334 |
+
tone === 'warn' && 'border-judge/30',
|
| 335 |
+
tone === 'neutral' && 'border-border',
|
| 336 |
+
)}>
|
| 337 |
+
<div className="mb-3 flex items-center gap-2 text-sm font-semibold">
|
| 338 |
+
{icon}
|
| 339 |
+
{title}
|
| 340 |
+
</div>
|
| 341 |
+
<div className="space-y-2">
|
| 342 |
+
{items.map((item) => (
|
| 343 |
+
<div key={item} className="rounded-lg border border-border bg-muted/30 px-3 py-2 text-xs text-muted-foreground">
|
| 344 |
+
{item}
|
| 345 |
+
</div>
|
| 346 |
+
))}
|
| 347 |
+
</div>
|
| 348 |
+
</div>
|
| 349 |
+
);
|
| 350 |
+
}
|
| 351 |
+
|
| 352 |
+
function InfoPanel({ title, lines }: { title: string; lines: string[] }) {
|
| 353 |
+
return (
|
| 354 |
+
<div className="rounded-xl border border-border bg-card p-4">
|
| 355 |
+
<h2 className="text-sm font-semibold">{title}</h2>
|
| 356 |
+
<div className="mt-3 space-y-2">
|
| 357 |
+
{lines.map((line) => (
|
| 358 |
+
<div key={line} className="rounded-lg border border-border bg-muted/30 px-3 py-2 text-xs text-muted-foreground">
|
| 359 |
+
{line}
|
| 360 |
+
</div>
|
| 361 |
+
))}
|
| 362 |
+
</div>
|
| 363 |
+
</div>
|
| 364 |
+
);
|
| 365 |
+
}
|
frontend/src/types/index.ts
CHANGED
|
@@ -3,6 +3,7 @@ export type LabManagerActionType = 'report_feasibility' | 'suggest_alternative'
|
|
| 3 |
export type Difficulty = 'easy' | 'medium' | 'hard';
|
| 4 |
export type ScenarioTemplate = 'math_reasoning' | 'ml_benchmark' | 'finance_trading';
|
| 5 |
export type Role = 'scientist' | 'lab_manager' | 'judge' | 'system';
|
|
|
|
| 6 |
|
| 7 |
// --- Backend-aligned action contracts ---
|
| 8 |
|
|
@@ -103,6 +104,21 @@ export interface JudgeAudit {
|
|
| 103 |
score_breakdown: ScoreBreakdown;
|
| 104 |
}
|
| 105 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
// --- Frontend episode state (assembled from backend responses) ---
|
| 107 |
|
| 108 |
export interface EpisodeState {
|
|
@@ -120,6 +136,9 @@ export interface EpisodeState {
|
|
| 120 |
conversation: NegotiationMessage[];
|
| 121 |
scores: ScoreBreakdown | null;
|
| 122 |
judge_audit: JudgeAudit | null;
|
|
|
|
|
|
|
|
|
|
| 123 |
}
|
| 124 |
|
| 125 |
export interface ResetParams {
|
|
@@ -190,8 +209,12 @@ export interface BackendStepInfo {
|
|
| 190 |
verdict: string | null;
|
| 191 |
top_failure_reasons: string[];
|
| 192 |
round: number;
|
| 193 |
-
stub: boolean;
|
| 194 |
episode_id: string;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
}
|
| 196 |
|
| 197 |
export interface BackendStepResult {
|
|
|
|
| 3 |
export type Difficulty = 'easy' | 'medium' | 'hard';
|
| 4 |
export type ScenarioTemplate = 'math_reasoning' | 'ml_benchmark' | 'finance_trading';
|
| 5 |
export type Role = 'scientist' | 'lab_manager' | 'judge' | 'system';
|
| 6 |
+
export type DemoCase = 'fast-agreement' | 'learning-opportunity' | 'no-agreement';
|
| 7 |
|
| 8 |
// --- Backend-aligned action contracts ---
|
| 9 |
|
|
|
|
| 104 |
score_breakdown: ScoreBreakdown;
|
| 105 |
}
|
| 106 |
|
| 107 |
+
export interface EpisodeStepTrace {
|
| 108 |
+
round: number;
|
| 109 |
+
reward: number;
|
| 110 |
+
cumulative_reward: number;
|
| 111 |
+
action_type: string;
|
| 112 |
+
scientist_message: string;
|
| 113 |
+
lab_manager_action_type?: string;
|
| 114 |
+
lab_manager_message?: string;
|
| 115 |
+
step_reward_components: Record<string, number>;
|
| 116 |
+
protocol: Protocol | null;
|
| 117 |
+
oracle_round_score?: Record<string, unknown> | null;
|
| 118 |
+
oracle_post_mortem?: Record<string, unknown> | null;
|
| 119 |
+
oracle_event?: Record<string, unknown> | null;
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
// --- Frontend episode state (assembled from backend responses) ---
|
| 123 |
|
| 124 |
export interface EpisodeState {
|
|
|
|
| 136 |
conversation: NegotiationMessage[];
|
| 137 |
scores: ScoreBreakdown | null;
|
| 138 |
judge_audit: JudgeAudit | null;
|
| 139 |
+
cumulative_reward: number;
|
| 140 |
+
step_history: EpisodeStepTrace[];
|
| 141 |
+
demo_case?: DemoCase;
|
| 142 |
}
|
| 143 |
|
| 144 |
export interface ResetParams {
|
|
|
|
| 209 |
verdict: string | null;
|
| 210 |
top_failure_reasons: string[];
|
| 211 |
round: number;
|
|
|
|
| 212 |
episode_id: string;
|
| 213 |
+
step_reward_components?: Record<string, number>;
|
| 214 |
+
cumulative_reward?: number;
|
| 215 |
+
oracle_round_score?: Record<string, unknown> | null;
|
| 216 |
+
oracle_post_mortem?: Record<string, unknown> | null;
|
| 217 |
+
oracle_event?: Record<string, unknown> | null;
|
| 218 |
}
|
| 219 |
|
| 220 |
export interface BackendStepResult {
|
scripts/build_demo_video.py
ADDED
|
@@ -0,0 +1,476 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import shutil
|
| 6 |
+
import subprocess
|
| 7 |
+
import sys
|
| 8 |
+
import textwrap
|
| 9 |
+
import time
|
| 10 |
+
import urllib.parse
|
| 11 |
+
import urllib.request
|
| 12 |
+
from dataclasses import dataclass
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
from typing import Iterable
|
| 15 |
+
|
| 16 |
+
from PIL import Image, ImageDraw, ImageFont, ImageOps
|
| 17 |
+
from selenium import webdriver
|
| 18 |
+
from selenium.common.exceptions import TimeoutException
|
| 19 |
+
from selenium.webdriver.chrome.options import Options
|
| 20 |
+
from selenium.webdriver.common.by import By
|
| 21 |
+
from selenium.webdriver.support.ui import WebDriverWait
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 25 |
+
OUTPUT_DIR = ROOT / "replicalab" / "outputs" / "demo_video"
|
| 26 |
+
SCREENS_DIR = OUTPUT_DIR / "screens"
|
| 27 |
+
SLIDES_DIR = OUTPUT_DIR / "slides"
|
| 28 |
+
AUDIO_DIR = OUTPUT_DIR / "audio"
|
| 29 |
+
TEXT_DIR = OUTPUT_DIR / "text"
|
| 30 |
+
CHROME_PATH = Path(r"C:\Program Files\Google\Chrome\Application\chrome.exe")
|
| 31 |
+
BASE_URL = "http://127.0.0.1:7860"
|
| 32 |
+
ONBOARDING_KEY = "replicalab-onboarded"
|
| 33 |
+
THEME_KEY = "replicalab-theme"
|
| 34 |
+
VIDEO_SIZE = (1920, 1080)
|
| 35 |
+
NARRATION = (
|
| 36 |
+
"ReplicaLab starts from a research paper and turns it into a seeded replication benchmark. "
|
| 37 |
+
"The Scientist proposes a protocol, the Lab Manager enforces budget, tools, and scheduling, "
|
| 38 |
+
"and a deterministic Judge scores rigor, feasibility, and fidelity. In our first scenario, "
|
| 39 |
+
"the agents agree immediately, so the paper looks replicable in this lab. In the second scenario, "
|
| 40 |
+
"they negotiate across all six rounds, which creates a rich reinforcement learning signal. "
|
| 41 |
+
"In the third, they never resolve the blockers, so the system rejects the paper for the current setup. "
|
| 42 |
+
"Because every outcome is scored deterministically, we can train the Scientist with Unsloth and TRL, "
|
| 43 |
+
"compare baseline versus trained runs, inspect real logs, and see exactly where more learning is still needed. "
|
| 44 |
+
"The training page is intentionally honest: the live run reached positive rewards, but the held-out compare still "
|
| 45 |
+
"shows that the trained Scientist has not beaten the deterministic baseline yet."
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
@dataclass(frozen=True)
|
| 50 |
+
class Scene:
|
| 51 |
+
id: str
|
| 52 |
+
url: str
|
| 53 |
+
title: str
|
| 54 |
+
subtitle: str
|
| 55 |
+
duration: float
|
| 56 |
+
expected_text: str | None = None
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
SCENES: tuple[Scene, ...] = (
|
| 60 |
+
Scene(
|
| 61 |
+
id="dashboard",
|
| 62 |
+
url=f"{BASE_URL}/",
|
| 63 |
+
title="Paper to benchmark",
|
| 64 |
+
subtitle="ReplicaLab turns a paper into a seeded replication benchmark.",
|
| 65 |
+
duration=8.0,
|
| 66 |
+
expected_text="ReplicaLab",
|
| 67 |
+
),
|
| 68 |
+
Scene(
|
| 69 |
+
id="fast_agreement",
|
| 70 |
+
url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1&demoCase=fast-agreement",
|
| 71 |
+
title="Scenario 1: first-round agreement",
|
| 72 |
+
subtitle="The agents converge quickly and the paper scores as a strong replication candidate.",
|
| 73 |
+
duration=11.0,
|
| 74 |
+
expected_text="Completed: First-round agreement",
|
| 75 |
+
),
|
| 76 |
+
Scene(
|
| 77 |
+
id="learning_opportunity",
|
| 78 |
+
url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=202&demo=1&autoplay=1&demoCase=learning-opportunity",
|
| 79 |
+
title="Scenario 2: multi-round learning",
|
| 80 |
+
subtitle="Six rounds of disagreement create a rich RL signal before the final acceptance.",
|
| 81 |
+
duration=13.0,
|
| 82 |
+
expected_text="Completed: Multi-round learning opportunity",
|
| 83 |
+
),
|
| 84 |
+
Scene(
|
| 85 |
+
id="no_agreement",
|
| 86 |
+
url=f"{BASE_URL}/episode?template=ml_benchmark&difficulty=medium&seed=303&demo=1&autoplay=1&demoCase=no-agreement",
|
| 87 |
+
title="Scenario 3: no agreement",
|
| 88 |
+
subtitle="The blockers remain unresolved, so the system rejects replication for this setup.",
|
| 89 |
+
duration=12.0,
|
| 90 |
+
expected_text="Completed: No agreement reached",
|
| 91 |
+
),
|
| 92 |
+
Scene(
|
| 93 |
+
id="training",
|
| 94 |
+
url=f"{BASE_URL}/training",
|
| 95 |
+
title="Artifact-backed training review",
|
| 96 |
+
subtitle="The training page shows real checkpoints, real compare metrics, and what still needs improvement.",
|
| 97 |
+
duration=16.0,
|
| 98 |
+
expected_text="Training Logs And Analysis",
|
| 99 |
+
),
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def load_env_value(key: str) -> str | None:
|
| 104 |
+
if os.getenv(key):
|
| 105 |
+
return os.getenv(key)
|
| 106 |
+
|
| 107 |
+
for path in (ROOT / ".env", ROOT / ".env.local", ROOT / "frontend" / ".env"):
|
| 108 |
+
if not path.exists():
|
| 109 |
+
continue
|
| 110 |
+
for raw in path.read_text(encoding="utf-8").splitlines():
|
| 111 |
+
line = raw.strip()
|
| 112 |
+
if not line or line.startswith("#") or "=" not in line:
|
| 113 |
+
continue
|
| 114 |
+
env_key, env_value = line.split("=", 1)
|
| 115 |
+
if env_key.strip() == key:
|
| 116 |
+
return env_value.strip().strip('"').strip("'")
|
| 117 |
+
return None
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def ensure_backend() -> None:
|
| 121 |
+
try:
|
| 122 |
+
with urllib.request.urlopen(f"{BASE_URL}/health", timeout=10) as response:
|
| 123 |
+
data = json.loads(response.read().decode("utf-8"))
|
| 124 |
+
except Exception as exc: # pragma: no cover - user-facing failure
|
| 125 |
+
raise RuntimeError(
|
| 126 |
+
f"Backend unavailable at {BASE_URL}. Start it with "
|
| 127 |
+
"\"python -m uvicorn server.app:app --host 127.0.0.1 --port 7860\"."
|
| 128 |
+
) from exc
|
| 129 |
+
if data.get("status") != "ok":
|
| 130 |
+
raise RuntimeError(f"Unexpected backend health payload: {data}")
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def ensure_output_dirs() -> None:
|
| 134 |
+
for directory in (OUTPUT_DIR, SCREENS_DIR, SLIDES_DIR, AUDIO_DIR, TEXT_DIR):
|
| 135 |
+
directory.mkdir(parents=True, exist_ok=True)
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def get_font(size: int, bold: bool = False) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
|
| 139 |
+
candidates = [
|
| 140 |
+
Path(r"C:\Windows\Fonts\segoeuib.ttf" if bold else r"C:\Windows\Fonts\segoeui.ttf"),
|
| 141 |
+
Path(r"C:\Windows\Fonts\arialbd.ttf" if bold else r"C:\Windows\Fonts\arial.ttf"),
|
| 142 |
+
]
|
| 143 |
+
for candidate in candidates:
|
| 144 |
+
if candidate.exists():
|
| 145 |
+
return ImageFont.truetype(str(candidate), size=size)
|
| 146 |
+
return ImageFont.load_default()
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def pick_voice(api_key: str, preferred_voice_id: str | None) -> str:
|
| 150 |
+
if preferred_voice_id:
|
| 151 |
+
return preferred_voice_id
|
| 152 |
+
|
| 153 |
+
request = urllib.request.Request(
|
| 154 |
+
"https://api.elevenlabs.io/v1/voices",
|
| 155 |
+
headers={"xi-api-key": api_key, "Accept": "application/json"},
|
| 156 |
+
method="GET",
|
| 157 |
+
)
|
| 158 |
+
with urllib.request.urlopen(request, timeout=30) as response:
|
| 159 |
+
payload = json.loads(response.read().decode("utf-8"))
|
| 160 |
+
|
| 161 |
+
voices = payload.get("voices", [])
|
| 162 |
+
if not voices:
|
| 163 |
+
raise RuntimeError("ElevenLabs returned no voices for the current API key.")
|
| 164 |
+
|
| 165 |
+
preferred_names = ("Rachel", "Aria", "Sarah", "Charlie", "George")
|
| 166 |
+
for name in preferred_names:
|
| 167 |
+
for voice in voices:
|
| 168 |
+
if voice.get("name") == name and voice.get("voice_id"):
|
| 169 |
+
return voice["voice_id"]
|
| 170 |
+
for voice in voices:
|
| 171 |
+
if voice.get("voice_id"):
|
| 172 |
+
return voice["voice_id"]
|
| 173 |
+
raise RuntimeError("No usable ElevenLabs voice_id found.")
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
def synthesize_voiceover(api_key: str, voice_id: str, text: str, output_path: Path) -> None:
|
| 177 |
+
payload = {
|
| 178 |
+
"text": text,
|
| 179 |
+
"model_id": "eleven_multilingual_v2",
|
| 180 |
+
"voice_settings": {
|
| 181 |
+
"stability": 0.4,
|
| 182 |
+
"similarity_boost": 0.8,
|
| 183 |
+
"style": 0.25,
|
| 184 |
+
"use_speaker_boost": True,
|
| 185 |
+
},
|
| 186 |
+
}
|
| 187 |
+
body = json.dumps(payload).encode("utf-8")
|
| 188 |
+
urls = (
|
| 189 |
+
f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}?output_format=mp3_44100_128",
|
| 190 |
+
f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
|
| 191 |
+
)
|
| 192 |
+
|
| 193 |
+
last_error: Exception | None = None
|
| 194 |
+
for url in urls:
|
| 195 |
+
request = urllib.request.Request(
|
| 196 |
+
url,
|
| 197 |
+
data=body,
|
| 198 |
+
headers={
|
| 199 |
+
"xi-api-key": api_key,
|
| 200 |
+
"Accept": "audio/mpeg",
|
| 201 |
+
"Content-Type": "application/json",
|
| 202 |
+
},
|
| 203 |
+
method="POST",
|
| 204 |
+
)
|
| 205 |
+
try:
|
| 206 |
+
with urllib.request.urlopen(request, timeout=90) as response:
|
| 207 |
+
audio = response.read()
|
| 208 |
+
if not audio:
|
| 209 |
+
raise RuntimeError("ElevenLabs returned an empty audio payload.")
|
| 210 |
+
output_path.write_bytes(audio)
|
| 211 |
+
return
|
| 212 |
+
except Exception as exc: # pragma: no cover - fallback path
|
| 213 |
+
last_error = exc
|
| 214 |
+
raise RuntimeError(f"Failed to synthesize ElevenLabs audio: {last_error}")
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def build_driver() -> webdriver.Chrome:
|
| 218 |
+
options = Options()
|
| 219 |
+
options.binary_location = str(CHROME_PATH)
|
| 220 |
+
options.add_argument("--headless=new")
|
| 221 |
+
options.add_argument("--disable-gpu")
|
| 222 |
+
options.add_argument("--window-size=1600,1200")
|
| 223 |
+
options.add_argument("--force-device-scale-factor=1")
|
| 224 |
+
options.add_argument("--hide-scrollbars")
|
| 225 |
+
options.add_argument("--mute-audio")
|
| 226 |
+
options.add_argument("--no-first-run")
|
| 227 |
+
options.add_argument("--no-default-browser-check")
|
| 228 |
+
options.add_argument(f"--user-data-dir={OUTPUT_DIR / 'chrome_profile'}")
|
| 229 |
+
driver = webdriver.Chrome(options=options)
|
| 230 |
+
driver.set_window_size(1600, 1200)
|
| 231 |
+
return driver
|
| 232 |
+
|
| 233 |
+
|
| 234 |
+
def capture_screens() -> list[Path]:
|
| 235 |
+
driver = build_driver()
|
| 236 |
+
files: list[Path] = []
|
| 237 |
+
try:
|
| 238 |
+
driver.get(BASE_URL)
|
| 239 |
+
WebDriverWait(driver, 20).until(
|
| 240 |
+
lambda d: d.execute_script("return document.readyState") == "complete"
|
| 241 |
+
)
|
| 242 |
+
driver.execute_script(
|
| 243 |
+
"window.localStorage.setItem(arguments[0], '1');"
|
| 244 |
+
"window.localStorage.setItem(arguments[1], 'light');",
|
| 245 |
+
ONBOARDING_KEY,
|
| 246 |
+
THEME_KEY,
|
| 247 |
+
)
|
| 248 |
+
|
| 249 |
+
for scene in SCENES:
|
| 250 |
+
driver.get(scene.url)
|
| 251 |
+
WebDriverWait(driver, 30).until(
|
| 252 |
+
lambda d: d.execute_script("return document.readyState") == "complete"
|
| 253 |
+
)
|
| 254 |
+
if scene.expected_text:
|
| 255 |
+
try:
|
| 256 |
+
WebDriverWait(driver, 35).until(
|
| 257 |
+
lambda d: scene.expected_text in d.page_source
|
| 258 |
+
)
|
| 259 |
+
except TimeoutException:
|
| 260 |
+
pass
|
| 261 |
+
time.sleep(1.5)
|
| 262 |
+
output = SCREENS_DIR / f"{scene.id}.png"
|
| 263 |
+
driver.save_screenshot(str(output))
|
| 264 |
+
files.append(output)
|
| 265 |
+
finally:
|
| 266 |
+
driver.quit()
|
| 267 |
+
return files
|
| 268 |
+
|
| 269 |
+
|
| 270 |
+
def wrap_text(draw: ImageDraw.ImageDraw, text: str, font: ImageFont.ImageFont, width: int) -> list[str]:
|
| 271 |
+
words = text.split()
|
| 272 |
+
lines: list[str] = []
|
| 273 |
+
current = ""
|
| 274 |
+
for word in words:
|
| 275 |
+
candidate = f"{current} {word}".strip()
|
| 276 |
+
if draw.textlength(candidate, font=font) <= width:
|
| 277 |
+
current = candidate
|
| 278 |
+
else:
|
| 279 |
+
if current:
|
| 280 |
+
lines.append(current)
|
| 281 |
+
current = word
|
| 282 |
+
if current:
|
| 283 |
+
lines.append(current)
|
| 284 |
+
return lines
|
| 285 |
+
|
| 286 |
+
|
| 287 |
+
def create_slides() -> list[Path]:
|
| 288 |
+
title_font = get_font(46, bold=True)
|
| 289 |
+
subtitle_font = get_font(28)
|
| 290 |
+
badge_font = get_font(24, bold=True)
|
| 291 |
+
output_paths: list[Path] = []
|
| 292 |
+
|
| 293 |
+
for scene in SCENES:
|
| 294 |
+
raw_image = Image.open(SCREENS_DIR / f"{scene.id}.png").convert("RGB")
|
| 295 |
+
canvas = ImageOps.fit(raw_image, VIDEO_SIZE, method=Image.Resampling.LANCZOS)
|
| 296 |
+
overlay = Image.new("RGBA", VIDEO_SIZE, (0, 0, 0, 0))
|
| 297 |
+
draw = ImageDraw.Draw(overlay)
|
| 298 |
+
draw.rounded_rectangle((60, 780, 1860, 1020), radius=36, fill=(7, 13, 36, 190))
|
| 299 |
+
draw.rounded_rectangle((60, 56, 520, 116), radius=30, fill=(99, 102, 241, 220))
|
| 300 |
+
draw.text((92, 72), "ReplicaLab - 60 second demo", font=badge_font, fill=(255, 255, 255))
|
| 301 |
+
draw.text((96, 820), scene.title, font=title_font, fill=(255, 255, 255))
|
| 302 |
+
subtitle_lines = wrap_text(draw, scene.subtitle, subtitle_font, width=1620)
|
| 303 |
+
y = 888
|
| 304 |
+
for line in subtitle_lines:
|
| 305 |
+
draw.text((96, y), line, font=subtitle_font, fill=(226, 232, 240))
|
| 306 |
+
y += 40
|
| 307 |
+
|
| 308 |
+
final = Image.alpha_composite(canvas.convert("RGBA"), overlay).convert("RGB")
|
| 309 |
+
slide_path = SLIDES_DIR / f"{scene.id}.png"
|
| 310 |
+
final.save(slide_path, quality=95)
|
| 311 |
+
output_paths.append(slide_path)
|
| 312 |
+
return output_paths
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
def write_concat_file(paths: Iterable[Path]) -> Path:
|
| 316 |
+
concat_path = TEXT_DIR / "slides.txt"
|
| 317 |
+
lines: list[str] = []
|
| 318 |
+
ordered = list(paths)
|
| 319 |
+
for scene, path in zip(SCENES, ordered):
|
| 320 |
+
lines.append(f"file '{path.as_posix()}'")
|
| 321 |
+
lines.append(f"duration {scene.duration:.2f}")
|
| 322 |
+
lines.append(f"file '{ordered[-1].as_posix()}'")
|
| 323 |
+
concat_path.write_text("\n".join(lines), encoding="utf-8")
|
| 324 |
+
return concat_path
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
def write_script_assets() -> None:
|
| 328 |
+
(TEXT_DIR / "voiceover.txt").write_text(NARRATION, encoding="utf-8")
|
| 329 |
+
(TEXT_DIR / "shot_list.json").write_text(
|
| 330 |
+
json.dumps(
|
| 331 |
+
[
|
| 332 |
+
{
|
| 333 |
+
"id": scene.id,
|
| 334 |
+
"title": scene.title,
|
| 335 |
+
"subtitle": scene.subtitle,
|
| 336 |
+
"url": scene.url,
|
| 337 |
+
"duration_seconds": scene.duration,
|
| 338 |
+
}
|
| 339 |
+
for scene in SCENES
|
| 340 |
+
],
|
| 341 |
+
indent=2,
|
| 342 |
+
),
|
| 343 |
+
encoding="utf-8",
|
| 344 |
+
)
|
| 345 |
+
|
| 346 |
+
|
| 347 |
+
def seconds_to_srt(value: float) -> str:
|
| 348 |
+
millis = int(round(value * 1000))
|
| 349 |
+
hours, millis = divmod(millis, 3_600_000)
|
| 350 |
+
minutes, millis = divmod(millis, 60_000)
|
| 351 |
+
seconds, millis = divmod(millis, 1000)
|
| 352 |
+
return f"{hours:02}:{minutes:02}:{seconds:02},{millis:03}"
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def write_srt() -> None:
|
| 356 |
+
lines = wrap_text(ImageDraw.Draw(Image.new("RGB", (1, 1))), NARRATION, get_font(30), 72 * 18)
|
| 357 |
+
segment_count = max(1, len(lines))
|
| 358 |
+
total_duration = sum(scene.duration for scene in SCENES)
|
| 359 |
+
step = total_duration / segment_count
|
| 360 |
+
chunks = []
|
| 361 |
+
start = 0.0
|
| 362 |
+
for index, line in enumerate(lines, start=1):
|
| 363 |
+
end = min(total_duration, start + step)
|
| 364 |
+
chunks.append(f"{index}\n{seconds_to_srt(start)} --> {seconds_to_srt(end)}\n{line}\n")
|
| 365 |
+
start = end
|
| 366 |
+
(TEXT_DIR / "voiceover.srt").write_text("\n".join(chunks), encoding="utf-8")
|
| 367 |
+
|
| 368 |
+
|
| 369 |
+
def ffprobe_duration(path: Path) -> float:
|
| 370 |
+
result = subprocess.run(
|
| 371 |
+
[
|
| 372 |
+
"ffprobe",
|
| 373 |
+
"-v",
|
| 374 |
+
"error",
|
| 375 |
+
"-show_entries",
|
| 376 |
+
"format=duration",
|
| 377 |
+
"-of",
|
| 378 |
+
"default=noprint_wrappers=1:nokey=1",
|
| 379 |
+
str(path),
|
| 380 |
+
],
|
| 381 |
+
check=True,
|
| 382 |
+
capture_output=True,
|
| 383 |
+
text=True,
|
| 384 |
+
)
|
| 385 |
+
return float(result.stdout.strip())
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
def run_ffmpeg(audio_path: Path, concat_path: Path) -> Path:
|
| 389 |
+
silent_video = OUTPUT_DIR / "replicalab_demo_60s_silent.mp4"
|
| 390 |
+
final_video = OUTPUT_DIR / "replicalab_demo_60s.mp4"
|
| 391 |
+
|
| 392 |
+
subprocess.run(
|
| 393 |
+
[
|
| 394 |
+
"ffmpeg",
|
| 395 |
+
"-y",
|
| 396 |
+
"-f",
|
| 397 |
+
"concat",
|
| 398 |
+
"-safe",
|
| 399 |
+
"0",
|
| 400 |
+
"-i",
|
| 401 |
+
str(concat_path),
|
| 402 |
+
"-vf",
|
| 403 |
+
"fps=30,format=yuv420p",
|
| 404 |
+
"-c:v",
|
| 405 |
+
"libx264",
|
| 406 |
+
"-pix_fmt",
|
| 407 |
+
"yuv420p",
|
| 408 |
+
str(silent_video),
|
| 409 |
+
],
|
| 410 |
+
check=True,
|
| 411 |
+
)
|
| 412 |
+
subprocess.run(
|
| 413 |
+
[
|
| 414 |
+
"ffmpeg",
|
| 415 |
+
"-y",
|
| 416 |
+
"-i",
|
| 417 |
+
str(silent_video),
|
| 418 |
+
"-i",
|
| 419 |
+
str(audio_path),
|
| 420 |
+
"-c:v",
|
| 421 |
+
"copy",
|
| 422 |
+
"-c:a",
|
| 423 |
+
"aac",
|
| 424 |
+
"-b:a",
|
| 425 |
+
"192k",
|
| 426 |
+
"-shortest",
|
| 427 |
+
str(final_video),
|
| 428 |
+
],
|
| 429 |
+
check=True,
|
| 430 |
+
)
|
| 431 |
+
return final_video
|
| 432 |
+
|
| 433 |
+
|
| 434 |
+
def main() -> int:
|
| 435 |
+
ensure_backend()
|
| 436 |
+
ensure_output_dirs()
|
| 437 |
+
write_script_assets()
|
| 438 |
+
write_srt()
|
| 439 |
+
|
| 440 |
+
api_key = load_env_value("ELEVENLABS_API_KEY")
|
| 441 |
+
if not api_key:
|
| 442 |
+
raise RuntimeError("ELEVENLABS_API_KEY was not found in the environment or .env file.")
|
| 443 |
+
|
| 444 |
+
voice_id = pick_voice(api_key, load_env_value("ELEVENLABS_VOICE_ID"))
|
| 445 |
+
audio_path = AUDIO_DIR / "voiceover.mp3"
|
| 446 |
+
synthesize_voiceover(api_key, voice_id, NARRATION, audio_path)
|
| 447 |
+
|
| 448 |
+
capture_screens()
|
| 449 |
+
slides = create_slides()
|
| 450 |
+
concat_path = write_concat_file(slides)
|
| 451 |
+
final_video = run_ffmpeg(audio_path, concat_path)
|
| 452 |
+
|
| 453 |
+
metadata = {
|
| 454 |
+
"voice_id": voice_id,
|
| 455 |
+
"audio_duration_seconds": round(ffprobe_duration(audio_path), 3),
|
| 456 |
+
"video_path": str(final_video),
|
| 457 |
+
"slides": [str(path) for path in slides],
|
| 458 |
+
}
|
| 459 |
+
(TEXT_DIR / "build_metadata.json").write_text(json.dumps(metadata, indent=2), encoding="utf-8")
|
| 460 |
+
|
| 461 |
+
print(textwrap.dedent(f"""
|
| 462 |
+
Built demo assets:
|
| 463 |
+
audio: {audio_path}
|
| 464 |
+
video: {final_video}
|
| 465 |
+
script: {TEXT_DIR / 'voiceover.txt'}
|
| 466 |
+
subtitles: {TEXT_DIR / 'voiceover.srt'}
|
| 467 |
+
""").strip())
|
| 468 |
+
return 0
|
| 469 |
+
|
| 470 |
+
|
| 471 |
+
if __name__ == "__main__":
|
| 472 |
+
try:
|
| 473 |
+
raise SystemExit(main())
|
| 474 |
+
except Exception as exc: # pragma: no cover - CLI path
|
| 475 |
+
print(f"ERROR: {exc}", file=sys.stderr)
|
| 476 |
+
raise
|