Add model-driven local runtime and dynamic demo flow

docs/ayush/notes.md

@@ -38,3 +38,14 @@ Current ART/OpenEnv runtime note:
 - The main remaining work is experiment quality iteration, not missing training
   infrastructure.
 
+Current localhost model-runtime note:
+
+- `server/app.py` now exposes `/runtime` and `/agent-step` so the local app can run a backend-selected Scientist policy instead of the frontend stub.
+- Anthropic-backed Scientist inference was wired, but the current Anthropic account cannot be used live because the API billing balance is too low.
+- Localhost therefore currently runs in `ollama` mode with `glm-5:cloud` as the working model-backed Scientist path.
+- The server applies a small deterministic safety adapter to model outputs before env stepping:
+  - trims controls to fit sample size
+  - aligns equipment and reagent requests to the available inventory
+  - clamps duration to the current lab time limit
+- If the local model stalls or errors, `/agent-step` falls back to the deterministic baseline Scientist and records that in the step metadata as `scientist_runtime=ollama_fallback`.
+

docs/changes.md

@@ -83,4 +83,10 @@ Rules:
 | 2026-03-08 | Person B (Ayush) | Frontend backend-availability diagnostics | Added a startup health check and contextual network-error messaging to the replication setup flow after the live demo surfaced a generic `Failed to fetch` banner | The page gave no actionable explanation when the local API server on port `7860` was not running, which made a recoverable environment issue look like an application bug | `Controls.tsx` now checks backend health on load, `frontend/src/lib/api.ts` rewrites fetch-network failures into an explicit uvicorn startup instruction, the frontend was rebuilt, and the local FastAPI server was restarted and verified healthy on `127.0.0.1:7860` | Keep the backend running during demo prep or use the integrated backend-served frontend on `http://127.0.0.1:7860` |
 | 2026-03-08 | Person B (Ayush) | Frontend live demo outcomes and results report | Extended the one-click demo into three seeded story modes with a detailed post-episode report instead of a single autoplay path that stopped at the generic episode view | The hackathon demo needs to show distinct judged outcomes: immediate agreement, multi-round learning opportunity, and failure to reach agreement, all backed by real episode values rather than static mock copy | Added `fast-agreement`, `learning-opportunity`, and `no-agreement` dashboard launches; routed episode autoplay through scripted but backend-valid action sequences; added a results report with live reward charts, terminal score bars, training interpretation, reliability labeling, and tool-install suggestions; rebuilt the frontend and verified all three seeded ML runs against the live backend | If Oracle-backed narrative summaries are added later, keep the deterministic judge verdict and real score traces as the source of truth for the report |
 | 2026-03-08 | Person B (Ayush) | Frontend training-status reporting | Replaced the packaged training teaser with an artifact-backed training page and honest improvement guidance | The demo needs a place to show training logs, real achieved values, and whether more training is still required; the earlier dashboard card implied progress but did not expose the real run outputs | Added a dedicated `/training` route, header navigation, a shared frontend data module sourced from real run summaries, and a new training page with checkpoint charts, compare bars, log highlights, preview-artifact status, and explicit `needs more training` analysis; rebuilt the frontend and reverified backend serving on `/training` | If future runs improve beyond baseline, update the shared training artifact data module first so the dashboard and training page stay consistent |
+| 2026-03-08 | Person B (Ayush) | Demo video generation | Added a reproducible local builder for the one-minute demo video instead of relying on manual screen recording only | The demo now needs a fast, repeatable way to regenerate the final video with current UI states, a fresh voiceover, and ffmpeg assembly whenever the frontend story changes | Added `scripts/build_demo_video.py` plus `docs/demo_video_script_60s.md`; the script reads the ElevenLabs key from `.env`, captures the real dashboard, episode, and training screens with Selenium, synthesizes the voiceover, writes subtitles, and builds `replicalab/outputs/demo_video/replicalab_demo_60s.mp4` | If the narration or demo scenes change, rerun `python scripts/build_demo_video.py` to regenerate the assets from the current app state |
+| 2026-03-08 | Person B (Ayush) | Hugging Face Space deployment | Redeployed the live HF Space from the current local app state after the hosted URL was serving an old backend-only container | The Space repo and runtime SHAs had drifted behind the local `master` branch, so the public URL showed the API landing page instead of the React app even though the repo already contained the multi-stage Docker build and SPA-serving server code | Synced the deployment files to `ayushozha/replicalab` through the Hugging Face API, restarted the Space, and verified that `https://ayushozha-replicalab.hf.space/` now serves the built frontend while `/health` still reports the real environment | If the Space serves the API-only page again, compare the Space repo SHA and runtime SHA first before assuming the frontend build is broken |
+| 2026-03-08 | Person B (Ayush) | Frontend policy-results clarification | Added a separate baseline-vs-trained-vs-oracle page and clarified that the current public compare bench is still running the deterministic live runtime | The existing `/compare` page looked like a model-policy comparison, but it actually replays seeded benchmark episodes with the default Scientist action builder plus deterministic backend logic, which was confusing for the demo narrative | Added `/policies` as a dedicated policy-results page with live/runtime status, baseline vs trained artifact values, and an explicit oracle-not-mounted status; updated the header navigation and added a runtime clarification callout plus deep link on `/compare` | Keep `/compare` focused on seeded scenario benchmarking and use `/policies` when the audience asks whether the current app is actually running a trained or oracle-backed model |
+| 2026-03-08 | Person B (Ayush) | Localhost model-driven Scientist runtime | Added a backend-selected Scientist runtime path for localhost episodes and switched the live local mode from the blocked Anthropic path to Ollama | The repo needed a real localhost model-driven flow rather than the frontend default action builder, but the current Anthropic account cannot make live API calls because its credit balance is exhausted | Added `/runtime` and `/agent-step`, wired Anthropic and Ollama Scientist backends, made non-demo episode stepping prefer the backend model path, added a deterministic safety adapter plus baseline fallback for fragile local generations, and verified live localhost stepping with `glm-5:cloud` through Ollama | If Anthropic credits are replenished later, restart the backend with `REPLICALAB_SCIENTIST_RUNTIME=anthropic` to use that path instead |
+| 2026-03-08 | Person B (Ayush) | Frontend live-run randomness and judge semantics | Changed the default dashboard live run from one fixed scripted scenario to a random seeded paper episode, and split accepted-with-weaknesses presentation from outright failure presentation | The main demo CTA kept launching the same fixed `fast-agreement` route, which made the product feel canned, and the judge UI was showing `Accept` alongside `Failure Reasons`, which looked contradictory even though the backend semantics were agreement-based | The hero CTA now generates a fresh live route per click, the fixed outcome cards are explicitly labeled scripted, and accepted verdicts with residual gaps now render as `Accept with caveats` / `Conditional` instead of green accept plus red failure messaging | If the team later changes backend verdict semantics, keep the UI wording aligned so agreement and replicability remain separate concepts |
+| 2026-03-08 | Person B (Ayush) | Frontend caveat-state consistency | Tightened the remaining frontend success states so caveated accepts no longer behave like clean wins | After the judge-panel wording fix, the stage animation, first-round “good paper” label, and completion toast could still celebrate an accepted-but-weak protocol as a full success | `CharacterStage`, `EpisodePage`, and `EpisodeResultsReport` now treat accepted-with-caveats runs as partial outcomes, and live reset checks confirmed the dynamic route surfaces distinct paper briefs across scenario families when using the real reset contract | Keep any future verdict-label changes aligned across audit copy, stage emotion, toasts, and post-episode summaries |
 

docs/kush/notes.md

@@ -42,3 +42,51 @@ Durable deviations belong in `docs/changes.md`.
   - more training and parser/invalid-action cleanup are still needed
 - Header nav now includes `Training`, dashboard training CTA points there, and the dashboard training teaser uses the same artifact-backed data.
 
+## 2026-03-08 automated demo video build
+
+- Added `scripts/build_demo_video.py` to synthesize an ElevenLabs voiceover from `.env`, capture clean frontend screenshots, generate captioned slides, and build the final mp4 with `ffmpeg`.
+- Added `docs/demo_video_script_60s.md` as the canonical one-minute narration and shot list.
+- Generated the current outputs under `replicalab/outputs/demo_video/`:
+  - `audio/voiceover.mp3`
+  - `replicalab_demo_60s.mp4`
+  - `text/voiceover.txt`
+  - `text/voiceover.srt`
+
+## 2026-03-08 Hugging Face Space redeploy
+
+- Investigated the public Space after it showed only the backend landing page instead of the React app.
+- Confirmed the repo already had the correct multi-stage Dockerfile and SPA-serving `server/app.py`, but the runtime SHA was still pinned to an older backend-only container.
+- Synced the current app files to `ayushozha/replicalab` through the Hugging Face API, restarted the Space, and waited for the runtime SHA to advance to the new repo revision.
+- Reverified:
+  - `https://ayushozha-replicalab.hf.space/` now serves the React frontend
+  - `https://ayushozha-replicalab.hf.space/episode?...` returns `200`
+  - `https://ayushozha-replicalab.hf.space/health` still reports `{\"status\":\"ok\",\"env\":\"real\",\"version\":\"0.1.0\"}`
+
+## 2026-03-08 policy-results clarification page
+
+- Added a dedicated `/policies` frontend route for the question: baseline vs trained vs oracle.
+- The new page makes the current runtime explicit:
+  - `/compare` is still the seeded deterministic benchmark bench
+  - the public app is not currently mounting the trained Scientist adapter
+  - the public app is not currently mounting the Anthropic oracle path
+  - the Judge remains deterministic
+- Updated `/compare` with a callout so it no longer implies that it is already comparing live mounted model policies.
+
+## 2026-03-08 localhost model-backed Scientist mode
+
+- Added live runtime detection to the episode flow through `/runtime`.
+- Non-demo localhost episodes now prefer the backend `/agent-step` route instead of the frontend default action builder when a model runtime is available.
+- The episode page now surfaces the current Scientist runtime directly in the UI so it is clear whether localhost is using baseline or a model-backed path.
+- Current live localhost mode is `ollama` with `glm-5:cloud`.
+- Anthropic-backed Scientist mode exists in code, but the current Anthropic account cannot run live due to insufficient API credits, so localhost falls back to the Ollama runtime for real model-driven stepping.
+
+## 2026-03-08 dynamic live-run and judge-caveat cleanup
+
+- The main dashboard CTA no longer launches the same fixed seeded flow every time.
+- `Replicate a Random Paper` now generates a fresh seeded route with a random scenario family, difficulty, and seed, then autostarts the live episode path.
+- The three fixed cards remain available, but are now labeled as scripted outcomes rather than the default live experience.
+- Accepted verdicts that still carry weak-component reasons are now shown as `Accept with caveats` in the judge-facing UI instead of `Accept` plus a contradictory `Failure Reasons` block.
+- The results page now reports those cases as conditional replication candidates rather than clean wins.
+- The stage animation and completion toast now treat accepted-with-caveats runs as partial wins instead of full celebratory successes.
+- Live reset verification confirmed the random path can surface distinct paper briefs across scenario families, including CIFAR-10 replication and offline mean-reversion backtest cases.
+

frontend/src/App.tsx

@@ -2,6 +2,7 @@ import { Routes, Route } from 'react-router-dom';
 import EpisodePage from '@/pages/EpisodePage';
 import DashboardPage from '@/pages/DashboardPage';
 import ComparePage from '@/pages/ComparePage';
+import PolicyComparePage from '@/pages/PolicyComparePage';
 import TrainingPage from '@/pages/TrainingPage';
 import Header from '@/components/Header';
 import { ToastProvider } from '@/components/Toast';
@@ -20,6 +21,7 @@ export default function App() {
           <Route path="/episode/:episodeId" element={<EpisodePage />} />
           <Route path="/training" element={<TrainingPage />} />
           <Route path="/compare" element={<ComparePage />} />
+          <Route path="/policies" element={<PolicyComparePage />} />
         </Routes>
         <Onboarding show={showOnboarding} onDismiss={dismissOnboarding} />
       </div>

frontend/src/components/CharacterStage.tsx

@@ -32,6 +32,8 @@ export default function CharacterStage({
 }: CharacterStageProps) {
   const lastAction = getActionFromMessage(lastMessage);
   const speakingRole = lastMessage?.role;
+  const hasAcceptCaveats =
+    judgeAudit?.verdict === 'accept' && (judgeAudit.top_failure_reasons?.length ?? 0) > 0;
 
   const scientistAction =
     speakingRole === 'scientist' ? lastAction : undefined;
@@ -42,7 +44,11 @@ export default function CharacterStage({
     phase === 'judging'
       ? 'scoring'
       : phase === 'complete' && judgeAudit
-        ? `verdict_${judgeAudit.verdict}`
+        ? hasAcceptCaveats
+          ? 'verdict_partial'
+          : judgeAudit.verdict === 'accept' || judgeAudit.verdict === 'success'
+            ? 'verdict_success'
+            : 'verdict_failure'
         : undefined;
 
   return (

frontend/src/components/Controls.tsx

@@ -1,6 +1,6 @@
 import { useEffect, useState } from 'react';
 import { Play, RotateCcw, Dices } from 'lucide-react';
-import type { Difficulty, ScenarioTemplate, ResetParams } from '@/types';
+import type { BackendRuntimeStatus, Difficulty, ScenarioTemplate, ResetParams } from '@/types';
 import { cn } from '@/lib/utils';
 import { sfx } from '@/lib/audio';
 import { healthCheck } from '@/lib/api';
@@ -26,6 +26,7 @@ interface ControlsProps {
   initialSeed?: number;
   initialTemplate?: ScenarioTemplate;
   initialDifficulty?: Difficulty;
+  runtimeStatus?: BackendRuntimeStatus | null;
 }
 
 export default function Controls({
@@ -37,6 +38,7 @@ export default function Controls({
   initialSeed,
   initialTemplate,
   initialDifficulty,
+  runtimeStatus,
 }: ControlsProps) {
   const [seed, setSeed] = useState<string>(initialSeed?.toString() ?? '42');
   const [template, setTemplate] = useState<ScenarioTemplate>(initialTemplate ?? 'ml_benchmark');
@@ -109,6 +111,12 @@ export default function Controls({
         >
           {backendMessage}
         </p>
+        {runtimeStatus && (
+          <p className="mt-1 text-xs text-muted-foreground">
+            Scientist runtime: <span className="font-medium text-foreground">{runtimeStatus.scientist_runtime}</span>
+            {' '}({runtimeStatus.scientist_model})
+          </p>
+        )}
       </div>
       <div className="space-y-3">
         <div>

frontend/src/components/EpisodeResultsReport.tsx

@@ -37,6 +37,10 @@ type OutcomeProfile = {
   icon: typeof CheckCircle2;
 };
 
+function hasAcceptCaveats(episode: EpisodeState): boolean {
+  return episode.judge_audit?.verdict === 'accept' && (episode.judge_audit?.top_failure_reasons.length ?? 0) > 0;
+}
+
 const TOOL_STACK: Record<string, { title: string; packages: string[]; commands: string[] }> = {
   math_reasoning: {
     title: 'Suggested proof-and-verification stack',
@@ -78,19 +82,24 @@ function computeConfidencePercent(episode: EpisodeState): number {
 }
 
 function buildReliabilityLabel(episode: EpisodeState): string {
-  if (episode.judge_audit?.verdict === 'accept' && episode.step_history[0]?.lab_manager_action_type === 'accept') {
-    return 'Good';
+  if (episode.judge_audit?.verdict !== 'accept') {
+    return 'Bad';
   }
-  if (episode.judge_audit?.verdict === 'accept') {
-    return 'Needs iteration';
+  if (hasAcceptCaveats(episode)) {
+    return 'Conditional';
+  }
+  if (episode.step_history[0]?.lab_manager_action_type === 'accept') {
+    return 'Good';
   }
-  return 'Bad';
+  return 'Needs iteration';
 }
 
 function buildOutcomeProfile(episode: EpisodeState): OutcomeProfile {
+  const acceptedWithCaveats = hasAcceptCaveats(episode);
   const firstRoundAccepted =
     episode.step_history[0]?.lab_manager_action_type === 'accept' &&
-    episode.judge_audit?.verdict === 'accept';
+    episode.judge_audit?.verdict === 'accept' &&
+    !acceptedWithCaveats;
   const disagreementRounds = episode.step_history.filter(
     (trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
   ).length;
@@ -105,6 +114,16 @@ function buildOutcomeProfile(episode: EpisodeState): OutcomeProfile {
     };
   }
 
+  if (acceptedWithCaveats) {
+    return {
+      tone: 'learning',
+      title: 'Completed: Accepted with caveats',
+      subtitle: 'The agents reached agreement, but the final plan still has visible gaps against the hidden reference requirements.',
+      badge: 'Conditional replication candidate',
+      icon: AlertTriangle,
+    };
+  }
+
   if (episode.judge_audit?.verdict === 'accept') {
     return {
       tone: 'learning',
@@ -203,6 +222,7 @@ export default function EpisodeResultsReport({ episode, className }: EpisodeResu
   const profile = buildOutcomeProfile(episode);
   const confidencePercent = computeConfidencePercent(episode);
   const reliabilityLabel = buildReliabilityLabel(episode);
+  const acceptedWithCaveats = hasAcceptCaveats(episode);
   const disagreementRounds = episode.step_history.filter(
     (trace) => trace.lab_manager_action_type && trace.lab_manager_action_type !== 'accept',
   ).length;
@@ -240,7 +260,9 @@ export default function EpisodeResultsReport({ episode, className }: EpisodeResu
           <p className="mt-2 max-w-3xl text-sm text-muted-foreground">{profile.subtitle}</p>
           <p className="mt-3 max-w-3xl text-sm">
             {profile.tone !== 'reject'
-              ? `ReplicaLab scores this paper as replicable in the current lab setup with ${confidencePercent}% confidence.`
+              ? acceptedWithCaveats
+                ? `ReplicaLab scores this paper as conditionally replicable in the current lab setup with ${confidencePercent}% confidence. The caveats below should be addressed before claiming a clean reproduction.`
+                : `ReplicaLab scores this paper as replicable in the current lab setup with ${confidencePercent}% confidence.`
               : `ReplicaLab rejects this paper for the current setup. The paper reliability score is ${confidencePercent}% until the blocking constraints are addressed.`}
           </p>
         </div>
@@ -256,13 +278,13 @@ export default function EpisodeResultsReport({ episode, className }: EpisodeResu
             icon={<CheckCircle2 className="h-4 w-4" />}
             label="Paper reliability quality"
             value={reliabilityLabel}
-            hint="Good, learning opportunity, or bad based on the judged outcome."
+            hint="Good, conditional, learning opportunity, or bad based on the judged outcome."
           />
           <MetricCard
             icon={<FileCheck2 className="h-4 w-4" />}
             label="Judge verdict"
-            value={episode.judge_audit?.verdict ?? 'unknown'}
-            hint="Canonical deterministic decision."
+            value={acceptedWithCaveats ? 'accept_with_caveats' : episode.judge_audit?.verdict ?? 'unknown'}
+            hint="Deterministic decision with caveats separated from outright rejection."
           />
           <MetricCard
             icon={<TrendingUp className="h-4 w-4" />}

frontend/src/components/Header.tsx

@@ -1,6 +1,6 @@
 import { useState } from 'react';
 import { Link, useLocation } from 'react-router-dom';
-import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows, GraduationCap } from 'lucide-react';
+import { FlaskConical, LayoutDashboard, Play, Sun, Moon, Volume2, VolumeX, HelpCircle, GitCompareArrows, GraduationCap, BrainCircuit } from 'lucide-react';
 import { cn } from '@/lib/utils';
 import { useTheme } from '@/lib/useTheme';
 import { toggleMute, isMuted } from '@/lib/audio';
@@ -10,6 +10,7 @@ const navItems = [
   { to: '/episode', label: 'Episode', icon: Play },
   { to: '/training', label: 'Training', icon: GraduationCap },
   { to: '/compare', label: 'Compare', icon: GitCompareArrows },
+  { to: '/policies', label: 'Policies', icon: BrainCircuit },
 ];
 
 export default function Header({ onShowTutorial }: { onShowTutorial?: () => void }) {

frontend/src/components/JudgeAuditPanel.tsx

@@ -1,4 +1,4 @@
-import { CheckCircle, XCircle, AlertCircle } from 'lucide-react';
+import { CheckCircle, XCircle, AlertCircle, AlertTriangle } from 'lucide-react';
 import type { JudgeAudit } from '@/types';
 import { cn, verdictColor } from '@/lib/utils';
 import CharacterAvatar from '@/components/CharacterAvatar';
@@ -11,12 +11,20 @@ interface JudgeAuditPanelProps {
 export default function JudgeAuditPanel({ audit, className }: JudgeAuditPanelProps) {
   if (!audit) return null;
 
+  const hasCaveats = audit.verdict === 'accept' && audit.top_failure_reasons.length > 0;
   const VerdictIcon =
-    audit.verdict === 'success' || audit.verdict === 'accept'
+    hasCaveats
+      ? AlertTriangle
+      : audit.verdict === 'success' || audit.verdict === 'accept'
       ? CheckCircle
       : audit.verdict === 'failure' || audit.verdict === 'reject'
         ? XCircle
         : AlertCircle;
+  const verdictLabel = hasCaveats
+    ? 'Accept with caveats'
+    : audit.verdict.charAt(0).toUpperCase() + audit.verdict.slice(1);
+  const reasonsLabel = hasCaveats ? 'Caveats to address' : 'Failure Reasons';
+  const reasonsColor = hasCaveats ? 'text-judge' : 'text-destructive';
 
   return (
     <div className={cn('rounded-lg border border-judge/30 bg-judge/5 p-4', className)}>
@@ -24,9 +32,9 @@ export default function JudgeAuditPanel({ audit, className }: JudgeAuditPanelPro
         <CharacterAvatar role="judge" size="sm" />
         <div>
           <h2 className="text-sm font-semibold">Judge Aldric's Verdict</h2>
-          <div className={cn('flex items-center gap-1 text-xs font-semibold', verdictColor(audit.verdict))}>
+          <div className={cn('flex items-center gap-1 text-xs font-semibold', hasCaveats ? 'text-judge' : verdictColor(audit.verdict))}>
             <VerdictIcon className="h-3.5 w-3.5" />
-            {audit.verdict.charAt(0).toUpperCase() + audit.verdict.slice(1)}
+            {verdictLabel}
           </div>
         </div>
       </div>
@@ -47,10 +55,10 @@ export default function JudgeAuditPanel({ audit, className }: JudgeAuditPanelPro
 
       {audit.top_failure_reasons.length > 0 && (
         <div>
-          <h3 className="mb-1.5 text-xs font-medium text-destructive">Failure Reasons</h3>
+          <h3 className={cn('mb-1.5 text-xs font-medium', reasonsColor)}>{reasonsLabel}</h3>
           <ul className="space-y-1">
             {audit.top_failure_reasons.map((reason, i) => (
-              <li key={i} className="flex items-start gap-1.5 text-sm text-destructive">
+              <li key={i} className={cn('flex items-start gap-1.5 text-sm', reasonsColor)}>
                 <XCircle className="mt-0.5 h-3 w-3 shrink-0" />
                 {reason}
               </li>

frontend/src/data/trainingArtifacts.ts

@@ -40,6 +40,21 @@ export interface PreviewArtifact {
   };
 }
 
+export interface PolicySnapshot {
+  id: 'baseline' | 'trained' | 'oracle';
+  label: string;
+  scientistMode: string;
+  labManagerMode: string;
+  judgeMode: string;
+  source: string;
+  status: 'live' | 'artifact' | 'planned';
+  averageReward: number | null;
+  agreementRate: number | null;
+  averageRounds: number | null;
+  invalidRate: number | null;
+  summary: string;
+}
+
 export const HOLDOUT_COMPARE: TrainingComparison = {
   baseline: [
     { episode: 1, reward: 4.925, rigor: 0.85, feasibility: 1.0, fidelity: 0.45, rounds_used: 2, agreement: true, invalid_actions: 0 },
@@ -208,3 +223,61 @@ export const TRAINING_ASSESSMENT = {
     'Finish the Lab Manager SFT run and evaluate Scientist-plus-Lab-Manager together instead of only Scientist RL.',
   ],
 };
+
+export const POLICY_COMPARE: PolicySnapshot[] = [
+  {
+    id: 'baseline',
+    label: 'Baseline runtime',
+    scientistMode: 'Deterministic frontend action builder, not a mounted LLM adapter',
+    labManagerMode: 'Deterministic feasibility pipeline in the backend',
+    judgeMode: 'Deterministic rubric and audit',
+    source: 'Live runtime and local baseline evaluation',
+    status: 'live',
+    averageReward: HOLDOUT_COMPARE.summary.baseline_avg_reward,
+    agreementRate: HOLDOUT_COMPARE.summary.baseline_agreement_rate,
+    averageRounds: HOLDOUT_COMPARE.summary.baseline_avg_rounds,
+    invalidRate: HOLDOUT_COMPARE.summary.baseline_invalid_rate,
+    summary:
+      'This is the policy path used by the current /compare page. It reaches agreement reliably and stays fully judge-grounded, but it is not yet using the trained Scientist adapter.',
+  },
+  {
+    id: 'trained',
+    label: 'Trained Scientist',
+    scientistMode: 'Artifact-backed Scientist RL adapter evaluation',
+    labManagerMode: 'Deterministic feasibility pipeline in the backend',
+    judgeMode: 'Deterministic rubric and audit',
+    source: 'Hold-out compare artifact from the training pipeline',
+    status: 'artifact',
+    averageReward: HOLDOUT_COMPARE.summary.trained_avg_reward,
+    agreementRate: HOLDOUT_COMPARE.summary.trained_agreement_rate,
+    averageRounds: HOLDOUT_COMPARE.summary.trained_avg_rounds,
+    invalidRate: HOLDOUT_COMPARE.summary.trained_invalid_rate,
+    summary:
+      'The training pipeline ran successfully, but this adapter still underperforms the baseline badly on held-out seeded evaluation because invalid actions remain too high.',
+  },
+  {
+    id: 'oracle',
+    label: 'Oracle-assisted V2',
+    scientistMode: 'Planned Anthropic-assisted path, not mounted in the public runtime',
+    labManagerMode: 'Optional oracle narration and post-mortem path exists in code, not live in demo runtime',
+    judgeMode: 'Still deterministic even when oracle features are enabled',
+    source: 'Architecture target only, no committed evaluation artifact yet',
+    status: 'planned',
+    averageReward: null,
+    agreementRate: null,
+    averageRounds: null,
+    invalidRate: null,
+    summary:
+      'The oracle path exists in the codebase as a V2 extension, but there is no live public run or artifact-backed benchmark result wired into the app yet, so we should not claim oracle gains here.',
+  },
+];
+
+export const CURRENT_RUNTIME_MODEL_STATUS = {
+  comparePageUsesLiveModel: false,
+  episodePageUsesLiveModel: false,
+  backendUsesOracle: false,
+  backendUsesDeterministicLabManager: true,
+  backendUsesDeterministicJudge: true,
+  note:
+    'Right now the public demo runtime is not loading a trained Scientist adapter or an Anthropic oracle. The Scientist moves come from the frontend default action builder or the protocol editor, while the backend Lab Manager and Judge stay deterministic.',
+};

frontend/src/lib/api.ts

@@ -16,6 +16,7 @@ import type {
   ScenarioTemplate,
   Difficulty,
   EpisodeStepTrace,
+  BackendRuntimeStatus,
 } from '@/types';
 
 const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? '/api';
@@ -157,7 +158,7 @@ function buildRoundTrace(
 // REST API functions
 // ---------------------------------------------------------------------------
 
-export async function healthCheck(): Promise<{ status: string }> {
+export async function healthCheck(): Promise<{ status: string; env?: string; version?: string }> {
   try {
     const res = await fetch(`${BASE_URL}/health`);
     if (!res.ok) {
@@ -169,6 +170,18 @@ export async function healthCheck(): Promise<{ status: string }> {
   }
 }
 
+export async function getRuntimeStatus(): Promise<BackendRuntimeStatus> {
+  try {
+    const res = await fetch(`${BASE_URL}/runtime`);
+    if (!res.ok) {
+      throw new Error(`Failed to fetch runtime status: ${res.status}`);
+    }
+    return res.json();
+  } catch (error) {
+    throw normalizeFetchError(error, 'Failed to fetch runtime status');
+  }
+}
+
 export async function getScenarios(): Promise<BackendScenarioFamily[]> {
   try {
     const res = await fetch(`${BASE_URL}/scenarios`);
@@ -271,6 +284,62 @@ export async function stepEpisode(
   };
 }
 
+export async function agentStepEpisode(
+  sessionId: string,
+  prevState: EpisodeState,
+): Promise<EpisodeState> {
+  let data: BackendStepResult;
+  try {
+    const res = await fetch(`${BASE_URL}/agent-step`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ session_id: sessionId }),
+    });
+    if (!res.ok) {
+      const text = await res.text();
+      throw new Error(`Failed to run model-backed scientist step: ${text}`);
+    }
+    data = await res.json();
+  } catch (error) {
+    throw normalizeFetchError(error, 'Failed to run model-backed scientist step');
+  }
+
+  let scores: ScoreBreakdown | null = null;
+  let judgeAudit: JudgeAudit | null = null;
+  if (data.done && data.info.reward_breakdown) {
+    scores = adaptRewardBreakdown(data.info.reward_breakdown, data.reward);
+    judgeAudit = {
+      verdict: data.info.verdict ?? 'unknown',
+      judge_notes: data.info.judge_notes ? [data.info.judge_notes] : [],
+      top_failure_reasons: data.info.top_failure_reasons,
+      score_breakdown: scores,
+    };
+  }
+
+  const obs = data.observation;
+  const conversation = obs?.scientist
+    ? adaptConversation(obs.scientist.conversation_history)
+    : prevState.conversation;
+  const protocol = obs?.scientist?.current_protocol ?? prevState.protocol;
+  const round = obs?.scientist?.round_number ?? prevState.round + 1;
+  const cumulativeReward = data.info.cumulative_reward ?? prevState.cumulative_reward + data.reward;
+  const labConstraints = obs ? adaptLabConstraints(obs) : prevState.lab_constraints;
+  const roundTrace = buildRoundTrace(prevState, data);
+
+  return {
+    ...prevState,
+    round,
+    done: data.done,
+    protocol,
+    conversation,
+    lab_constraints: labConstraints,
+    scores,
+    judge_audit: judgeAudit,
+    cumulative_reward: cumulativeReward,
+    step_history: [...prevState.step_history, roundTrace],
+  };
+}
+
 export async function getReplay(episodeId: string): Promise<unknown> {
   try {
     const res = await fetch(`${BASE_URL}/replay/${episodeId}`);

frontend/src/lib/demo.ts

@@ -33,6 +33,20 @@ export const DEMO_CASES: DemoCaseMeta[] = [
   },
 ];
 
+const LIVE_TEMPLATES = ['math_reasoning', 'ml_benchmark', 'finance_trading'] as const;
+const LIVE_DIFFICULTIES = ['easy', 'medium', 'medium', 'hard'] as const;
+
+function sampleRandom<T>(items: readonly T[]): T {
+  return items[Math.floor(Math.random() * items.length)];
+}
+
+export function buildLiveEpisodePath(): string {
+  const template = sampleRandom(LIVE_TEMPLATES);
+  const difficulty = sampleRandom(LIVE_DIFFICULTIES);
+  const seed = Math.floor(Math.random() * 9000) + 1000;
+  return `/episode?template=${template}&difficulty=${difficulty}&seed=${seed}&autostart=1&autoplay=1`;
+}
+
 export function parseDemoCase(value: string | null): DemoCase | undefined {
   return DEMO_CASES.find((item) => item.id === value)?.id;
 }

frontend/src/pages/ComparePage.tsx

@@ -1,5 +1,6 @@
 import { motion } from 'framer-motion';
 import { GitCompareArrows } from 'lucide-react';
+import { Link } from 'react-router-dom';
 import EpisodeComparison from '@/components/EpisodeComparison';
 
 export default function ComparePage() {
@@ -42,6 +43,25 @@ export default function ComparePage() {
         </div>
       </motion.div>
 
+      <motion.div
+        className="mb-6 rounded-lg border border-judge/30 bg-judge/5 p-4"
+        initial={{ opacity: 0, y: 10 }}
+        animate={{ opacity: 1, y: 0 }}
+        transition={{ delay: 0.15 }}
+      >
+        <h2 className="text-sm font-semibold text-judge">Important: what is running here</h2>
+        <p className="mt-1 text-sm text-muted-foreground">
+          This page currently benchmarks the live deterministic runtime. It does not mount the trained Scientist adapter
+          or the Anthropic oracle in the public app yet.
+        </p>
+        <Link
+          to="/policies"
+          className="mt-3 inline-flex items-center rounded-md bg-primary px-3 py-1.5 text-xs font-medium text-primary-foreground transition-colors hover:bg-primary/90"
+        >
+          Open baseline vs trained vs oracle
+        </Link>
+      </motion.div>
+
       <motion.div
         initial={{ opacity: 0, y: 20 }}
         animate={{ opacity: 1, y: 0 }}

frontend/src/pages/DashboardPage.tsx

@@ -1,5 +1,5 @@
-import { Suspense } from 'react';
-import { Link } from 'react-router-dom';
+import { Suspense, useCallback } from 'react';
+import { Link, useNavigate } from 'react-router-dom';
 import {
   FlaskConical,
   Play,
@@ -19,7 +19,7 @@ import AnimatedCharacter from '@/components/AnimatedCharacter';
 import MoleculeScene from '@/components/MoleculeScene';
 import TiltCard from '@/components/TiltCard';
 import { cn } from '@/lib/utils';
-import { DEMO_CASES } from '@/lib/demo';
+import { buildLiveEpisodePath, DEMO_CASES } from '@/lib/demo';
 
 const SCENARIOS = [
   {
@@ -105,6 +105,12 @@ const FLOW = [
 ];
 
 export default function DashboardPage() {
+  const navigate = useNavigate();
+
+  const launchDynamicEpisode = useCallback(() => {
+    navigate(buildLiveEpisodePath());
+  }, [navigate]);
+
   return (
     <div className="mx-auto max-w-screen-xl px-4 py-8">
       {/* Hero with 3D background */}
@@ -179,13 +185,14 @@ export default function DashboardPage() {
           </div>
 
           <div className="mt-8 flex items-center justify-center gap-3">
-            <Link
-              to="/episode?template=ml_benchmark&difficulty=medium&seed=101&demo=1&autoplay=1&demoCase=fast-agreement"
+            <button
+              type="button"
+              onClick={launchDynamicEpisode}
               className="inline-flex items-center gap-2 rounded-lg bg-primary px-5 py-2.5 text-sm font-medium text-primary-foreground transition-colors hover:bg-primary/90 shadow-lg shadow-primary/25"
             >
               <Play className="h-4 w-4" />
-              Replicate a Paper
-            </Link>
+              Replicate a Random Paper
+            </button>
             <Link
               to="/training"
               className="inline-flex items-center gap-2 rounded-lg border border-border bg-background/80 backdrop-blur-sm px-5 py-2.5 text-sm font-medium text-foreground transition-colors hover:bg-muted"
@@ -204,6 +211,9 @@ export default function DashboardPage() {
               </div>
             ))}
           </div>
+          <p className="mx-auto mt-3 max-w-2xl text-xs text-muted-foreground">
+            The main button now launches a different seeded paper-derived benchmark each time. Use the cards below only when you want a fixed scripted outcome for the demo.
+          </p>
         </div>
       </section>
 
@@ -301,9 +311,9 @@ export default function DashboardPage() {
       </section>
 
       <section className="mb-16">
-        <h2 className="mb-2 text-center text-xl font-semibold">Live Demo Outcomes</h2>
+        <h2 className="mb-2 text-center text-xl font-semibold">Scripted Demo Outcomes</h2>
         <p className="mb-8 text-center text-sm text-muted-foreground">
-          Pick the exact story you want to show: immediate agreement, multi-round learning, or clear rejection.
+          Use these only when you want a fixed story: immediate agreement, multi-round learning, or clear rejection.
         </p>
         <div className="grid gap-4 md:grid-cols-3">
           {DEMO_CASES.map((demo) => {

frontend/src/pages/EpisodePage.tsx

@@ -2,8 +2,8 @@ import { useState, useCallback, useMemo, useEffect, useRef, Suspense } from 'rea
 import { useNavigate, useSearchParams } from 'react-router-dom';
 import { motion, AnimatePresence } from 'framer-motion';
 import { FileText, FlaskConical, Scale, TrendingUp, Play, Pencil } from 'lucide-react';
-import type { EpisodeState, ResetParams, ScientistAction } from '@/types';
-import { resetEpisode, stepEpisode } from '@/lib/api';
+import type { BackendRuntimeStatus, EpisodeState, ResetParams, ScientistAction } from '@/types';
+import { agentStepEpisode, getRuntimeStatus, resetEpisode, stepEpisode } from '@/lib/api';
 import { sfx, startAmbient, stopAmbient } from '@/lib/audio';
 import { fireSuccessConfetti, fireGavelConfetti } from '@/lib/confetti';
 import { buildDemoScientistAction, DEMO_CASES, parseDemoCase } from '@/lib/demo';
@@ -40,6 +40,7 @@ export default function EpisodePage() {
   const [isJudging, setIsJudging] = useState(false);
   const [error, setError] = useState<string | null>(null);
   const [autoStartTriggered, setAutoStartTriggered] = useState(false);
+  const [runtimeStatus, setRuntimeStatus] = useState<BackendRuntimeStatus | null>(null);
 
   // Feature 5: Auto-play
   const [autoPlaying, setAutoPlaying] = useState(false);
@@ -64,18 +65,26 @@ export default function EpisodePage() {
     const parsed = Number.parseInt(value, 10);
     return Number.isNaN(parsed) ? undefined : parsed;
   }, [searchParams]);
-  const demoRequested = useMemo(
-    () => searchParams.get('demo') === '1' || searchParams.get('autostart') === '1',
+  const scriptedDemoRequested = useMemo(
+    () => searchParams.get('demo') === '1',
     [searchParams],
   );
+  const autoStartRequested = useMemo(
+    () => scriptedDemoRequested || searchParams.get('autostart') === '1',
+    [scriptedDemoRequested, searchParams],
+  );
   const demoCase = useMemo(() => parseDemoCase(searchParams.get('demoCase')), [searchParams]);
   const demoMeta = useMemo(
     () => DEMO_CASES.find((item) => item.id === demoCase),
     [demoCase],
   );
   const autoPlayRequested = useMemo(
-    () => demoRequested || searchParams.get('autoplay') === '1',
-    [demoRequested, searchParams],
+    () => autoStartRequested || searchParams.get('autoplay') === '1',
+    [autoStartRequested, searchParams],
+  );
+  const backendModelStepAvailable = useMemo(
+    () => !scriptedDemoRequested && Boolean(runtimeStatus?.agent_step_available),
+    [scriptedDemoRequested, runtimeStatus?.agent_step_available],
   );
 
   const phase = useMemo(() => {
@@ -98,6 +107,9 @@ export default function EpisodePage() {
   useEffect(() => {
     const prev = prevPhaseRef.current;
     prevPhaseRef.current = phase;
+    const hasAcceptCaveats =
+      episode?.judge_audit?.verdict === 'accept' &&
+      (episode.judge_audit?.top_failure_reasons.length ?? 0) > 0;
 
     if (prev === 'waiting' && phase === 'negotiating') {
       sfx.episodeStart();
@@ -112,18 +124,21 @@ export default function EpisodePage() {
       stopAmbient();
       sfx.scoreReveal();
       const verdict = episode?.judge_audit?.verdict;
-      if (verdict === 'accept' || verdict === 'success') {
+      if ((verdict === 'accept' || verdict === 'success') && !hasAcceptCaveats) {
         setTimeout(() => {
           sfx.success();
           fireSuccessConfetti();
         }, 400);
         toast('Episode complete - Agreement reached!', 'success');
+      } else if (verdict === 'accept' && hasAcceptCaveats) {
+        setTimeout(() => sfx.roundTick(), 400);
+        toast('Episode complete - Accepted with caveats', 'warning');
       } else if (verdict) {
         setTimeout(() => sfx.failure(), 400);
         toast(`Episode complete - Verdict: ${verdict}`, 'warning');
       }
     }
-  }, [phase, episode?.judge_audit?.verdict, toast]);
+  }, [phase, episode?.judge_audit, toast]);
 
   useEffect(() => {
     if (!episode?.conversation.length) return;
@@ -142,6 +157,28 @@ export default function EpisodePage() {
     return () => stopAmbient();
   }, []);
 
+  useEffect(() => {
+    let cancelled = false;
+
+    async function loadRuntimeStatus() {
+      try {
+        const status = await getRuntimeStatus();
+        if (!cancelled) {
+          setRuntimeStatus(status);
+        }
+      } catch (runtimeError) {
+        if (!cancelled) {
+          console.warn('Failed to load runtime status', runtimeError);
+        }
+      }
+    }
+
+    void loadRuntimeStatus();
+    return () => {
+      cancelled = true;
+    };
+  }, []);
+
   useEffect(() => {
     if (episode?.done) {
       resultsRef.current?.scrollIntoView({ behavior: 'smooth', block: 'start' });
@@ -165,9 +202,12 @@ export default function EpisodePage() {
       nextSearch.set('template', state.template);
       nextSearch.set('difficulty', state.difficulty);
       nextSearch.set('seed', String(state.seed));
-      if (demoRequested) {
+      if (scriptedDemoRequested) {
         nextSearch.set('demo', '1');
       }
+      if (autoStartRequested && !scriptedDemoRequested) {
+        nextSearch.set('autostart', '1');
+      }
       if (autoPlayRequested) {
         nextSearch.set('autoplay', '1');
       }
@@ -183,7 +223,7 @@ export default function EpisodePage() {
     } finally {
       setLoading(false);
     }
-  }, [autoPlayRequested, demoCase, demoRequested, navigate, toast]);
+  }, [autoPlayRequested, autoStartRequested, demoCase, navigate, scriptedDemoRequested, toast]);
 
   const handleStepWithAction = useCallback(async (action?: ScientistAction) => {
     if (!episode || episode.done) return;
@@ -203,7 +243,9 @@ export default function EpisodePage() {
         await new Promise((r) => setTimeout(r, 2000));
       }
 
-      const state = await stepEpisode(episode.session_id, finalAction, episode);
+      const state = !action && backendModelStepAvailable
+        ? await agentStepEpisode(episode.session_id, episode)
+        : await stepEpisode(episode.session_id, finalAction, episode);
 
       if (state.done && !isLastRound) {
         setIsJudging(true);
@@ -223,10 +265,10 @@ export default function EpisodePage() {
     } finally {
       setLoading(false);
     }
-  }, [demoCase, episode, toast]);
+  }, [backendModelStepAvailable, demoCase, episode, toast]);
 
   useEffect(() => {
-    if (!demoRequested || autoStartTriggered || episode || loading) {
+    if (!autoStartRequested || autoStartTriggered || episode || loading) {
       return;
     }
     setAutoStartTriggered(true);
@@ -237,7 +279,7 @@ export default function EpisodePage() {
     });
   }, [
     autoStartTriggered,
-    demoRequested,
+    autoStartRequested,
     episode,
     handleStart,
     initialDifficulty,
@@ -318,11 +360,11 @@ export default function EpisodePage() {
 
           <motion.div className="mb-8 text-center" initial={{ opacity: 0 }} animate={{ opacity: 1 }} transition={{ delay: 0.4 }}>
             <h1 className="mb-2 text-2xl font-bold">
-              {demoRequested ? 'Launching Live Replication Demo' : 'Start a Paper Replication Episode'}
+              {autoStartRequested ? 'Launching Live Replication Run' : 'Start a Paper Replication Episode'}
             </h1>
             <p className="text-muted-foreground">
-              {demoRequested
-                ? 'Loading the seeded benchmark, starting the agents, and running the negotiation automatically.'
+              {autoStartRequested
+                ? 'Loading the benchmark, starting the agents, and running the negotiation automatically.'
                 : 'Pick a seeded paper-derived benchmark, parse it into the environment, and watch the agents negotiate a reproducible plan.'}
             </p>
             <ShortcutHint className="mt-2" />
@@ -334,7 +376,7 @@ export default function EpisodePage() {
             </motion.div>
           )}
 
-          {!demoRequested && (
+          {!autoStartRequested && (
             <motion.div className="w-full max-w-sm" initial={{ opacity: 0, y: 10 }} animate={{ opacity: 1, y: 0 }} transition={{ delay: 0.5 }}>
               <Controls
                 onStart={handleStart}
@@ -343,6 +385,7 @@ export default function EpisodePage() {
                 initialSeed={initialSeed}
                 initialTemplate={initialTemplate}
                 initialDifficulty={initialDifficulty}
+                runtimeStatus={runtimeStatus}
               />
             </motion.div>
           )}
@@ -413,7 +456,7 @@ export default function EpisodePage() {
         ))}
       </motion.div>
 
-      {demoMeta && (
+      {demoMeta && scriptedDemoRequested && (
         <motion.div
           className="mb-4 rounded-xl border border-primary/20 bg-primary/5 p-4"
           initial={{ opacity: 0, y: 8 }}
@@ -435,6 +478,34 @@ export default function EpisodePage() {
         </motion.div>
       )}
 
+      {!scriptedDemoRequested && runtimeStatus && (
+        <motion.div
+          className="mb-4 rounded-xl border border-emerald-500/20 bg-emerald-500/5 p-4"
+          initial={{ opacity: 0, y: 8 }}
+          animate={{ opacity: 1, y: 0 }}
+          transition={{ delay: 0.16 }}
+        >
+          <div className="flex flex-col gap-2 md:flex-row md:items-center md:justify-between">
+            <div>
+              <div className="text-xs font-semibold uppercase tracking-[0.18em] text-emerald-600">
+                Scientist Runtime
+              </div>
+              <h2 className="mt-1 text-base font-semibold">
+                {(runtimeStatus.scientist_runtime === 'anthropic' || runtimeStatus.scientist_runtime === 'ollama') && runtimeStatus.scientist_ready
+                  ? 'Localhost is using a model-backed Scientist policy.'
+                  : runtimeStatus.scientist_runtime === 'anthropic' || runtimeStatus.scientist_runtime === 'ollama'
+                    ? 'A model runtime is configured, but it is not ready.'
+                    : 'Localhost is using the deterministic baseline Scientist policy.'}
+              </h2>
+              <p className="mt-1 text-sm text-muted-foreground">{runtimeStatus.note}</p>
+            </div>
+            <div className="rounded-lg border border-border bg-background px-3 py-2 text-xs text-muted-foreground">
+              {runtimeStatus.scientist_runtime} · {runtimeStatus.scientist_model}
+            </div>
+          </div>
+        </motion.div>
+      )}
+
       {!episode.done && episode.conversation.length === 0 && (
         <motion.div
           className="mb-4 rounded-lg border border-primary/20 bg-primary/5 p-4"
@@ -519,6 +590,7 @@ export default function EpisodePage() {
             onStep={!episode.done ? handleStep : undefined}
             disabled={loading}
             episodeActive={true}
+            runtimeStatus={runtimeStatus}
           />
 
           {/* Feature 5: Auto-play controls */}

frontend/src/pages/PolicyComparePage.tsx

@@ -0,0 +1,244 @@
+import type { ReactNode } from 'react';
+import { motion } from 'framer-motion';
+import { Bot, BrainCircuit, CheckCircle2, FlaskConical, ShieldAlert } from 'lucide-react';
+import {
+  ResponsiveContainer,
+  BarChart,
+  Bar,
+  CartesianGrid,
+  XAxis,
+  YAxis,
+  Tooltip,
+  Legend,
+} from 'recharts';
+import { CURRENT_RUNTIME_MODEL_STATUS, POLICY_COMPARE } from '@/data/trainingArtifacts';
+import { cn, formatReward, formatScore } from '@/lib/utils';
+
+const measuredPolicies = POLICY_COMPARE.filter((policy) => policy.averageReward !== null);
+const chartRows = measuredPolicies.map((policy) => ({
+  label: policy.label,
+  reward: Number((policy.averageReward ?? 0).toFixed(2)),
+  agreement: Number(((policy.agreementRate ?? 0) * 100).toFixed(1)),
+  invalid: Number(((policy.invalidRate ?? 0) * 100).toFixed(1)),
+}));
+
+export default function PolicyComparePage() {
+  return (
+    <div className="mx-auto max-w-screen-xl px-4 py-8">
+      <motion.div
+        className="mb-8 text-center"
+        initial={{ opacity: 0, y: -10 }}
+        animate={{ opacity: 1, y: 0 }}
+      >
+        <div className="mb-3 inline-flex items-center gap-2 rounded-full bg-primary/10 px-4 py-1.5 text-sm font-medium text-primary">
+          <BrainCircuit className="h-4 w-4" />
+          Baseline Vs Trained Vs Oracle
+        </div>
+        <h1 className="mb-2 text-3xl font-bold tracking-tight">Policy Runtime And Results</h1>
+        <p className="mx-auto max-w-3xl text-muted-foreground">
+          This page separates three things that were easy to conflate in the demo: the live deterministic baseline
+          runtime, the trained Scientist artifact, and the planned oracle-assisted V2 path.
+        </p>
+      </motion.div>
+
+      <motion.div
+        className="mb-6 rounded-xl border border-judge/30 bg-judge/5 p-5"
+        initial={{ opacity: 0, y: 8 }}
+        animate={{ opacity: 1, y: 0 }}
+        transition={{ delay: 0.05 }}
+      >
+        <div className="mb-2 flex items-center gap-2 text-sm font-semibold text-judge">
+          <ShieldAlert className="h-4 w-4" />
+          Are we even running a model right now?
+        </div>
+        <p className="text-sm text-muted-foreground">{CURRENT_RUNTIME_MODEL_STATUS.note}</p>
+        <div className="mt-4 grid gap-3 md:grid-cols-2 xl:grid-cols-4">
+          <RuntimeFlag
+            label="Compare page"
+            value={CURRENT_RUNTIME_MODEL_STATUS.comparePageUsesLiveModel ? 'Model-backed' : 'Deterministic runtime'}
+            positive={CURRENT_RUNTIME_MODEL_STATUS.comparePageUsesLiveModel}
+          />
+          <RuntimeFlag
+            label="Episode page"
+            value={CURRENT_RUNTIME_MODEL_STATUS.episodePageUsesLiveModel ? 'Model-backed' : 'Deterministic runtime'}
+            positive={CURRENT_RUNTIME_MODEL_STATUS.episodePageUsesLiveModel}
+          />
+          <RuntimeFlag
+            label="Oracle path"
+            value={CURRENT_RUNTIME_MODEL_STATUS.backendUsesOracle ? 'Enabled' : 'Disabled in public runtime'}
+            positive={CURRENT_RUNTIME_MODEL_STATUS.backendUsesOracle}
+          />
+          <RuntimeFlag
+            label="Judge"
+            value={CURRENT_RUNTIME_MODEL_STATUS.backendUsesDeterministicJudge ? 'Deterministic' : 'Model-scored'}
+            positive={CURRENT_RUNTIME_MODEL_STATUS.backendUsesDeterministicJudge}
+          />
+        </div>
+      </motion.div>
+
+      <motion.div
+        className="mb-6 grid gap-4 lg:grid-cols-3"
+        initial={{ opacity: 0, y: 10 }}
+        animate={{ opacity: 1, y: 0 }}
+        transition={{ delay: 0.1 }}
+      >
+        {POLICY_COMPARE.map((policy) => (
+          <PolicyCard key={policy.id} policy={policy} />
+        ))}
+      </motion.div>
+
+      <motion.div
+        className="mb-6 rounded-xl border border-border bg-card p-5"
+        initial={{ opacity: 0, y: 12 }}
+        animate={{ opacity: 1, y: 0 }}
+        transition={{ delay: 0.15 }}
+      >
+        <div className="mb-4">
+          <h2 className="text-base font-semibold">Measured policies only</h2>
+          <p className="mt-1 text-sm text-muted-foreground">
+            The chart below only includes policy variants that already have actual numeric results. The oracle lane is
+            intentionally excluded until a real evaluation artifact exists.
+          </p>
+        </div>
+        <div className="h-72">
+          <ResponsiveContainer width="100%" height="100%">
+            <BarChart data={chartRows}>
+              <CartesianGrid strokeDasharray="3 3" stroke="var(--color-border)" />
+              <XAxis dataKey="label" tick={{ fontSize: 11 }} />
+              <YAxis tick={{ fontSize: 11 }} />
+              <Tooltip />
+              <Legend wrapperStyle={{ fontSize: '12px' }} />
+              <Bar dataKey="reward" fill="var(--color-primary)" radius={[6, 6, 0, 0]} name="Avg reward" />
+              <Bar dataKey="agreement" fill="var(--color-lab-manager)" radius={[6, 6, 0, 0]} name="Agreement %" />
+              <Bar dataKey="invalid" fill="var(--color-destructive)" radius={[6, 6, 0, 0]} name="Invalid %" />
+            </BarChart>
+          </ResponsiveContainer>
+        </div>
+      </motion.div>
+
+      <motion.div
+        className="rounded-xl border border-border bg-card p-5"
+        initial={{ opacity: 0, y: 12 }}
+        animate={{ opacity: 1, y: 0 }}
+        transition={{ delay: 0.2 }}
+      >
+        <h2 className="mb-4 text-base font-semibold">What each lane actually means</h2>
+        <div className="grid gap-4 lg:grid-cols-3">
+          <MeaningCard
+            icon={<FlaskConical className="h-4 w-4" />}
+            title="Baseline"
+            body="This is the current live runtime. It uses the default Scientist action builder plus the deterministic Lab Manager and Judge. It is stable, but it is not a trained LLM policy."
+          />
+          <MeaningCard
+            icon={<Bot className="h-4 w-4" />}
+            title="Trained"
+            body="This lane uses the artifact-backed Scientist training results. The adapter exists and was evaluated, but it still loses badly to the deterministic baseline on hold-out seeds."
+          />
+          <MeaningCard
+            icon={<CheckCircle2 className="h-4 w-4" />}
+            title="Oracle"
+            body="This is the planned Anthropic-assisted V2 lane. The code path exists, but the public app is not currently mounting it and there is no benchmark result we should claim yet."
+          />
+        </div>
+      </motion.div>
+    </div>
+  );
+}
+
+function RuntimeFlag({
+  label,
+  value,
+  positive,
+}: {
+  label: string;
+  value: string;
+  positive: boolean;
+}) {
+  return (
+    <div className="rounded-lg border border-border bg-background px-3 py-3">
+      <div className="text-xs font-medium text-muted-foreground">{label}</div>
+      <div className={cn('mt-1 text-sm font-semibold', positive ? 'text-lab-manager' : 'text-judge')}>
+        {value}
+      </div>
+    </div>
+  );
+}
+
+function PolicyCard({
+  policy,
+}: {
+  policy: (typeof POLICY_COMPARE)[number];
+}) {
+  const tone =
+    policy.status === 'live'
+      ? 'border-lab-manager/30'
+      : policy.status === 'artifact'
+        ? 'border-judge/30'
+        : 'border-border';
+  const badgeTone =
+    policy.status === 'live'
+      ? 'bg-lab-manager/10 text-lab-manager'
+      : policy.status === 'artifact'
+        ? 'bg-judge/10 text-judge'
+        : 'bg-muted text-muted-foreground';
+
+  return (
+    <div className={cn('rounded-xl border bg-card p-5', tone)}>
+      <div className="mb-3 flex items-start justify-between gap-3">
+        <div>
+          <h2 className="text-base font-semibold">{policy.label}</h2>
+          <p className="mt-1 text-xs text-muted-foreground">{policy.source}</p>
+        </div>
+        <span className={cn('rounded-full px-2 py-1 text-[11px] font-medium', badgeTone)}>
+          {policy.status}
+        </span>
+      </div>
+
+      <div className="mb-4 grid grid-cols-2 gap-2">
+        <MetricTile label="Avg reward" value={policy.averageReward === null ? 'Not run' : formatReward(policy.averageReward)} />
+        <MetricTile label="Agreement" value={policy.agreementRate === null ? 'Not run' : formatScore(policy.agreementRate)} />
+        <MetricTile label="Avg rounds" value={policy.averageRounds === null ? 'Not run' : policy.averageRounds.toFixed(1)} />
+        <MetricTile label="Invalid rate" value={policy.invalidRate === null ? 'Not run' : formatScore(policy.invalidRate)} />
+      </div>
+
+      <div className="space-y-2 text-xs text-muted-foreground">
+        <div><span className="font-semibold text-foreground">Scientist:</span> {policy.scientistMode}</div>
+        <div><span className="font-semibold text-foreground">Lab Manager:</span> {policy.labManagerMode}</div>
+        <div><span className="font-semibold text-foreground">Judge:</span> {policy.judgeMode}</div>
+      </div>
+
+      <p className="mt-4 rounded-lg border border-border bg-muted/30 px-3 py-3 text-xs text-muted-foreground">
+        {policy.summary}
+      </p>
+    </div>
+  );
+}
+
+function MetricTile({ label, value }: { label: string; value: string }) {
+  return (
+    <div className="rounded-lg border border-border bg-muted/30 px-3 py-2">
+      <div className="text-[11px] text-muted-foreground">{label}</div>
+      <div className="mt-1 text-sm font-semibold">{value}</div>
+    </div>
+  );
+}
+
+function MeaningCard({
+  icon,
+  title,
+  body,
+}: {
+  icon: ReactNode;
+  title: string;
+  body: string;
+}) {
+  return (
+    <div className="rounded-lg border border-border bg-muted/20 p-4">
+      <div className="mb-2 flex items-center gap-2 text-sm font-semibold">
+        {icon}
+        {title}
+      </div>
+      <p className="text-sm text-muted-foreground">{body}</p>
+    </div>
+  );
+}

frontend/src/types/index.ts

@@ -229,6 +229,15 @@ export interface BackendScenarioFamily {
   difficulties: string[];
 }
 
+export interface BackendRuntimeStatus {
+  scientist_runtime: string;
+  scientist_model: string;
+  scientist_ready: boolean;
+  agent_step_available: boolean;
+  available_runtimes: string[];
+  note: string;
+}
+
 // --- Training metrics ---
 
 export interface TrainingMetrics {

pyproject.toml

@@ -17,6 +17,7 @@ dependencies = [
   "fastapi>=0.115,<1.0",
   "uvicorn[standard]>=0.34,<1.0",
   "websockets>=15.0,<17.0",
+  "httpx>=0.27,<1.0",
   "openenv-core[core]>=0.2.1,<0.3.0",
 ]
 

replicalab/agents/__init__.py

@@ -17,7 +17,9 @@ from .scientist_policy import (
     RetryMetadata,
     ScientistCallResult,
     ScientistOutputParseError,
+    build_anthropic_scientist_policy,
     build_baseline_scientist_action,
+    build_ollama_scientist_policy,
     build_remote_scientist_policy,
     build_scientist_system_prompt,
     call_scientist_with_retry,
@@ -33,7 +35,9 @@ __all__ = [
     "RetryMetadata",
     "ScientistCallResult",
     "ScientistOutputParseError",
+    "build_anthropic_scientist_policy",
     "SuggestionChange",
+    "build_ollama_scientist_policy",
     "build_baseline_scientist_action",
     "build_remote_scientist_policy",
     "build_judge_audit",

replicalab/agents/scientist_policy.py

@@ -19,6 +19,7 @@ import re
 from importlib import import_module
 from typing import Any, Callable, Literal, Mapping
 
+import httpx
 from pydantic import BaseModel, ConfigDict, ValidationError
 
 from replicalab.models import (
@@ -804,6 +805,127 @@ def build_remote_scientist_policy(
     return policy_fn
 
 
+def build_anthropic_scientist_policy(
+    *,
+    api_key: str,
+    model: str,
+    max_completion_tokens: int = 450,
+    temperature: float = 0.0,
+    max_retries: int = 2,
+    timeout_seconds: float = 60.0,
+    base_url: str = "https://api.anthropic.com/v1/messages",
+    client: httpx.Client | None = None,
+) -> Callable[[ScientistObservation], ScientistAction]:
+    """Create a sync Scientist policy callable backed by Anthropic Messages API."""
+
+    owned_client = client is None
+    transport = client or httpx.Client(timeout=timeout_seconds)
+
+    def generate_fn(messages: list[dict[str, str]]) -> str:
+        system_blocks = [msg["content"] for msg in messages if msg["role"] == "system"]
+        request_messages = [
+            {"role": msg["role"], "content": msg["content"]}
+            for msg in messages
+            if msg["role"] in {"user", "assistant"}
+        ]
+        response = transport.post(
+            base_url,
+            headers={
+                "x-api-key": api_key,
+                "anthropic-version": "2023-06-01",
+                "content-type": "application/json",
+            },
+            json={
+                "model": model,
+                "max_tokens": max_completion_tokens,
+                "temperature": temperature,
+                "system": "\n\n".join(system_blocks),
+                "messages": request_messages,
+            },
+        )
+        response.raise_for_status()
+        payload = response.json()
+        return _extract_anthropic_message_text(payload.get("content", []))
+
+    def policy_fn(
+        observation: ScientistObservation,
+        *,
+        seed: int | None = None,
+        scenario: str | None = None,
+        difficulty: str | None = None,
+    ) -> ScientistAction:
+        result = call_scientist_with_retry(
+            generate_fn,
+            _build_live_scientist_system_prompt(
+                observation,
+                difficulty=difficulty,
+                scenario=scenario,
+            ),
+            observation,
+            max_retries=max_retries,
+        )
+        return result.action
+
+    setattr(policy_fn, "_replicalab_client", transport)
+    setattr(policy_fn, "_replicalab_owned_client", owned_client)
+    return policy_fn
+
+
+def build_ollama_scientist_policy(
+    *,
+    model: str,
+    max_retries: int = 2,
+    temperature: float = 0.0,
+    timeout_seconds: float = 60.0,
+    base_url: str = "http://127.0.0.1:11434/api/chat",
+    client: httpx.Client | None = None,
+) -> Callable[[ScientistObservation], ScientistAction]:
+    """Create a sync Scientist policy callable backed by a local Ollama model."""
+
+    owned_client = client is None
+    transport = client or httpx.Client(timeout=timeout_seconds)
+
+    def generate_fn(messages: list[dict[str, str]]) -> str:
+        response = transport.post(
+            base_url,
+            json={
+                "model": model,
+                "messages": messages,
+                "stream": False,
+                "format": "json",
+                "options": {
+                    "temperature": temperature,
+                },
+            },
+        )
+        response.raise_for_status()
+        payload = response.json()
+        return _extract_message_content(payload.get("message", {}).get("content", ""))
+
+    def policy_fn(
+        observation: ScientistObservation,
+        *,
+        seed: int | None = None,
+        scenario: str | None = None,
+        difficulty: str | None = None,
+    ) -> ScientistAction:
+        result = call_scientist_with_retry(
+            generate_fn,
+            _build_live_scientist_system_prompt(
+                observation,
+                difficulty=difficulty,
+                scenario=scenario,
+            ),
+            observation,
+            max_retries=max_retries,
+        )
+        return result.action
+
+    setattr(policy_fn, "_replicalab_client", transport)
+    setattr(policy_fn, "_replicalab_owned_client", owned_client)
+    return policy_fn
+
+
 def _build_live_scientist_system_prompt(
     observation: ScientistObservation,
     *,
@@ -877,3 +999,16 @@ def _extract_message_content(content: Any) -> str:
                 parts.append(str(text))
         return "\n".join(parts)
     return ""
+
+
+def _extract_anthropic_message_text(content: list[dict[str, Any]]) -> str:
+    parts: list[str] = []
+    for block in content:
+        if not isinstance(block, dict):
+            continue
+        if block.get("type") != "text":
+            continue
+        text = block.get("text")
+        if text:
+            parts.append(str(text))
+    return "\n".join(parts)

replicalab/config.py

@@ -10,6 +10,26 @@ from __future__ import annotations
 
 import os
 
+
+def _get_env_float(name: str, default: float) -> float:
+    raw = os.environ.get(name)
+    if raw is None:
+        return default
+    try:
+        return float(raw)
+    except ValueError:
+        return default
+
+
+def _get_env_int(name: str, default: int) -> int:
+    raw = os.environ.get(name)
+    if raw is None:
+        return default
+    try:
+        return int(raw)
+    except ValueError:
+        return default
+
 DEFAULT_SCENARIO_TEMPLATE = "math_reasoning"
 DEFAULT_DIFFICULTY = "easy"
 
@@ -41,6 +61,44 @@ ORACLE_SCENARIO_CACHE_DIR = os.environ.get(
     ".scenario_cache",
 )
 
+
+def get_scientist_runtime() -> str:
+    configured = os.environ.get("REPLICALAB_SCIENTIST_RUNTIME")
+    if configured:
+        return configured.strip().lower()
+    return "anthropic" if os.environ.get("ANTHROPIC_API_KEY") else "baseline"
+
+
+def get_scientist_model() -> str:
+    return os.environ.get("REPLICALAB_SCIENTIST_MODEL", "claude-3-5-haiku-latest")
+
+
+def get_scientist_ollama_model() -> str:
+    return os.environ.get("REPLICALAB_SCIENTIST_OLLAMA_MODEL", "glm-5:cloud")
+
+
+def get_scientist_ollama_base_url() -> str:
+    return os.environ.get(
+        "REPLICALAB_SCIENTIST_OLLAMA_BASE_URL",
+        "http://127.0.0.1:11434/api/chat",
+    )
+
+
+def get_scientist_max_retries() -> int:
+    return _get_env_int("REPLICALAB_SCIENTIST_MAX_RETRIES", 2)
+
+
+def get_scientist_max_completion_tokens() -> int:
+    return _get_env_int("REPLICALAB_SCIENTIST_MAX_COMPLETION_TOKENS", 450)
+
+
+def get_scientist_temperature() -> float:
+    return _get_env_float("REPLICALAB_SCIENTIST_TEMPERATURE", 0.0)
+
+
+def get_scientist_timeout_seconds() -> float:
+    return _get_env_float("REPLICALAB_SCIENTIST_TIMEOUT_SECONDS", 60.0)
+
 # Deterministic reward shaping constants.
 STEP_PROTOCOL_DELTA_SCALE = 0.25
 STEP_PROTOCOL_DELTA_CAP = 0.3

server/app.py

@@ -22,6 +22,7 @@ from __future__ import annotations
 import asyncio
 import json
 import logging
+import os
 import time
 import uuid
 from contextlib import asynccontextmanager
@@ -35,7 +36,28 @@ from fastapi.responses import FileResponse, HTMLResponse
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
 
+
+def _load_local_env() -> None:
+    env_path = Path(__file__).resolve().parent.parent / ".env"
+    if not env_path.is_file():
+        return
+    for raw_line in env_path.read_text(encoding="utf-8", errors="ignore").splitlines():
+        line = raw_line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, value = line.split("=", 1)
+        key = key.strip()
+        if not key:
+            continue
+        os.environ.setdefault(key, value.strip().strip('"').strip("'"))
+
+
+_load_local_env()
+
 from replicalab.agents import (
+    build_anthropic_scientist_policy,
+    build_baseline_scientist_action,
+    build_ollama_scientist_policy,
     check_feasibility,
     compose_lab_manager_response,
     suggest_alternative,
@@ -50,6 +72,14 @@ from replicalab.config import (
     SESSION_TTL_SECONDS,
     STUB_ACCEPT_REWARD,
     WS_IDLE_TIMEOUT_SECONDS,
+    get_scientist_max_completion_tokens,
+    get_scientist_max_retries,
+    get_scientist_model,
+    get_scientist_ollama_base_url,
+    get_scientist_ollama_model,
+    get_scientist_runtime,
+    get_scientist_temperature,
+    get_scientist_timeout_seconds,
 )
 from replicalab.utils.logging import log_episode_reward, write_episode_log
 from replicalab.scenarios import (
@@ -347,6 +377,225 @@ _sessions: dict[str, dict[str, Any]] = {}
 _replay_store: dict[str, EpisodeLog] = {}
 # { episode_id: EpisodeLog }
 
+_SCIENTIST_POLICY_CACHE: dict[tuple[Any, ...], Any] = {}
+
+
+def _scientist_runtime_status() -> dict[str, Any]:
+    runtime = get_scientist_runtime()
+    if runtime == "anthropic":
+        model = get_scientist_model()
+    elif runtime == "ollama":
+        model = get_scientist_ollama_model()
+    else:
+        model = "baseline-heuristic"
+    anthropic_ready = bool(os.environ.get("ANTHROPIC_API_KEY"))
+    ready = (
+        runtime == "baseline"
+        or (runtime == "anthropic" and anthropic_ready)
+        or runtime == "ollama"
+    )
+    if runtime == "anthropic" and ready:
+        note = "Episodes can use backend model-driven Scientist inference through Anthropic."
+    elif runtime == "ollama":
+        note = "Episodes can use backend model-driven Scientist inference through the local Ollama runtime."
+    else:
+        note = "Episodes use the deterministic baseline Scientist policy."
+    return {
+        "scientist_runtime": runtime,
+        "scientist_model": model,
+        "scientist_ready": ready,
+        "agent_step_available": ready,
+        "available_runtimes": ["baseline", "anthropic", "ollama"],
+        "note": note,
+    }
+
+
+def _get_scientist_policy():
+    runtime = get_scientist_runtime()
+    if runtime == "baseline":
+        return build_baseline_scientist_action
+    if runtime == "anthropic":
+        api_key = os.environ.get("ANTHROPIC_API_KEY", "").strip()
+        if not api_key:
+            raise RuntimeError("ANTHROPIC_API_KEY is not configured for Anthropic Scientist mode.")
+        cache_key = (
+            runtime,
+            get_scientist_model(),
+            get_scientist_max_completion_tokens(),
+            get_scientist_temperature(),
+            get_scientist_max_retries(),
+            get_scientist_timeout_seconds(),
+        )
+    elif runtime == "ollama":
+        cache_key = (
+            runtime,
+            get_scientist_ollama_model(),
+            get_scientist_ollama_base_url(),
+            get_scientist_temperature(),
+            get_scientist_max_retries(),
+            get_scientist_timeout_seconds(),
+        )
+    else:
+        raise RuntimeError(f"Unsupported scientist runtime '{runtime}'.")
+    cached = _SCIENTIST_POLICY_CACHE.get(cache_key)
+    if cached is not None:
+        return cached
+
+    if runtime == "anthropic":
+        policy = build_anthropic_scientist_policy(
+            api_key=api_key,
+            model=get_scientist_model(),
+            max_completion_tokens=get_scientist_max_completion_tokens(),
+            temperature=get_scientist_temperature(),
+            max_retries=get_scientist_max_retries(),
+            timeout_seconds=get_scientist_timeout_seconds(),
+        )
+    else:
+        policy = build_ollama_scientist_policy(
+            model=get_scientist_ollama_model(),
+            base_url=get_scientist_ollama_base_url(),
+            temperature=get_scientist_temperature(),
+            max_retries=0,
+            timeout_seconds=get_scientist_timeout_seconds(),
+        )
+    _SCIENTIST_POLICY_CACHE.clear()
+    _SCIENTIST_POLICY_CACHE[cache_key] = policy
+    return policy
+
+
+def _normalize_runtime_scientist_action(
+    session: dict[str, Any],
+    action: ScientistAction,
+) -> tuple[ScientistAction, list[str]]:
+    observation = session.get("last_observation")
+    lab_obs = observation.lab_manager if observation is not None else None
+    action_type = (
+        action.action_type.value
+        if hasattr(action.action_type, "value")
+        else str(action.action_type)
+    )
+    if action_type not in {"propose_protocol", "revise_protocol"}:
+        return action, []
+
+    updates: dict[str, Any] = {}
+    notes: list[str] = []
+    max_controls = max(0, action.sample_size - 1)
+    if len(action.controls) > max_controls:
+        updates["controls"] = list(action.controls[:max_controls])
+        notes.append("trimmed_controls_to_fit_sample_size")
+
+    if lab_obs is not None:
+        if action.duration_days > lab_obs.time_limit_days:
+            updates["duration_days"] = lab_obs.time_limit_days
+            notes.append("clamped_duration_to_time_limit")
+
+        if lab_obs.equipment_available:
+            available_equipment = set(lab_obs.equipment_available)
+            filtered_equipment = [
+                item for item in action.required_equipment if item in available_equipment
+            ]
+            if not filtered_equipment:
+                filtered_equipment = list(lab_obs.equipment_available[:1])
+            if filtered_equipment != list(action.required_equipment):
+                updates["required_equipment"] = filtered_equipment
+                notes.append("aligned_equipment_to_available_inventory")
+
+        if lab_obs.reagents_in_stock:
+            available_reagents = set(lab_obs.reagents_in_stock)
+            filtered_reagents = [
+                item for item in action.required_reagents if item in available_reagents
+            ]
+            if not filtered_reagents:
+                filtered_reagents = list(lab_obs.reagents_in_stock[:1])
+            if filtered_reagents != list(action.required_reagents):
+                updates["required_reagents"] = filtered_reagents
+                notes.append("aligned_reagents_to_available_inventory")
+
+    if not updates:
+        return action, []
+    return action.model_copy(update=updates), notes
+
+
+def _resolve_scientist_action(session: dict[str, Any]) -> tuple[ScientistAction, dict[str, Any]]:
+    observation = session.get("last_observation")
+    if observation is None or observation.scientist is None:
+        raise RuntimeError("Session has no active Scientist observation. Reset the episode first.")
+
+    runtime = get_scientist_runtime()
+    if runtime == "baseline":
+        action = build_baseline_scientist_action(observation.scientist)
+    else:
+        policy = _get_scientist_policy()
+        action = policy(
+            observation.scientist,
+            seed=session.get("seed"),
+            scenario=session.get("scenario"),
+            difficulty=session.get("difficulty"),
+        )
+    raw_action = action.model_dump(mode="json")
+    action, normalization_notes = _normalize_runtime_scientist_action(session, action)
+
+    metadata = {
+        "scientist_runtime": runtime,
+        "scientist_model": (
+            get_scientist_model()
+            if runtime == "anthropic"
+            else get_scientist_ollama_model()
+            if runtime == "ollama"
+            else "baseline-heuristic"
+        ),
+        "scientist_action": action.model_dump(mode="json"),
+        "scientist_action_raw": raw_action,
+        "scientist_safety_adjustments": normalization_notes,
+    }
+    return action, metadata
+
+
+def _record_session_step(session_id: str, result: StepResult) -> StepResult:
+    session = _sessions[session_id]
+    session["total_steps"] = session.get("total_steps", 0) + 1
+    if result.observation is not None:
+        session["last_observation"] = result.observation
+    if result.info.error is not None:
+        session["invalid_action_count"] = session.get("invalid_action_count", 0) + 1
+
+    if result.done:
+        state = session["env"].state()
+        episode_log = _build_episode_log(
+            session["episode_id"],
+            state,
+            result,
+            invalid_action_count=session.get("invalid_action_count", 0),
+            total_steps=session.get("total_steps", 0),
+        )
+        _replay_store[session["episode_id"]] = episode_log
+
+        try:
+            write_episode_log(episode_log)
+            log_episode_reward(
+                episode_id=session["episode_id"],
+                seed=state.seed,
+                scenario_template=state.scenario_template,
+                difficulty=state.difficulty,
+                total_reward=state.reward,
+                breakdown=result.info.reward_breakdown,
+                rounds_used=state.round_number,
+                agreement_reached=result.info.agreement_reached,
+                verdict=result.info.verdict or "",
+                judge_notes=result.info.judge_notes or "",
+            )
+        except Exception:
+            log.exception("Failed to persist episode log to disk")
+
+        log.info(
+            "Episode done | session=%s episode=%s reward=%.2f",
+            session_id,
+            session["episode_id"],
+            result.reward,
+        )
+
+    return result
+
 
 def _touch(session_id: str) -> None:
     if session_id in _sessions:
@@ -444,6 +693,19 @@ class StepRequest(BaseModel):
     action: ScientistAction
 
 
+class AgentStepRequest(BaseModel):
+    session_id: str
+
+
+class RuntimeStatusResponse(BaseModel):
+    scientist_runtime: str
+    scientist_model: str
+    scientist_ready: bool
+    agent_step_available: bool
+    available_runtimes: list[str]
+    note: str
+
+
 # ---------------------------------------------------------------------------
 # REST endpoints
 # ---------------------------------------------------------------------------
@@ -776,6 +1038,11 @@ async def health():
     }
 
 
+@_api.get("/runtime", response_model=RuntimeStatusResponse)
+async def runtime_status():
+    return RuntimeStatusResponse.model_validate(_scientist_runtime_status())
+
+
 @_api.get("/scenarios", response_model=ScenariosResponse)
 async def list_scenarios():
     return ScenariosResponse(scenarios=SCENARIOS)
@@ -802,6 +1069,10 @@ async def reset_episode(req: ResetRequest):
         "episode_id": episode_id,
         "total_steps": 0,
         "invalid_action_count": 0,
+        "last_observation": obs,
+        "seed": req.seed,
+        "scenario": req.scenario,
+        "difficulty": req.difficulty,
     }
 
     log.info("REST reset | session=%s episode=%s", session_id, episode_id)
@@ -818,48 +1089,46 @@ async def step_episode(req: StepRequest):
     env = session["env"]
 
     result = env.step(req.action)
+    return _record_session_step(req.session_id, result)
 
-    session["total_steps"] = session.get("total_steps", 0) + 1
-    if result.info.error is not None:
-        session["invalid_action_count"] = session.get("invalid_action_count", 0) + 1
 
-    # Store completed episode log for replay and persist to disk (ENV 09)
-    if result.done:
-        state = env.state()
-        episode_log = _build_episode_log(
-            session["episode_id"],
-            state,
-            result,
-            invalid_action_count=session.get("invalid_action_count", 0),
-            total_steps=session.get("total_steps", 0),
-        )
-        _replay_store[session["episode_id"]] = episode_log
+@_api.post("/agent-step", response_model=StepResult)
+async def agent_step_episode(req: AgentStepRequest):
+    if req.session_id not in _sessions:
+        raise HTTPException(status_code=404, detail="Session not found. Call /reset first.")
 
-        try:
-            write_episode_log(episode_log)
-            log_episode_reward(
-                episode_id=session["episode_id"],
-                seed=state.seed,
-                scenario_template=state.scenario_template,
-                difficulty=state.difficulty,
-                total_reward=state.reward,
-                breakdown=result.info.reward_breakdown,
-                rounds_used=state.round_number,
-                agreement_reached=result.info.agreement_reached,
-                verdict=result.info.verdict or "",
-                judge_notes=result.info.judge_notes or "",
-            )
-        except Exception:
-            log.exception("Failed to persist episode log to disk")
+    _touch(req.session_id)
+    session = _sessions[req.session_id]
+    env = session["env"]
 
-        log.info(
-            "Episode done | session=%s episode=%s reward=%.2f",
+    try:
+        action, metadata = _resolve_scientist_action(session)
+    except Exception as exc:
+        runtime = get_scientist_runtime()
+        observation = session.get("last_observation")
+        if observation is None or observation.scientist is None or runtime == "baseline":
+            log.exception("Scientist runtime failed for session %s", req.session_id)
+            raise HTTPException(status_code=503, detail=f"Scientist runtime failed: {exc}") from exc
+        log.exception(
+            "Scientist runtime failed for session %s; falling back to baseline",
             req.session_id,
-            session["episode_id"],
-            result.reward,
         )
-
-    return result
+        action = build_baseline_scientist_action(observation.scientist)
+        metadata = {
+            "scientist_runtime": f"{runtime}_fallback",
+            "scientist_model": "baseline-heuristic",
+            "scientist_action": action.model_dump(mode="json"),
+            "scientist_action_raw": None,
+            "scientist_safety_adjustments": ["fallback_to_baseline_after_runtime_error"],
+            "scientist_error": str(exc),
+        }
+
+    result = env.step(action)
+    result.info = StepInfo.model_validate({
+        **result.info.model_dump(mode="json"),
+        **metadata,
+    })
+    return _record_session_step(req.session_id, result)
 
 
 @_api.get("/replay/{episode_id}", response_model=EpisodeLog)

tests/test_scientist_policy.py

@@ -1,12 +1,17 @@
 from __future__ import annotations
 
+import json
+
+import httpx
 import pytest
 
 from replicalab.agents.scientist_policy import (
     RetryMetadata,
     ScientistCallResult,
     ScientistOutputParseError,
+    build_anthropic_scientist_policy,
     build_baseline_scientist_action,
+    build_ollama_scientist_policy,
     build_scientist_system_prompt,
     call_scientist_with_retry,
     format_scientist_observation,
@@ -900,3 +905,83 @@ def test_baseline_scientist_accepts_at_final_round_even_with_blocker() -> None:
     action = build_baseline_scientist_action(obs)
 
     assert action.action_type is ScientistActionType.ACCEPT
+
+
+def test_build_anthropic_scientist_policy_calls_messages_api() -> None:
+    captured: dict[str, object] = {}
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        captured["headers"] = dict(request.headers)
+        captured["payload"] = json.loads(request.content.decode("utf-8"))
+        return httpx.Response(
+            200,
+            json={
+                "content": [
+                    {
+                        "type": "text",
+                        "text": _VALID_REQUEST_INFO_JSON,
+                    }
+                ]
+            },
+        )
+
+    client = httpx.Client(transport=httpx.MockTransport(handler))
+    policy = build_anthropic_scientist_policy(
+        api_key="test-key",
+        model="claude-test",
+        client=client,
+        base_url="https://api.anthropic.com/v1/messages",
+    )
+
+    action = policy(
+        _base_observation(),
+        scenario="ml_benchmark",
+        difficulty="medium",
+    )
+
+    assert action.action_type is ScientistActionType.REQUEST_INFO
+    payload = captured["payload"]
+    assert isinstance(payload, dict)
+    assert payload["model"] == "claude-test"
+    assert payload["temperature"] == 0.0
+    assert payload["messages"][0]["role"] == "user"
+    headers = captured["headers"]
+    assert isinstance(headers, dict)
+    assert headers["x-api-key"] == "test-key"
+
+
+def test_build_ollama_scientist_policy_calls_local_chat_api() -> None:
+    captured: dict[str, object] = {}
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        captured["payload"] = json.loads(request.content.decode("utf-8"))
+        return httpx.Response(
+            200,
+            json={
+                "message": {
+                    "role": "assistant",
+                    "content": _VALID_REQUEST_INFO_JSON,
+                }
+            },
+        )
+
+    client = httpx.Client(transport=httpx.MockTransport(handler))
+    policy = build_ollama_scientist_policy(
+        model="glm-5:cloud",
+        client=client,
+        base_url="http://127.0.0.1:11434/api/chat",
+    )
+
+    action = policy(
+        _base_observation(),
+        scenario="finance_trading",
+        difficulty="medium",
+    )
+
+    assert action.action_type is ScientistActionType.REQUEST_INFO
+    payload = captured["payload"]
+    assert isinstance(payload, dict)
+    assert payload["model"] == "glm-5:cloud"
+    assert payload["stream"] is False
+    assert payload["format"] == "json"
+    assert payload["messages"][0]["role"] == "system"

tests/test_server.py

@@ -18,6 +18,7 @@ import pytest
 from fastapi.testclient import TestClient
 from starlette.websockets import WebSocketDisconnect
 
+from replicalab.models import ScientistAction
 from server.app import app
 
 _EXPECTED_FAMILIES = {"math_reasoning", "ml_benchmark", "finance_trading"}
@@ -54,6 +55,38 @@ class TestHealthEndpoint:
         assert r1 == r2
 
 
+class TestRuntimeEndpoint:
+    """Local runtime metadata for model-backed Scientist stepping."""
+
+    def test_runtime_defaults_to_baseline_without_api_key(
+        self, client: TestClient, monkeypatch
+    ) -> None:
+        monkeypatch.delenv("REPLICALAB_SCIENTIST_RUNTIME", raising=False)
+        monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
+
+        resp = client.get("/runtime")
+
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["scientist_runtime"] == "baseline"
+        assert data["scientist_model"] == "baseline-heuristic"
+        assert data["agent_step_available"] is True
+
+    def test_runtime_reports_anthropic_when_enabled(
+        self, client: TestClient, monkeypatch
+    ) -> None:
+        monkeypatch.setenv("REPLICALAB_SCIENTIST_RUNTIME", "anthropic")
+        monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key")
+
+        resp = client.get("/runtime")
+
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["scientist_runtime"] == "anthropic"
+        assert data["scientist_ready"] is True
+        assert data["agent_step_available"] is True
+
+
 class TestLogConfig:
     """OBS 02 — log level configurability."""
 
@@ -491,6 +524,65 @@ class TestStepEndpoint:
         )
 
 
+class TestAgentStepEndpoint:
+    """POST /agent-step uses the configured Scientist runtime."""
+
+    def test_agent_step_runs_runtime_policy(
+        self, client: TestClient, monkeypatch
+    ) -> None:
+        import server.app as server_app
+
+        monkeypatch.setenv("REPLICALAB_SCIENTIST_RUNTIME", "anthropic")
+        monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key")
+        server_app._SCIENTIST_POLICY_CACHE.clear()
+
+        action = ScientistAction.model_validate(_good_action_payload(client))
+
+        def fake_policy(observation, **kwargs):
+            assert observation.paper_title
+            assert kwargs["scenario"] == "ml_benchmark"
+            assert kwargs["difficulty"] == "medium"
+            return action
+
+        monkeypatch.setattr(server_app, "_get_scientist_policy", lambda: fake_policy)
+
+        reset_data = _reset(client, scenario="ml_benchmark", difficulty="medium")
+        resp = client.post("/agent-step", json={"session_id": reset_data["session_id"]})
+
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["info"]["scientist_runtime"] == "anthropic"
+        assert data["info"]["scientist_model"]
+        assert data["info"]["scientist_action"]["action_type"] == "propose_protocol"
+        assert data["observation"]["scientist"]["round_number"] == 1
+
+    def test_agent_step_invalid_session_returns_404(self, client: TestClient) -> None:
+        resp = client.post("/agent-step", json={"session_id": "missing"})
+
+        assert resp.status_code == 404
+        assert "Session not found" in resp.json()["detail"]
+
+    def test_agent_step_falls_back_to_baseline_on_runtime_failure(
+        self, client: TestClient, monkeypatch
+    ) -> None:
+        import server.app as server_app
+
+        monkeypatch.setenv("REPLICALAB_SCIENTIST_RUNTIME", "ollama")
+        monkeypatch.setattr(
+            server_app,
+            "_resolve_scientist_action",
+            lambda session: (_ for _ in ()).throw(RuntimeError("model timeout")),
+        )
+
+        reset_data = _reset(client, scenario="math_reasoning", difficulty="easy")
+        resp = client.post("/agent-step", json={"session_id": reset_data["session_id"]})
+
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["info"]["scientist_runtime"] == "ollama_fallback"
+        assert data["info"]["scientist_error"] == "model timeout"
+
+
 # ---------------------------------------------------------------------------
 # WebSocket handler — API 06
 # ---------------------------------------------------------------------------