ThomsenDrake commited on
Commit
674c20a
·
verified ·
1 Parent(s): 94cbe85

Refresh submission evidence gates

Browse files
docs/model_parameter_evidence_ledger.md CHANGED
@@ -1,15 +1,16 @@
1
  # Figment Model Parameter And Evidence Ledger
2
 
3
- Date: 2026-06-07
4
 
5
  Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmation claims in one evidence-gated place. README, submission copy, demo video, and social posts should not upgrade a claim beyond this ledger.
6
 
7
  ## Current Claim Boundary
8
 
9
  - Hosted Omni has measured eval evidence through the eval harness. The public Space is now verified runnable in no-secret canned-fallback mode, but that public Space proof is not live hosted Omni generation evidence.
10
- - The public Space target exists and cold-boots from the Space URL. Current public API evidence: `runtime.stage=RUNNING`, `sdk=gradio`, `sha=5dcfc5c830de7331eca9020b17e1c571a8619654`, 92 siblings, and `app.py` present. Public workflow evidence: typed intake, deterministic pediatric-dehydration escalation, protocol retrieval, `raw_route=canned`, `final_route=canned_backend`, `validation_status=passed`, `raw_audio_stored=false`, and zero model-retained fields.
11
- - The local 4B + Parakeet route is the preferred no-cloud/off-grid proof path. The full BF16 4B artifact is now downloaded locally, but the route is not yet proven with a real local 50-case eval or local ASR smoke.
12
- - No published Figment adapter is recorded yet. Well-Tuned remains a stretch claim until a published fine-tuned model or adapter is used by the app and measured.
 
13
  - Organizer confirmation is still needed for the Omni 31B body-count versus 33B sidebar ambiguity and for any additive local stack or adapter-count interpretation.
14
 
15
  ## Parameter Ledger
@@ -18,20 +19,20 @@ Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmati
18
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
19
  | Hosted Omni primary | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`; API route `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | NVIDIA model-card body: 31B total parameters; HF sidebar has been observed as 33B in planning docs | Roughly 3B active parameters per token is a runtime/MoE note, not the compliance number | None used in current evals | Native Omni speech encoder is part of the Omni model-card count; no separate ASR model is claimed for hosted Omni | Hosted NVIDIA API route in current evals; self-hosted no-cloud route not recorded | Pending: ask organizers whether model-card body count is acceptable if sidebar count differs | Baseline eval: 28/50 whole-output competence, 22/50 full fallback, 50/50 final validation. Follow-up eval: 31/50 competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation |
20
  | Self-hosted Omni no-cloud target | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`, FP8, or NVFP4 variant if served locally | Same Omni 31B body-count claim, with same 33B sidebar ambiguity | Active parameters do not decide compliance | None recorded | Native Omni audio if used locally; included in Omni count if organizers accept the model-card count | Would be local/self-hosted only if served with no runtime cloud APIs | Pending for count ambiguity and hardware/runtime proof | No recorded no-cloud eval or public demo trace yet. Do not claim Off the Grid achieved |
21
- | Local 4B + Parakeet proof path | Text: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`; ASR: `nvidia/parakeet-rnnt-1.1b`; local route `MODEL_BACKEND=llama_cpp`, `MODEL_STACK=local_4b_parakeet` | Workback model-card notes: 3.97B text model plus about 1.1B Parakeet, roughly 5.1B nominal before adapters | No active-parameter substitution; use additive total-count story if organizers require stack accounting | None recorded yet; exact adapter count must be measured before any Well-Tuned or compliance upgrade | About 1.1B for Parakeet RNNT ASR, only if real local ASR is enabled and proven | Intended local OpenAI-compatible endpoint serving the full BF16 model plus local ASR; no-cloud only after recorded proof | Pending: confirm additive multi-model counting and adapter counting | Full BF16 4B snapshot downloaded at repo revision `dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f`; `model.safetensors` is `7947142640` bytes with SHA-256 `55d4e2519456c4a9bddf596b0748d630e3b2ce6ff6f4c2b7ed3e07e2b00dad42`. Parakeet artifact downloaded at repo revision `a07b19e98a26c1873a3f2622c446a4a1ca6316cb`; `parakeet-rnnt-1.1b.nemo` is `4283105280` bytes with SHA-256 `535896f014953d945b287ac533560e20da8103c6781b152de4645528e2b60738`. No real local 50-case eval, no local ASR provider proof, and no trace hash recorded yet |
22
- | 4B Figment adapter stretch | Planned adapter name: `nvidia-nemotron-3-nano-4b-figment-lora-v1` | Base model count is 3.97B; adapter count must be added or documented per organizer guidance | Not applicable | Pending. Record exact trainable and published adapter parameter count before claiming | Parakeet count applies only if adapter demo also uses local ASR | Local route or published HF model route, depending on final artifact | Pending for adapter accounting and Well-Tuned eligibility | Not trained, published, or measured in this ledger |
23
  | Canned fallback | No live model | Not a model-compliance artifact | Not applicable | Not applicable | Not applicable | Local deterministic fallback; public Space no-secret fallback verified | Not applicable | Useful for safety, deployment health, and cold-start fallback only. Cannot count as model competence, Off the Grid proof, Llama Champion proof, or Well-Tuned proof |
24
 
25
  ## Submission Gates
26
 
27
  | Claim | Required upgrade evidence |
28
  | --- | --- |
29
- | Public Space runnable | Satisfied for no-secret canned-fallback mode at Space commit `5dcfc5c830de7331eca9020b17e1c571a8619654`: app files present, clean cold boot from the Space URL, typed intake run, and trace showing actual route/fallback status |
30
  | Hosted model load-bearing | Cite hosted eval metrics separately from final validation: 31/50 whole-output competence and 480/650 model-retained fields in the follow-up run |
31
  | <=32B hosted Omni compliance | Organizer accepts the 31B model-card body count or the submission falls back to a clearly eligible smaller route |
32
  | Off the Grid | Recorded no-cloud run with trace evidence, either self-hosted Omni or local 4B + Parakeet/typed intake |
33
  | Llama Champion | Eligible model route runs through llama.cpp with trace or eval evidence |
34
- | Well-Tuned | Published fine-tuned model or adapter is used by the app, measured, and still passes safety validation |
35
  | Backyard AI user use | Completed user-test notes from a real trained responder on synthetic or de-identified scenarios |
36
  | Demo video and social post | Final links exist and wording says achieved only for artifacts supported by this ledger |
37
 
 
1
  # Figment Model Parameter And Evidence Ledger
2
 
3
+ Date: 2026-06-15
4
 
5
  Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmation claims in one evidence-gated place. README, submission copy, demo video, and social posts should not upgrade a claim beyond this ledger.
6
 
7
  ## Current Claim Boundary
8
 
9
  - Hosted Omni has measured eval evidence through the eval harness. The public Space is now verified runnable in no-secret canned-fallback mode, but that public Space proof is not live hosted Omni generation evidence.
10
+ - The public Space target exists and cold-boots from the Space URL. Current public API evidence: `runtime.stage=RUNNING`, `hardware=cpu-basic`, `sha=94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`, 167 siblings, `app.py` present, `figment/observation_targets.py` present, and Space URL HTTP 200. Public workflow evidence remains no-secret canned fallback unless a later non-canned public trace is recorded.
11
+ - The public model archive now records published measured Figment 4B LoRA merged artifacts through `figment_sft_v14p`. The strongest measured result is v14p repair-union on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback. This is tuned model-system evidence, not no-cloud or clinical validation evidence.
12
+ - The local 4B + Parakeet route is still the preferred no-cloud/off-grid proof path. The full BF16 4B artifact and Parakeet artifact are present locally, and the tuned v14p GGUF is published, but no-cloud route proof and local ASR provider proof are still separate gates.
13
+ - Well-Tuned wording may cite the published measured v14p tuned artifacts if the submission/judging interpretation accepts the merged-model archive and local route support. Do not imply the public no-secret Space is serving the tuned model.
14
  - Organizer confirmation is still needed for the Omni 31B body-count versus 33B sidebar ambiguity and for any additive local stack or adapter-count interpretation.
15
 
16
  ## Parameter Ledger
 
19
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
20
  | Hosted Omni primary | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`; API route `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | NVIDIA model-card body: 31B total parameters; HF sidebar has been observed as 33B in planning docs | Roughly 3B active parameters per token is a runtime/MoE note, not the compliance number | None used in current evals | Native Omni speech encoder is part of the Omni model-card count; no separate ASR model is claimed for hosted Omni | Hosted NVIDIA API route in current evals; self-hosted no-cloud route not recorded | Pending: ask organizers whether model-card body count is acceptable if sidebar count differs | Baseline eval: 28/50 whole-output competence, 22/50 full fallback, 50/50 final validation. Follow-up eval: 31/50 competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation |
21
  | Self-hosted Omni no-cloud target | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`, FP8, or NVFP4 variant if served locally | Same Omni 31B body-count claim, with same 33B sidebar ambiguity | Active parameters do not decide compliance | None recorded | Native Omni audio if used locally; included in Omni count if organizers accept the model-card count | Would be local/self-hosted only if served with no runtime cloud APIs | Pending for count ambiguity and hardware/runtime proof | No recorded no-cloud eval or public demo trace yet. Do not claim Off the Grid achieved |
22
+ | Local 4B + Parakeet proof path | Text base: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`; tuned archive: `build-small-hackathon/figment-finetuned-model-archive`; ASR: `nvidia/parakeet-rnnt-1.1b`; local route `MODEL_BACKEND=llama_cpp`, `MODEL_STACK=local_4b_parakeet` | Workback model-card notes: 3.97B text model plus about 1.1B Parakeet, roughly 5.1B nominal before adapters | No active-parameter substitution; use additive total-count story if organizers require stack accounting | Published merged LoRA artifacts exist through `figment_sft_v14p`; exact adapter-only parameter count is not used as the primary public claim | About 1.1B for Parakeet RNNT ASR, only if real local ASR is enabled and proven | Intended local OpenAI-compatible endpoint serving the tuned GGUF plus local ASR; no-cloud only after recorded proof | Pending: confirm additive multi-model counting and adapter-count treatment | Full BF16 4B snapshot downloaded at repo revision `dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f`; `model.safetensors` is `7947142640` bytes with SHA-256 `55d4e2519456c4a9bddf596b0748d630e3b2ce6ff6f4c2b7ed3e07e2b00dad42`. Parakeet artifact downloaded at repo revision `a07b19e98a26c1873a3f2622c446a4a1ca6316cb`; `parakeet-rnnt-1.1b.nemo` is `4283105280` bytes with SHA-256 `535896f014953d945b287ac533560e20da8103c6781b152de4645528e2b60738`. The v14p repair-union measured result is 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback on the corrected 150-case field-workflow holdout. No no-cloud local route proof, local ASR provider proof, or target-user trace is recorded yet |
23
+ | 4B Figment tuned archive | `build-small-hackathon/figment-finetuned-model-archive`, especially `figment_sft_v14p/figment-sft-v14p-lora-merged-bf16/` and `figment_sft_v14p/figment-sft-v14p-lora-merged-bf16.bf16.gguf` | Base model is the 3.97B Nemotron 4B BF16 target; merged LoRA artifacts are derived from that base | Not applicable | Published as merged BF16 and GGUF artifacts with merge manifests; adapter-only count remains a documentation detail if needed by organizers | None for text-only eval | Public HF model repo plus local OpenAI-compatible app route support | Pending only for any organizer-specific adapter-count interpretation | Public model repo is `private=False`, current commit `7da772ec7c0de20011d42780ea8afa65af4aef70`, with v1 pilot plus v5-v14p artifacts. v14p model card records the repair-union result and GGUF SHA. |
24
  | Canned fallback | No live model | Not a model-compliance artifact | Not applicable | Not applicable | Not applicable | Local deterministic fallback; public Space no-secret fallback verified | Not applicable | Useful for safety, deployment health, and cold-start fallback only. Cannot count as model competence, Off the Grid proof, Llama Champion proof, or Well-Tuned proof |
25
 
26
  ## Submission Gates
27
 
28
  | Claim | Required upgrade evidence |
29
  | --- | --- |
30
+ | Public Space runnable | Satisfied for no-secret canned-fallback mode at Space commit `94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`: app files present, clean cold boot from the Space URL, and route/fallback labeling preserved. Non-canned public model traces remain separate proof. |
31
  | Hosted model load-bearing | Cite hosted eval metrics separately from final validation: 31/50 whole-output competence and 480/650 model-retained fields in the follow-up run |
32
  | <=32B hosted Omni compliance | Organizer accepts the 31B model-card body count or the submission falls back to a clearly eligible smaller route |
33
  | Off the Grid | Recorded no-cloud run with trace evidence, either self-hosted Omni or local 4B + Parakeet/typed intake |
34
  | Llama Champion | Eligible model route runs through llama.cpp with trace or eval evidence |
35
+ | Well-Tuned | Cite the published measured v14p tuned archive only with wording that does not imply the no-secret public Space is serving it |
36
  | Backyard AI user use | Completed user-test notes from a real trained responder on synthetic or de-identified scenarios |
37
  | Demo video and social post | Final links exist and wording says achieved only for artifacts supported by this ledger |
38
 
docs/safety_statement.md CHANGED
@@ -1,6 +1,6 @@
1
  # Figment Safety Statement
2
 
3
- Status: submission-ready draft. Update the evidence links before final submission; do not turn pending items into claims until the artifacts exist.
4
 
5
  ## Intended Use
6
 
@@ -45,18 +45,21 @@ Achieved in-repo artifacts:
45
  - hosted NVIDIA Omni client path and labeled canned fallback path;
46
  - trace and validator surfaces for review;
47
  - synthetic demo-audio asset path;
 
 
 
 
48
  - this safety statement;
49
  - Apache-2.0 license file.
50
 
51
  Proof still needed before final submission claims:
52
 
53
- - public Hugging Face Space cold boot with app files present;
54
  - recorded live model run whose trace is not canned fallback;
55
  - recorded no-cloud run before claiming Off the Grid as achieved;
56
  - real trained-responder user-test notes using synthetic or de-identified scenarios;
 
57
  - demo video link;
58
- - social post link;
59
- - measured eval results before claiming model quality thresholds.
60
 
61
  ## Non-Goals
62
 
 
1
  # Figment Safety Statement
2
 
3
+ Status: submission-ready draft. Evidence links refreshed 2026-06-15; do not turn pending items into claims until the artifacts exist.
4
 
5
  ## Intended Use
6
 
 
45
  - hosted NVIDIA Omni client path and labeled canned fallback path;
46
  - trace and validator surfaces for review;
47
  - synthetic demo-audio asset path;
48
+ - public Hugging Face Space cold boot with app files present at commit `94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`;
49
+ - published 4B LoRA model archive and eval/training dataset repo;
50
+ - measured v14p repair-union result on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback;
51
+ - hosted Omni follow-up eval: 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation;
52
  - this safety statement;
53
  - Apache-2.0 license file.
54
 
55
  Proof still needed before final submission claims:
56
 
 
57
  - recorded live model run whose trace is not canned fallback;
58
  - recorded no-cloud run before claiming Off the Grid as achieved;
59
  - real trained-responder user-test notes using synthetic or de-identified scenarios;
60
+ - local Parakeet ASR provider proof before claiming local audio;
61
  - demo video link;
62
+ - social post link.
 
63
 
64
  ## Non-Goals
65
 
docs/submission_checklist.md CHANGED
@@ -1,6 +1,6 @@
1
  # Figment Submission Checklist
2
 
3
- Status: living checklist for evidence. Keep claims in README, submission copy, demo video, and social posts aligned with this file.
4
 
5
  Primary tracker: [adversarial review action items](adversarial-review-action-items.md).
6
 
@@ -9,7 +9,10 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
9
  | Artifact | Status | Link / evidence |
10
  | -------- | ------ | --------------- |
11
  | Public Hugging Face Space | Runnable in no-secret canned-fallback mode | https://huggingface.co/spaces/build-small-hackathon/figment |
12
- | Space cold boot with app files present | Verified 2026-06-07 | Space API `runtime.stage=RUNNING`, `sdk=gradio`, `sha=5dcfc5c830de7331eca9020b17e1c571a8619654`, 92 siblings, `app.py` present; Space URL served HTTP 200 |
 
 
 
13
  | Demo video | Proof needed | Pending |
14
  | Social post | Proof needed | Pending |
15
  | Safety statement | Present | [safety_statement.md](safety_statement.md) |
@@ -18,10 +21,11 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
18
  | Live hosted Omni trace | Eval traces present; final demo trace still needed | Baseline: `traces/hosted_omni_eval_20260607T194833Z.jsonl`; follow-up: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl` |
19
  | No-cloud/off-grid trace | Proof needed before claiming Off the Grid achieved | Pending |
20
  | Hosted Omni eval results | Measured | [hosted_omni_eval_results.md](hosted_omni_eval_results.md): 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation in the follow-up run |
21
- | Local 4B + Parakeet eval results | Artifacts and evidence helpers ready; proof needed | Full BF16 4B snapshot and Parakeet `.nemo` artifact are present locally; `scripts/run_local_4b_evidence.py` captures endpoint metadata, route smoke, eval records, eval summary, and `eval_evidence_manifest.json` once the local endpoint is live; `scripts/run_local_asr_evidence.py` captures Parakeet ASR provider evidence and `asr_evidence_manifest.json`. Pending no-cloud 50-case eval and real local ASR proof |
 
22
  | Parameter/evidence ledger | Present; organizer confirmation pending | [model_parameter_evidence_ledger.md](model_parameter_evidence_ledger.md) |
23
  | Submission claim audit | Present | `make audit-claims` scans submission-facing copy for premature Off the Grid, Llama Champion, Well-Tuned, Backyard user-use, local 4B, local ASR, demo-video, and social-post claims |
24
- | Evidence gate status report | Present; incomplete by design until external proofs exist | `make evidence-gates` reports each evidence gate, paths found, and next actions. Current missing gates include local 4B 50-case eval, no-cloud route, Llama Champion route, local ASR provider proof, trained-responder user test, demo video, social post, and Well-Tuned adapter |
25
 
26
  ## Badge And Claim Status
27
 
@@ -33,8 +37,8 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
33
  | Demo video | Targeted / proof-needed | Final video link showing only verified routes and labeling fallbacks honestly |
34
  | Social post | Targeted / proof-needed | Final social post link with achieved-versus-targeted wording |
35
  | Llama Champion | Targeted / proof-needed | Eligible local model route through llama.cpp with trace or eval evidence |
36
- | Sharing is Caring | Targeted / proof-needed | Public Space, repo, demo video, social post, and open trace links |
37
- | Well-Tuned | Stretch only / proof-needed | Published fine-tuned model or adapter used by the app, plus measured result. Fallback output cannot count |
38
  | Field Notes | Tentative | Organizer confirmation and final field-note artifact |
39
  | Off-Brand | Targeted / proof-needed | Final demo or social artifact meeting organizer criteria |
40
 
@@ -50,6 +54,8 @@ The README may say the project is built for a real trained responder. It should
50
 
51
  The hosted Omni eval proves model and fallback behavior through the eval harness. The public Space proof currently proves deployment health and no-secret fallback behavior, not live hosted Omni generation. Use the follow-up hosted eval run as the current hosted eval score: 31/50 whole-output hosted competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation.
52
 
 
 
53
  Final validation is app safety. Whole-output model competence and field-level model retention are the model-load-bearing metrics. Deterministic fallback and deterministic patches must stay visible in traces, scorecards, submission copy, and the demo.
54
 
55
  ## Submission Copy Boundaries
@@ -59,7 +65,7 @@ Final validation is app safety. Whole-output model competence and field-level mo
59
  - Backyard AI: may say built for a real trained responder; do not say the target user used or tested Figment until factual notes exist.
60
  - Off the Grid: claim only after a recorded no-cloud run.
61
  - Llama Champion: claim only after an eligible llama.cpp route runs with trace or eval evidence.
62
- - Well-Tuned: claim only after a published fine-tuned model or adapter is used by the app and measured.
63
  - Parameter compliance: cite the [model parameter/evidence ledger](model_parameter_evidence_ledger.md), including the Omni 31B body-count versus 33B sidebar ambiguity and organizer-confirmation status.
64
 
65
  Run `make audit-claims` before final README, demo-script, or social-copy edits. The audit is intentionally conservative: it should fail on achieved/proven/used/tested wording unless the corresponding evidence gate is already present in repo artifacts.
 
1
  # Figment Submission Checklist
2
 
3
+ Status: living checklist for evidence. Last live Hub verification: 2026-06-15. Keep claims in README, submission copy, demo video, and social posts aligned with this file.
4
 
5
  Primary tracker: [adversarial review action items](adversarial-review-action-items.md).
6
 
 
9
  | Artifact | Status | Link / evidence |
10
  | -------- | ------ | --------------- |
11
  | Public Hugging Face Space | Runnable in no-secret canned-fallback mode | https://huggingface.co/spaces/build-small-hackathon/figment |
12
+ | Space cold boot with app files present | Verified 2026-06-15 | Space API `runtime.stage=RUNNING`, `hardware=cpu-basic`, `sha=94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`, 167 siblings, `app.py` and `figment/observation_targets.py` present; Space URL served HTTP 200 |
13
+ | GitHub source repo | Pushed | https://github.com/ThomsenDrake/figment at `c15ed004ddf25ca4726e43d1a74da563b5504830` |
14
+ | Public fine-tuned model archive | Published | https://huggingface.co/build-small-hackathon/figment-finetuned-model-archive at `7da772ec7c0de20011d42780ea8afa65af4aef70`; includes v1 pilot plus v5-v14p BF16/GGUF artifacts, merge manifests, and model card |
15
+ | Public eval/training dataset repo | Published | https://huggingface.co/datasets/build-small-hackathon/figment-eval-traces at `92d9564fe6c984c55d65c7ba35a4e04eddcdea01`; configs load for `default` plus `figment_sft_v1` through `figment_sft_v14p`; v14p viewer split is 4801 train / 534 validation rows with 47 columns |
16
  | Demo video | Proof needed | Pending |
17
  | Social post | Proof needed | Pending |
18
  | Safety statement | Present | [safety_statement.md](safety_statement.md) |
 
21
  | Live hosted Omni trace | Eval traces present; final demo trace still needed | Baseline: `traces/hosted_omni_eval_20260607T194833Z.jsonl`; follow-up: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl` |
22
  | No-cloud/off-grid trace | Proof needed before claiming Off the Grid achieved | Pending |
23
  | Hosted Omni eval results | Measured | [hosted_omni_eval_results.md](hosted_omni_eval_results.md): 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation in the follow-up run |
24
+ | 4B LoRA system eval results | Published and measured | v14p repair-union on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, 0 fallback; raw first-pass success is 146/150 and 4/150 close through focused model repair. This is model-system evidence, not no-cloud proof or clinical validation |
25
+ | Local 4B + Parakeet no-cloud evidence | Helpers ready; proof needed | Full BF16 4B snapshot and Parakeet `.nemo` artifact are present locally; `scripts/run_local_4b_evidence.py` captures endpoint metadata, route smoke, eval records, eval summary, and `eval_evidence_manifest.json` once the local endpoint is live; `scripts/run_local_asr_evidence.py` captures Parakeet ASR provider evidence and `asr_evidence_manifest.json`. Pending no-cloud 50-case eval and real local ASR proof |
26
  | Parameter/evidence ledger | Present; organizer confirmation pending | [model_parameter_evidence_ledger.md](model_parameter_evidence_ledger.md) |
27
  | Submission claim audit | Present | `make audit-claims` scans submission-facing copy for premature Off the Grid, Llama Champion, Well-Tuned, Backyard user-use, local 4B, local ASR, demo-video, and social-post claims |
28
+ | Evidence gate status report | Present; incomplete by design until external proofs exist | `make evidence-gates` reports each evidence gate, paths found, and next actions. Current missing gates are badge/demo/user-proof gates, not missing Hub repos: no-cloud route, Llama Champion route, local ASR provider proof, trained-responder user test, demo video, and social post |
29
 
30
  ## Badge And Claim Status
31
 
 
37
  | Demo video | Targeted / proof-needed | Final video link showing only verified routes and labeling fallbacks honestly |
38
  | Social post | Targeted / proof-needed | Final social post link with achieved-versus-targeted wording |
39
  | Llama Champion | Targeted / proof-needed | Eligible local model route through llama.cpp with trace or eval evidence |
40
+ | Sharing is Caring | Public Space, GitHub repo, model archive, dataset repo, and trace links are ready; demo video and social post still pending | Final demo video and social post links |
41
+ | Well-Tuned | Published measured tuned 4B artifacts exist; claim wording should stay tied to the model archive and local route support, not the no-secret hosted Space route | Organizer accepts the published merged-model archive plus measured v14p result as the Well-Tuned artifact. Fallback output cannot count |
42
  | Field Notes | Tentative | Organizer confirmation and final field-note artifact |
43
  | Off-Brand | Targeted / proof-needed | Final demo or social artifact meeting organizer criteria |
44
 
 
54
 
55
  The hosted Omni eval proves model and fallback behavior through the eval harness. The public Space proof currently proves deployment health and no-secret fallback behavior, not live hosted Omni generation. Use the follow-up hosted eval run as the current hosted eval score: 31/50 whole-output hosted competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation.
56
 
57
+ The 4B LoRA model archive proves a published, measured tuned local-model artifact. Use the v14p repair-union result as the current small-model score: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback on the corrected 150-case field-workflow holdout. Do not convert that into Off the Grid, local ASR, target-user, or clinical-validation evidence.
58
+
59
  Final validation is app safety. Whole-output model competence and field-level model retention are the model-load-bearing metrics. Deterministic fallback and deterministic patches must stay visible in traces, scorecards, submission copy, and the demo.
60
 
61
  ## Submission Copy Boundaries
 
65
  - Backyard AI: may say built for a real trained responder; do not say the target user used or tested Figment until factual notes exist.
66
  - Off the Grid: claim only after a recorded no-cloud run.
67
  - Llama Champion: claim only after an eligible llama.cpp route runs with trace or eval evidence.
68
+ - Well-Tuned: may cite the published measured v14p tuned artifacts, but do not imply the public no-secret Space is serving them.
69
  - Parameter compliance: cite the [model parameter/evidence ledger](model_parameter_evidence_ledger.md), including the Omni 31B body-count versus 33B sidebar ambiguity and organizer-confirmation status.
70
 
71
  Run `make audit-claims` before final README, demo-script, or social-copy edits. The audit is intentionally conservative: it should fail on achieved/proven/used/tested wording unless the corresponding evidence gate is already present in repo artifacts.
scripts/audit_submission_claims.py CHANGED
@@ -224,9 +224,14 @@ def _has_no_cloud_evidence(repo_root: Path) -> bool:
224
 
225
  def _has_well_tuned_evidence(repo_root: Path) -> bool:
226
  ledger = (repo_root / "docs/model_parameter_evidence_ledger.md").read_text(encoding="utf-8")
 
227
  return (
228
- "published fine-tuned model" in ledger.lower()
229
- and "not trained, published, or measured" not in ledger.lower()
 
 
 
 
230
  )
231
 
232
 
 
224
 
225
  def _has_well_tuned_evidence(repo_root: Path) -> bool:
226
  ledger = (repo_root / "docs/model_parameter_evidence_ledger.md").read_text(encoding="utf-8")
227
+ lowered = ledger.lower()
228
  return (
229
+ "build-small-hackathon/figment-finetuned-model-archive" in lowered
230
+ and "figment_sft_v14p" in lowered
231
+ and "v14p repair-union" in lowered
232
+ and "150/150 competence" in lowered
233
+ and "do not imply the public no-secret space is serving" in lowered
234
+ and "not trained, published, or measured" not in lowered
235
  )
236
 
237
 
scripts/evidence_gate_status.py CHANGED
@@ -91,10 +91,22 @@ def _public_space_gate(repo_root: Path) -> dict[str, Any]:
91
 
92
  def _hosted_eval_gate(repo_root: Path) -> dict[str, Any]:
93
  traces = sorted(repo_root.glob("traces/hosted_omni_eval*.jsonl"))
 
 
 
 
 
 
 
 
 
 
 
 
94
  return _simple_gate(
95
- passed=bool(traces),
96
  label="Hosted Omni eval",
97
- required_evidence="Hosted Omni eval JSONL trace and scorecard.",
98
  evidence_paths=[str(path) for path in traces]
99
  + _existing_paths(repo_root, [Path("docs/hosted_omni_eval_results.md")]),
100
  next_action="Run or refresh the hosted Omni eval and update docs/hosted_omni_eval_results.md.",
 
91
 
92
  def _hosted_eval_gate(repo_root: Path) -> dict[str, Any]:
93
  traces = sorted(repo_root.glob("traces/hosted_omni_eval*.jsonl"))
94
+ scorecard_path = repo_root / "docs/hosted_omni_eval_results.md"
95
+ scorecard = _read_text(scorecard_path)
96
+ scorecard_has_current_metrics = all(
97
+ marker in scorecard
98
+ for marker in (
99
+ "31/50",
100
+ "8/50",
101
+ "480/650",
102
+ "170/650",
103
+ "50/50",
104
+ )
105
+ )
106
  return _simple_gate(
107
+ passed=bool(traces) or scorecard_has_current_metrics,
108
  label="Hosted Omni eval",
109
+ required_evidence="Hosted Omni eval JSONL trace or committed scorecard.",
110
  evidence_paths=[str(path) for path in traces]
111
  + _existing_paths(repo_root, [Path("docs/hosted_omni_eval_results.md")]),
112
  next_action="Run or refresh the hosted Omni eval and update docs/hosted_omni_eval_results.md.",
tests/test_evidence_gate_status.py CHANGED
@@ -10,6 +10,7 @@ def test_current_repo_report_keeps_remaining_external_gates_incomplete() -> None
10
  assert report["status"] == "incomplete"
11
  assert report["ready_for_badge_claims"] is False
12
  assert report["gates"]["claim_audit"]["passed"] is True
 
13
  assert report["gates"]["local_4b_50_case_eval"]["passed"] is False
14
  assert report["gates"]["no_cloud_route"]["passed"] is False
15
  assert report["gates"]["llama_champion_route"]["passed"] is False
 
10
  assert report["status"] == "incomplete"
11
  assert report["ready_for_badge_claims"] is False
12
  assert report["gates"]["claim_audit"]["passed"] is True
13
+ assert report["gates"]["hosted_omni_eval"]["passed"] is True
14
  assert report["gates"]["local_4b_50_case_eval"]["passed"] is False
15
  assert report["gates"]["no_cloud_route"]["passed"] is False
16
  assert report["gates"]["llama_champion_route"]["passed"] is False
tests/test_submission_claim_audit.py CHANGED
@@ -11,6 +11,7 @@ def test_current_submission_claims_stay_evidence_gated() -> None:
11
  assert report["gate_status"]["off_grid"] is False
12
  assert report["gate_status"]["local_4b"] is False
13
  assert report["gate_status"]["local_asr"] is False
 
14
  assert report["gate_status"]["backyard_user_use"] is False
15
 
16
 
@@ -51,3 +52,35 @@ def test_audit_flags_unproven_user_use_and_local_asr_claims() -> None:
51
 
52
  assert [violation["gate"] for violation in violations] == ["backyard_user_use", "local_asr"]
53
  assert {violation["file"] for violation in violations} == {"submission.md"}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  assert report["gate_status"]["off_grid"] is False
12
  assert report["gate_status"]["local_4b"] is False
13
  assert report["gate_status"]["local_asr"] is False
14
+ assert report["gate_status"]["well_tuned"] is True
15
  assert report["gate_status"]["backyard_user_use"] is False
16
 
17
 
 
52
 
53
  assert [violation["gate"] for violation in violations] == ["backyard_user_use", "local_asr"]
54
  assert {violation["file"] for violation in violations} == {"submission.md"}
55
+
56
+
57
+ def test_well_tuned_gate_requires_archive_result_and_public_space_caveat(tmp_path: Path) -> None:
58
+ docs_dir = tmp_path / "docs"
59
+ docs_dir.mkdir()
60
+ ledger = docs_dir / "model_parameter_evidence_ledger.md"
61
+ ledger.write_text(
62
+ "\n".join(
63
+ [
64
+ "Archive: build-small-hackathon/figment-finetuned-model-archive",
65
+ "Version: figment_sft_v14p",
66
+ "Result: v14p repair-union with 150/150 competence",
67
+ "Boundary: Do not imply the public no-secret Space is serving this tuned model.",
68
+ ]
69
+ ),
70
+ encoding="utf-8",
71
+ )
72
+
73
+ assert audit_submission_claims._has_well_tuned_evidence(tmp_path) is True
74
+
75
+ ledger.write_text(
76
+ "\n".join(
77
+ [
78
+ "Archive: build-small-hackathon/figment-finetuned-model-archive",
79
+ "Version: figment_sft_v14p",
80
+ "Result: v14p repair-union with 150/150 competence",
81
+ ]
82
+ ),
83
+ encoding="utf-8",
84
+ )
85
+
86
+ assert audit_submission_claims._has_well_tuned_evidence(tmp_path) is False