Spaces:
Running on Zero
Running on Zero
Refresh submission evidence gates
Browse files
docs/model_parameter_evidence_ledger.md
CHANGED
|
@@ -1,15 +1,16 @@
|
|
| 1 |
# Figment Model Parameter And Evidence Ledger
|
| 2 |
|
| 3 |
-
Date: 2026-06-
|
| 4 |
|
| 5 |
Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmation claims in one evidence-gated place. README, submission copy, demo video, and social posts should not upgrade a claim beyond this ledger.
|
| 6 |
|
| 7 |
## Current Claim Boundary
|
| 8 |
|
| 9 |
- Hosted Omni has measured eval evidence through the eval harness. The public Space is now verified runnable in no-secret canned-fallback mode, but that public Space proof is not live hosted Omni generation evidence.
|
| 10 |
-
- The public Space target exists and cold-boots from the Space URL. Current public API evidence: `runtime.stage=RUNNING`, `
|
| 11 |
-
- The
|
| 12 |
-
-
|
|
|
|
| 13 |
- Organizer confirmation is still needed for the Omni 31B body-count versus 33B sidebar ambiguity and for any additive local stack or adapter-count interpretation.
|
| 14 |
|
| 15 |
## Parameter Ledger
|
|
@@ -18,20 +19,20 @@ Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmati
|
|
| 18 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 19 |
| Hosted Omni primary | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`; API route `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | NVIDIA model-card body: 31B total parameters; HF sidebar has been observed as 33B in planning docs | Roughly 3B active parameters per token is a runtime/MoE note, not the compliance number | None used in current evals | Native Omni speech encoder is part of the Omni model-card count; no separate ASR model is claimed for hosted Omni | Hosted NVIDIA API route in current evals; self-hosted no-cloud route not recorded | Pending: ask organizers whether model-card body count is acceptable if sidebar count differs | Baseline eval: 28/50 whole-output competence, 22/50 full fallback, 50/50 final validation. Follow-up eval: 31/50 competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation |
|
| 20 |
| Self-hosted Omni no-cloud target | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`, FP8, or NVFP4 variant if served locally | Same Omni 31B body-count claim, with same 33B sidebar ambiguity | Active parameters do not decide compliance | None recorded | Native Omni audio if used locally; included in Omni count if organizers accept the model-card count | Would be local/self-hosted only if served with no runtime cloud APIs | Pending for count ambiguity and hardware/runtime proof | No recorded no-cloud eval or public demo trace yet. Do not claim Off the Grid achieved |
|
| 21 |
-
| Local 4B + Parakeet proof path | Text: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`; ASR: `nvidia/parakeet-rnnt-1.1b`; local route `MODEL_BACKEND=llama_cpp`, `MODEL_STACK=local_4b_parakeet` | Workback model-card notes: 3.97B text model plus about 1.1B Parakeet, roughly 5.1B nominal before adapters | No active-parameter substitution; use additive total-count story if organizers require stack accounting |
|
| 22 |
-
| 4B Figment
|
| 23 |
| Canned fallback | No live model | Not a model-compliance artifact | Not applicable | Not applicable | Not applicable | Local deterministic fallback; public Space no-secret fallback verified | Not applicable | Useful for safety, deployment health, and cold-start fallback only. Cannot count as model competence, Off the Grid proof, Llama Champion proof, or Well-Tuned proof |
|
| 24 |
|
| 25 |
## Submission Gates
|
| 26 |
|
| 27 |
| Claim | Required upgrade evidence |
|
| 28 |
| --- | --- |
|
| 29 |
-
| Public Space runnable | Satisfied for no-secret canned-fallback mode at Space commit `
|
| 30 |
| Hosted model load-bearing | Cite hosted eval metrics separately from final validation: 31/50 whole-output competence and 480/650 model-retained fields in the follow-up run |
|
| 31 |
| <=32B hosted Omni compliance | Organizer accepts the 31B model-card body count or the submission falls back to a clearly eligible smaller route |
|
| 32 |
| Off the Grid | Recorded no-cloud run with trace evidence, either self-hosted Omni or local 4B + Parakeet/typed intake |
|
| 33 |
| Llama Champion | Eligible model route runs through llama.cpp with trace or eval evidence |
|
| 34 |
-
| Well-Tuned |
|
| 35 |
| Backyard AI user use | Completed user-test notes from a real trained responder on synthetic or de-identified scenarios |
|
| 36 |
| Demo video and social post | Final links exist and wording says achieved only for artifacts supported by this ledger |
|
| 37 |
|
|
|
|
| 1 |
# Figment Model Parameter And Evidence Ledger
|
| 2 |
|
| 3 |
+
Date: 2026-06-15
|
| 4 |
|
| 5 |
Purpose: keep parameter, route, locality, adapter, ASR, and organizer-confirmation claims in one evidence-gated place. README, submission copy, demo video, and social posts should not upgrade a claim beyond this ledger.
|
| 6 |
|
| 7 |
## Current Claim Boundary
|
| 8 |
|
| 9 |
- Hosted Omni has measured eval evidence through the eval harness. The public Space is now verified runnable in no-secret canned-fallback mode, but that public Space proof is not live hosted Omni generation evidence.
|
| 10 |
+
- The public Space target exists and cold-boots from the Space URL. Current public API evidence: `runtime.stage=RUNNING`, `hardware=cpu-basic`, `sha=94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`, 167 siblings, `app.py` present, `figment/observation_targets.py` present, and Space URL HTTP 200. Public workflow evidence remains no-secret canned fallback unless a later non-canned public trace is recorded.
|
| 11 |
+
- The public model archive now records published measured Figment 4B LoRA merged artifacts through `figment_sft_v14p`. The strongest measured result is v14p repair-union on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback. This is tuned model-system evidence, not no-cloud or clinical validation evidence.
|
| 12 |
+
- The local 4B + Parakeet route is still the preferred no-cloud/off-grid proof path. The full BF16 4B artifact and Parakeet artifact are present locally, and the tuned v14p GGUF is published, but no-cloud route proof and local ASR provider proof are still separate gates.
|
| 13 |
+
- Well-Tuned wording may cite the published measured v14p tuned artifacts if the submission/judging interpretation accepts the merged-model archive and local route support. Do not imply the public no-secret Space is serving the tuned model.
|
| 14 |
- Organizer confirmation is still needed for the Omni 31B body-count versus 33B sidebar ambiguity and for any additive local stack or adapter-count interpretation.
|
| 15 |
|
| 16 |
## Parameter Ledger
|
|
|
|
| 19 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 20 |
| Hosted Omni primary | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`; API route `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | NVIDIA model-card body: 31B total parameters; HF sidebar has been observed as 33B in planning docs | Roughly 3B active parameters per token is a runtime/MoE note, not the compliance number | None used in current evals | Native Omni speech encoder is part of the Omni model-card count; no separate ASR model is claimed for hosted Omni | Hosted NVIDIA API route in current evals; self-hosted no-cloud route not recorded | Pending: ask organizers whether model-card body count is acceptable if sidebar count differs | Baseline eval: 28/50 whole-output competence, 22/50 full fallback, 50/50 final validation. Follow-up eval: 31/50 competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation |
|
| 21 |
| Self-hosted Omni no-cloud target | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`, FP8, or NVFP4 variant if served locally | Same Omni 31B body-count claim, with same 33B sidebar ambiguity | Active parameters do not decide compliance | None recorded | Native Omni audio if used locally; included in Omni count if organizers accept the model-card count | Would be local/self-hosted only if served with no runtime cloud APIs | Pending for count ambiguity and hardware/runtime proof | No recorded no-cloud eval or public demo trace yet. Do not claim Off the Grid achieved |
|
| 22 |
+
| Local 4B + Parakeet proof path | Text base: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`; tuned archive: `build-small-hackathon/figment-finetuned-model-archive`; ASR: `nvidia/parakeet-rnnt-1.1b`; local route `MODEL_BACKEND=llama_cpp`, `MODEL_STACK=local_4b_parakeet` | Workback model-card notes: 3.97B text model plus about 1.1B Parakeet, roughly 5.1B nominal before adapters | No active-parameter substitution; use additive total-count story if organizers require stack accounting | Published merged LoRA artifacts exist through `figment_sft_v14p`; exact adapter-only parameter count is not used as the primary public claim | About 1.1B for Parakeet RNNT ASR, only if real local ASR is enabled and proven | Intended local OpenAI-compatible endpoint serving the tuned GGUF plus local ASR; no-cloud only after recorded proof | Pending: confirm additive multi-model counting and adapter-count treatment | Full BF16 4B snapshot downloaded at repo revision `dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f`; `model.safetensors` is `7947142640` bytes with SHA-256 `55d4e2519456c4a9bddf596b0748d630e3b2ce6ff6f4c2b7ed3e07e2b00dad42`. Parakeet artifact downloaded at repo revision `a07b19e98a26c1873a3f2622c446a4a1ca6316cb`; `parakeet-rnnt-1.1b.nemo` is `4283105280` bytes with SHA-256 `535896f014953d945b287ac533560e20da8103c6781b152de4645528e2b60738`. The v14p repair-union measured result is 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback on the corrected 150-case field-workflow holdout. No no-cloud local route proof, local ASR provider proof, or target-user trace is recorded yet |
|
| 23 |
+
| 4B Figment tuned archive | `build-small-hackathon/figment-finetuned-model-archive`, especially `figment_sft_v14p/figment-sft-v14p-lora-merged-bf16/` and `figment_sft_v14p/figment-sft-v14p-lora-merged-bf16.bf16.gguf` | Base model is the 3.97B Nemotron 4B BF16 target; merged LoRA artifacts are derived from that base | Not applicable | Published as merged BF16 and GGUF artifacts with merge manifests; adapter-only count remains a documentation detail if needed by organizers | None for text-only eval | Public HF model repo plus local OpenAI-compatible app route support | Pending only for any organizer-specific adapter-count interpretation | Public model repo is `private=False`, current commit `7da772ec7c0de20011d42780ea8afa65af4aef70`, with v1 pilot plus v5-v14p artifacts. v14p model card records the repair-union result and GGUF SHA. |
|
| 24 |
| Canned fallback | No live model | Not a model-compliance artifact | Not applicable | Not applicable | Not applicable | Local deterministic fallback; public Space no-secret fallback verified | Not applicable | Useful for safety, deployment health, and cold-start fallback only. Cannot count as model competence, Off the Grid proof, Llama Champion proof, or Well-Tuned proof |
|
| 25 |
|
| 26 |
## Submission Gates
|
| 27 |
|
| 28 |
| Claim | Required upgrade evidence |
|
| 29 |
| --- | --- |
|
| 30 |
+
| Public Space runnable | Satisfied for no-secret canned-fallback mode at Space commit `94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`: app files present, clean cold boot from the Space URL, and route/fallback labeling preserved. Non-canned public model traces remain separate proof. |
|
| 31 |
| Hosted model load-bearing | Cite hosted eval metrics separately from final validation: 31/50 whole-output competence and 480/650 model-retained fields in the follow-up run |
|
| 32 |
| <=32B hosted Omni compliance | Organizer accepts the 31B model-card body count or the submission falls back to a clearly eligible smaller route |
|
| 33 |
| Off the Grid | Recorded no-cloud run with trace evidence, either self-hosted Omni or local 4B + Parakeet/typed intake |
|
| 34 |
| Llama Champion | Eligible model route runs through llama.cpp with trace or eval evidence |
|
| 35 |
+
| Well-Tuned | Cite the published measured v14p tuned archive only with wording that does not imply the no-secret public Space is serving it |
|
| 36 |
| Backyard AI user use | Completed user-test notes from a real trained responder on synthetic or de-identified scenarios |
|
| 37 |
| Demo video and social post | Final links exist and wording says achieved only for artifacts supported by this ledger |
|
| 38 |
|
docs/safety_statement.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Figment Safety Statement
|
| 2 |
|
| 3 |
-
Status: submission-ready draft.
|
| 4 |
|
| 5 |
## Intended Use
|
| 6 |
|
|
@@ -45,18 +45,21 @@ Achieved in-repo artifacts:
|
|
| 45 |
- hosted NVIDIA Omni client path and labeled canned fallback path;
|
| 46 |
- trace and validator surfaces for review;
|
| 47 |
- synthetic demo-audio asset path;
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
- this safety statement;
|
| 49 |
- Apache-2.0 license file.
|
| 50 |
|
| 51 |
Proof still needed before final submission claims:
|
| 52 |
|
| 53 |
-
- public Hugging Face Space cold boot with app files present;
|
| 54 |
- recorded live model run whose trace is not canned fallback;
|
| 55 |
- recorded no-cloud run before claiming Off the Grid as achieved;
|
| 56 |
- real trained-responder user-test notes using synthetic or de-identified scenarios;
|
|
|
|
| 57 |
- demo video link;
|
| 58 |
-
- social post link
|
| 59 |
-
- measured eval results before claiming model quality thresholds.
|
| 60 |
|
| 61 |
## Non-Goals
|
| 62 |
|
|
|
|
| 1 |
# Figment Safety Statement
|
| 2 |
|
| 3 |
+
Status: submission-ready draft. Evidence links refreshed 2026-06-15; do not turn pending items into claims until the artifacts exist.
|
| 4 |
|
| 5 |
## Intended Use
|
| 6 |
|
|
|
|
| 45 |
- hosted NVIDIA Omni client path and labeled canned fallback path;
|
| 46 |
- trace and validator surfaces for review;
|
| 47 |
- synthetic demo-audio asset path;
|
| 48 |
+
- public Hugging Face Space cold boot with app files present at commit `94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`;
|
| 49 |
+
- published 4B LoRA model archive and eval/training dataset repo;
|
| 50 |
+
- measured v14p repair-union result on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback;
|
| 51 |
+
- hosted Omni follow-up eval: 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation;
|
| 52 |
- this safety statement;
|
| 53 |
- Apache-2.0 license file.
|
| 54 |
|
| 55 |
Proof still needed before final submission claims:
|
| 56 |
|
|
|
|
| 57 |
- recorded live model run whose trace is not canned fallback;
|
| 58 |
- recorded no-cloud run before claiming Off the Grid as achieved;
|
| 59 |
- real trained-responder user-test notes using synthetic or de-identified scenarios;
|
| 60 |
+
- local Parakeet ASR provider proof before claiming local audio;
|
| 61 |
- demo video link;
|
| 62 |
+
- social post link.
|
|
|
|
| 63 |
|
| 64 |
## Non-Goals
|
| 65 |
|
docs/submission_checklist.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Figment Submission Checklist
|
| 2 |
|
| 3 |
-
Status: living checklist for evidence. Keep claims in README, submission copy, demo video, and social posts aligned with this file.
|
| 4 |
|
| 5 |
Primary tracker: [adversarial review action items](adversarial-review-action-items.md).
|
| 6 |
|
|
@@ -9,7 +9,10 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
|
|
| 9 |
| Artifact | Status | Link / evidence |
|
| 10 |
| -------- | ------ | --------------- |
|
| 11 |
| Public Hugging Face Space | Runnable in no-secret canned-fallback mode | https://huggingface.co/spaces/build-small-hackathon/figment |
|
| 12 |
-
| Space cold boot with app files present | Verified 2026-06-
|
|
|
|
|
|
|
|
|
|
| 13 |
| Demo video | Proof needed | Pending |
|
| 14 |
| Social post | Proof needed | Pending |
|
| 15 |
| Safety statement | Present | [safety_statement.md](safety_statement.md) |
|
|
@@ -18,10 +21,11 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
|
|
| 18 |
| Live hosted Omni trace | Eval traces present; final demo trace still needed | Baseline: `traces/hosted_omni_eval_20260607T194833Z.jsonl`; follow-up: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl` |
|
| 19 |
| No-cloud/off-grid trace | Proof needed before claiming Off the Grid achieved | Pending |
|
| 20 |
| Hosted Omni eval results | Measured | [hosted_omni_eval_results.md](hosted_omni_eval_results.md): 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation in the follow-up run |
|
| 21 |
-
|
|
|
|
|
| 22 |
| Parameter/evidence ledger | Present; organizer confirmation pending | [model_parameter_evidence_ledger.md](model_parameter_evidence_ledger.md) |
|
| 23 |
| Submission claim audit | Present | `make audit-claims` scans submission-facing copy for premature Off the Grid, Llama Champion, Well-Tuned, Backyard user-use, local 4B, local ASR, demo-video, and social-post claims |
|
| 24 |
-
| Evidence gate status report | Present; incomplete by design until external proofs exist | `make evidence-gates` reports each evidence gate, paths found, and next actions. Current missing gates
|
| 25 |
|
| 26 |
## Badge And Claim Status
|
| 27 |
|
|
@@ -33,8 +37,8 @@ Primary tracker: [adversarial review action items](adversarial-review-action-ite
|
|
| 33 |
| Demo video | Targeted / proof-needed | Final video link showing only verified routes and labeling fallbacks honestly |
|
| 34 |
| Social post | Targeted / proof-needed | Final social post link with achieved-versus-targeted wording |
|
| 35 |
| Llama Champion | Targeted / proof-needed | Eligible local model route through llama.cpp with trace or eval evidence |
|
| 36 |
-
| Sharing is Caring |
|
| 37 |
-
| Well-Tuned |
|
| 38 |
| Field Notes | Tentative | Organizer confirmation and final field-note artifact |
|
| 39 |
| Off-Brand | Targeted / proof-needed | Final demo or social artifact meeting organizer criteria |
|
| 40 |
|
|
@@ -50,6 +54,8 @@ The README may say the project is built for a real trained responder. It should
|
|
| 50 |
|
| 51 |
The hosted Omni eval proves model and fallback behavior through the eval harness. The public Space proof currently proves deployment health and no-secret fallback behavior, not live hosted Omni generation. Use the follow-up hosted eval run as the current hosted eval score: 31/50 whole-output hosted competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation.
|
| 52 |
|
|
|
|
|
|
|
| 53 |
Final validation is app safety. Whole-output model competence and field-level model retention are the model-load-bearing metrics. Deterministic fallback and deterministic patches must stay visible in traces, scorecards, submission copy, and the demo.
|
| 54 |
|
| 55 |
## Submission Copy Boundaries
|
|
@@ -59,7 +65,7 @@ Final validation is app safety. Whole-output model competence and field-level mo
|
|
| 59 |
- Backyard AI: may say built for a real trained responder; do not say the target user used or tested Figment until factual notes exist.
|
| 60 |
- Off the Grid: claim only after a recorded no-cloud run.
|
| 61 |
- Llama Champion: claim only after an eligible llama.cpp route runs with trace or eval evidence.
|
| 62 |
-
- Well-Tuned:
|
| 63 |
- Parameter compliance: cite the [model parameter/evidence ledger](model_parameter_evidence_ledger.md), including the Omni 31B body-count versus 33B sidebar ambiguity and organizer-confirmation status.
|
| 64 |
|
| 65 |
Run `make audit-claims` before final README, demo-script, or social-copy edits. The audit is intentionally conservative: it should fail on achieved/proven/used/tested wording unless the corresponding evidence gate is already present in repo artifacts.
|
|
|
|
| 1 |
# Figment Submission Checklist
|
| 2 |
|
| 3 |
+
Status: living checklist for evidence. Last live Hub verification: 2026-06-15. Keep claims in README, submission copy, demo video, and social posts aligned with this file.
|
| 4 |
|
| 5 |
Primary tracker: [adversarial review action items](adversarial-review-action-items.md).
|
| 6 |
|
|
|
|
| 9 |
| Artifact | Status | Link / evidence |
|
| 10 |
| -------- | ------ | --------------- |
|
| 11 |
| Public Hugging Face Space | Runnable in no-secret canned-fallback mode | https://huggingface.co/spaces/build-small-hackathon/figment |
|
| 12 |
+
| Space cold boot with app files present | Verified 2026-06-15 | Space API `runtime.stage=RUNNING`, `hardware=cpu-basic`, `sha=94cbe85db8d4b766d7c6a59dfee7071fe98e9a3f`, 167 siblings, `app.py` and `figment/observation_targets.py` present; Space URL served HTTP 200 |
|
| 13 |
+
| GitHub source repo | Pushed | https://github.com/ThomsenDrake/figment at `c15ed004ddf25ca4726e43d1a74da563b5504830` |
|
| 14 |
+
| Public fine-tuned model archive | Published | https://huggingface.co/build-small-hackathon/figment-finetuned-model-archive at `7da772ec7c0de20011d42780ea8afa65af4aef70`; includes v1 pilot plus v5-v14p BF16/GGUF artifacts, merge manifests, and model card |
|
| 15 |
+
| Public eval/training dataset repo | Published | https://huggingface.co/datasets/build-small-hackathon/figment-eval-traces at `92d9564fe6c984c55d65c7ba35a4e04eddcdea01`; configs load for `default` plus `figment_sft_v1` through `figment_sft_v14p`; v14p viewer split is 4801 train / 534 validation rows with 47 columns |
|
| 16 |
| Demo video | Proof needed | Pending |
|
| 17 |
| Social post | Proof needed | Pending |
|
| 18 |
| Safety statement | Present | [safety_statement.md](safety_statement.md) |
|
|
|
|
| 21 |
| Live hosted Omni trace | Eval traces present; final demo trace still needed | Baseline: `traces/hosted_omni_eval_20260607T194833Z.jsonl`; follow-up: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl` |
|
| 22 |
| No-cloud/off-grid trace | Proof needed before claiming Off the Grid achieved | Pending |
|
| 23 |
| Hosted Omni eval results | Measured | [hosted_omni_eval_results.md](hosted_omni_eval_results.md): 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation in the follow-up run |
|
| 24 |
+
| 4B LoRA system eval results | Published and measured | v14p repair-union on the corrected 150-case field-workflow holdout: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, 0 fallback; raw first-pass success is 146/150 and 4/150 close through focused model repair. This is model-system evidence, not no-cloud proof or clinical validation |
|
| 25 |
+
| Local 4B + Parakeet no-cloud evidence | Helpers ready; proof needed | Full BF16 4B snapshot and Parakeet `.nemo` artifact are present locally; `scripts/run_local_4b_evidence.py` captures endpoint metadata, route smoke, eval records, eval summary, and `eval_evidence_manifest.json` once the local endpoint is live; `scripts/run_local_asr_evidence.py` captures Parakeet ASR provider evidence and `asr_evidence_manifest.json`. Pending no-cloud 50-case eval and real local ASR proof |
|
| 26 |
| Parameter/evidence ledger | Present; organizer confirmation pending | [model_parameter_evidence_ledger.md](model_parameter_evidence_ledger.md) |
|
| 27 |
| Submission claim audit | Present | `make audit-claims` scans submission-facing copy for premature Off the Grid, Llama Champion, Well-Tuned, Backyard user-use, local 4B, local ASR, demo-video, and social-post claims |
|
| 28 |
+
| Evidence gate status report | Present; incomplete by design until external proofs exist | `make evidence-gates` reports each evidence gate, paths found, and next actions. Current missing gates are badge/demo/user-proof gates, not missing Hub repos: no-cloud route, Llama Champion route, local ASR provider proof, trained-responder user test, demo video, and social post |
|
| 29 |
|
| 30 |
## Badge And Claim Status
|
| 31 |
|
|
|
|
| 37 |
| Demo video | Targeted / proof-needed | Final video link showing only verified routes and labeling fallbacks honestly |
|
| 38 |
| Social post | Targeted / proof-needed | Final social post link with achieved-versus-targeted wording |
|
| 39 |
| Llama Champion | Targeted / proof-needed | Eligible local model route through llama.cpp with trace or eval evidence |
|
| 40 |
+
| Sharing is Caring | Public Space, GitHub repo, model archive, dataset repo, and trace links are ready; demo video and social post still pending | Final demo video and social post links |
|
| 41 |
+
| Well-Tuned | Published measured tuned 4B artifacts exist; claim wording should stay tied to the model archive and local route support, not the no-secret hosted Space route | Organizer accepts the published merged-model archive plus measured v14p result as the Well-Tuned artifact. Fallback output cannot count |
|
| 42 |
| Field Notes | Tentative | Organizer confirmation and final field-note artifact |
|
| 43 |
| Off-Brand | Targeted / proof-needed | Final demo or social artifact meeting organizer criteria |
|
| 44 |
|
|
|
|
| 54 |
|
| 55 |
The hosted Omni eval proves model and fallback behavior through the eval harness. The public Space proof currently proves deployment health and no-secret fallback behavior, not live hosted Omni generation. Use the follow-up hosted eval run as the current hosted eval score: 31/50 whole-output hosted competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, and 50/50 final validation.
|
| 56 |
|
| 57 |
+
The 4B LoRA model archive proves a published, measured tuned local-model artifact. Use the v14p repair-union result as the current small-model score: 150/150 competence, 150/150 expected labels, 150/150 final validation, 0 deterministic patches, and 0 fallback on the corrected 150-case field-workflow holdout. Do not convert that into Off the Grid, local ASR, target-user, or clinical-validation evidence.
|
| 58 |
+
|
| 59 |
Final validation is app safety. Whole-output model competence and field-level model retention are the model-load-bearing metrics. Deterministic fallback and deterministic patches must stay visible in traces, scorecards, submission copy, and the demo.
|
| 60 |
|
| 61 |
## Submission Copy Boundaries
|
|
|
|
| 65 |
- Backyard AI: may say built for a real trained responder; do not say the target user used or tested Figment until factual notes exist.
|
| 66 |
- Off the Grid: claim only after a recorded no-cloud run.
|
| 67 |
- Llama Champion: claim only after an eligible llama.cpp route runs with trace or eval evidence.
|
| 68 |
+
- Well-Tuned: may cite the published measured v14p tuned artifacts, but do not imply the public no-secret Space is serving them.
|
| 69 |
- Parameter compliance: cite the [model parameter/evidence ledger](model_parameter_evidence_ledger.md), including the Omni 31B body-count versus 33B sidebar ambiguity and organizer-confirmation status.
|
| 70 |
|
| 71 |
Run `make audit-claims` before final README, demo-script, or social-copy edits. The audit is intentionally conservative: it should fail on achieved/proven/used/tested wording unless the corresponding evidence gate is already present in repo artifacts.
|
scripts/audit_submission_claims.py
CHANGED
|
@@ -224,9 +224,14 @@ def _has_no_cloud_evidence(repo_root: Path) -> bool:
|
|
| 224 |
|
| 225 |
def _has_well_tuned_evidence(repo_root: Path) -> bool:
|
| 226 |
ledger = (repo_root / "docs/model_parameter_evidence_ledger.md").read_text(encoding="utf-8")
|
|
|
|
| 227 |
return (
|
| 228 |
-
"
|
| 229 |
-
and "
|
|
|
|
|
|
|
|
|
|
|
|
|
| 230 |
)
|
| 231 |
|
| 232 |
|
|
|
|
| 224 |
|
| 225 |
def _has_well_tuned_evidence(repo_root: Path) -> bool:
|
| 226 |
ledger = (repo_root / "docs/model_parameter_evidence_ledger.md").read_text(encoding="utf-8")
|
| 227 |
+
lowered = ledger.lower()
|
| 228 |
return (
|
| 229 |
+
"build-small-hackathon/figment-finetuned-model-archive" in lowered
|
| 230 |
+
and "figment_sft_v14p" in lowered
|
| 231 |
+
and "v14p repair-union" in lowered
|
| 232 |
+
and "150/150 competence" in lowered
|
| 233 |
+
and "do not imply the public no-secret space is serving" in lowered
|
| 234 |
+
and "not trained, published, or measured" not in lowered
|
| 235 |
)
|
| 236 |
|
| 237 |
|
scripts/evidence_gate_status.py
CHANGED
|
@@ -91,10 +91,22 @@ def _public_space_gate(repo_root: Path) -> dict[str, Any]:
|
|
| 91 |
|
| 92 |
def _hosted_eval_gate(repo_root: Path) -> dict[str, Any]:
|
| 93 |
traces = sorted(repo_root.glob("traces/hosted_omni_eval*.jsonl"))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
return _simple_gate(
|
| 95 |
-
passed=bool(traces),
|
| 96 |
label="Hosted Omni eval",
|
| 97 |
-
required_evidence="Hosted Omni eval JSONL trace
|
| 98 |
evidence_paths=[str(path) for path in traces]
|
| 99 |
+ _existing_paths(repo_root, [Path("docs/hosted_omni_eval_results.md")]),
|
| 100 |
next_action="Run or refresh the hosted Omni eval and update docs/hosted_omni_eval_results.md.",
|
|
|
|
| 91 |
|
| 92 |
def _hosted_eval_gate(repo_root: Path) -> dict[str, Any]:
|
| 93 |
traces = sorted(repo_root.glob("traces/hosted_omni_eval*.jsonl"))
|
| 94 |
+
scorecard_path = repo_root / "docs/hosted_omni_eval_results.md"
|
| 95 |
+
scorecard = _read_text(scorecard_path)
|
| 96 |
+
scorecard_has_current_metrics = all(
|
| 97 |
+
marker in scorecard
|
| 98 |
+
for marker in (
|
| 99 |
+
"31/50",
|
| 100 |
+
"8/50",
|
| 101 |
+
"480/650",
|
| 102 |
+
"170/650",
|
| 103 |
+
"50/50",
|
| 104 |
+
)
|
| 105 |
+
)
|
| 106 |
return _simple_gate(
|
| 107 |
+
passed=bool(traces) or scorecard_has_current_metrics,
|
| 108 |
label="Hosted Omni eval",
|
| 109 |
+
required_evidence="Hosted Omni eval JSONL trace or committed scorecard.",
|
| 110 |
evidence_paths=[str(path) for path in traces]
|
| 111 |
+ _existing_paths(repo_root, [Path("docs/hosted_omni_eval_results.md")]),
|
| 112 |
next_action="Run or refresh the hosted Omni eval and update docs/hosted_omni_eval_results.md.",
|
tests/test_evidence_gate_status.py
CHANGED
|
@@ -10,6 +10,7 @@ def test_current_repo_report_keeps_remaining_external_gates_incomplete() -> None
|
|
| 10 |
assert report["status"] == "incomplete"
|
| 11 |
assert report["ready_for_badge_claims"] is False
|
| 12 |
assert report["gates"]["claim_audit"]["passed"] is True
|
|
|
|
| 13 |
assert report["gates"]["local_4b_50_case_eval"]["passed"] is False
|
| 14 |
assert report["gates"]["no_cloud_route"]["passed"] is False
|
| 15 |
assert report["gates"]["llama_champion_route"]["passed"] is False
|
|
|
|
| 10 |
assert report["status"] == "incomplete"
|
| 11 |
assert report["ready_for_badge_claims"] is False
|
| 12 |
assert report["gates"]["claim_audit"]["passed"] is True
|
| 13 |
+
assert report["gates"]["hosted_omni_eval"]["passed"] is True
|
| 14 |
assert report["gates"]["local_4b_50_case_eval"]["passed"] is False
|
| 15 |
assert report["gates"]["no_cloud_route"]["passed"] is False
|
| 16 |
assert report["gates"]["llama_champion_route"]["passed"] is False
|
tests/test_submission_claim_audit.py
CHANGED
|
@@ -11,6 +11,7 @@ def test_current_submission_claims_stay_evidence_gated() -> None:
|
|
| 11 |
assert report["gate_status"]["off_grid"] is False
|
| 12 |
assert report["gate_status"]["local_4b"] is False
|
| 13 |
assert report["gate_status"]["local_asr"] is False
|
|
|
|
| 14 |
assert report["gate_status"]["backyard_user_use"] is False
|
| 15 |
|
| 16 |
|
|
@@ -51,3 +52,35 @@ def test_audit_flags_unproven_user_use_and_local_asr_claims() -> None:
|
|
| 51 |
|
| 52 |
assert [violation["gate"] for violation in violations] == ["backyard_user_use", "local_asr"]
|
| 53 |
assert {violation["file"] for violation in violations} == {"submission.md"}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
assert report["gate_status"]["off_grid"] is False
|
| 12 |
assert report["gate_status"]["local_4b"] is False
|
| 13 |
assert report["gate_status"]["local_asr"] is False
|
| 14 |
+
assert report["gate_status"]["well_tuned"] is True
|
| 15 |
assert report["gate_status"]["backyard_user_use"] is False
|
| 16 |
|
| 17 |
|
|
|
|
| 52 |
|
| 53 |
assert [violation["gate"] for violation in violations] == ["backyard_user_use", "local_asr"]
|
| 54 |
assert {violation["file"] for violation in violations} == {"submission.md"}
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def test_well_tuned_gate_requires_archive_result_and_public_space_caveat(tmp_path: Path) -> None:
|
| 58 |
+
docs_dir = tmp_path / "docs"
|
| 59 |
+
docs_dir.mkdir()
|
| 60 |
+
ledger = docs_dir / "model_parameter_evidence_ledger.md"
|
| 61 |
+
ledger.write_text(
|
| 62 |
+
"\n".join(
|
| 63 |
+
[
|
| 64 |
+
"Archive: build-small-hackathon/figment-finetuned-model-archive",
|
| 65 |
+
"Version: figment_sft_v14p",
|
| 66 |
+
"Result: v14p repair-union with 150/150 competence",
|
| 67 |
+
"Boundary: Do not imply the public no-secret Space is serving this tuned model.",
|
| 68 |
+
]
|
| 69 |
+
),
|
| 70 |
+
encoding="utf-8",
|
| 71 |
+
)
|
| 72 |
+
|
| 73 |
+
assert audit_submission_claims._has_well_tuned_evidence(tmp_path) is True
|
| 74 |
+
|
| 75 |
+
ledger.write_text(
|
| 76 |
+
"\n".join(
|
| 77 |
+
[
|
| 78 |
+
"Archive: build-small-hackathon/figment-finetuned-model-archive",
|
| 79 |
+
"Version: figment_sft_v14p",
|
| 80 |
+
"Result: v14p repair-union with 150/150 competence",
|
| 81 |
+
]
|
| 82 |
+
),
|
| 83 |
+
encoding="utf-8",
|
| 84 |
+
)
|
| 85 |
+
|
| 86 |
+
assert audit_submission_claims._has_well_tuned_evidence(tmp_path) is False
|