Add files using upload-large-folder tool

README.md

@@ -0,0 +1,53 @@
+
+---
+license: cc-by-4.0
+library_name: saelens
+---
+
+# 1. Gemma Scope 2
+
+Gemma Scope 2 is a comprehensive, open suite of sparse autoencoders and transcoders for a range of model 4bs and versions in the Gemma 3 model family. We have SAEs on three different sites (as well as transcoders) for every layer of the pretrained and instruction-tuned models of parameter 4bs 270M, 1B, 4B, 12B and 27B. We also include several multi-layer SAE variants: partial residual stream crosscoders for every base Gemma 3 model, and cross-layer transcoders for the 270M and 1B models.
+
+Sparse Autoencoders are a "microscope" of sorts that can help us break down a model's internal activations into the underlying concepts, just as biologists use microscopes to study the individual cells of plants and animals.
+
+See our landing page for details on the whole suite.
+
+# 2. What Is In This Repo?
+
+This repo contains a specific set of SAEs and transcoders: the ones trained on Gemma V3 {4b_upper} {pt_upper}. Every folder here contains a different suite of models. Each of the folders in this page are named for the type of model that was trained:
+
+- Single-layer models
+  - `resid_post`, `attn_out` and `mlp_out` contain SAEs at 4 different layers (25%, 50%, 65% and 85% depth) and a variety of widths & L0 values, trained on the model's residual stream, attention output, and MLP output respectively.
+  - `transcoder` contain a range of transcoders (or skip-transcoders) trained on the same 4 layers, with a variety of widths & L0 values.
+  - `resid_post_all`, `attn_out_all`, `mlp_out_all` and `transcoder_all` contain a smaller range of widths & L0 values, but for every single layer in the model.
+- Multi-layer models
+  - `crosscoder` contains a set of weakly causal crosscoders which were trained on 4 concatenated layers of the residual stream (the same as those we trained our subsets on)
+  - `clt` contains a set of cross-layer transcoders, which were trained to reconstruct the whole model's MLP outputs from the residual stream values just before each MLP layer.
+
+So for example, `google/gemma-scope-2-{4b}-{pt}/resid_post` contains a range of SAEs trained on the residual stream of `gemma-v3-270m-pt` at 4 different layers.
+
+# 3. How can I use these SAEs straight away?
+
+```py
+from sae_lens import SAE  # pip install sae-lens
+
+sae, cfg_dict, sparsity = SAE.from_pretrained(
+    release = "gemma-scope-2-{4b}-{pt}-resid_post",
+    sae_id = "layer_12_width_16k_l0_small",
+)
+```
+
+# 4. Which SAE should I use?
+
+Unless you're doing full circuit-style analysis, we recommend using SAEs / transcoders from the layer subset folders, e.g. `resid_post` or `transcoder`. Assuming you're using residual stream SAEs from `resid_post`, then:
+
+- **Width**: our SAEs have widths 16k, 64k, 256k, 1m. You can visit Neuronpedia to get a qualitative sense of what kinds of features you can find at different widths, but we generally recommend using 64k or 256k.
+- **L0**: our SAEs have target L0 values "small" (10-20), "medium" (30-60) or "large" (60-150)". You can also look at the `config.json` file saved with every SAE's parameters to check exactly what the L0 is (or just visit the Neuronpedia page!). We generally recommend using "medium" which is useful for most tasks, although this might vary depending on your exact use case. Again you can visit Neuronpedia to get a sense of what kind of features each model type finds.
+
+# 4. Point of Contact
+
+Point of contact: Callum McDougall
+Contact by email: [email protected]
+
+# 5. Citation
+Paper: (link to go here)

attn_out_all/layer_0_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.0.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.0.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 60,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_0_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.0.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.0.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 10,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_10_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.10.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.10.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 112,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_10_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.10.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.10.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 18,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_11_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.11.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.11.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 118,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_12_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.12.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.12.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_12_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.12.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.12.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_12_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.12.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.12.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_12_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.12.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.12.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_13_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.13.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.13.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_13_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.13.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.13.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_14_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.14.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.14.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_16_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.16.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.16.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_16_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.16.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.16.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_17_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.17.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.17.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_18_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.18.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.18.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_1_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.1.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.1.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 10,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_20_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.20.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.20.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_20_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.20.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.20.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_21_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.21.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.21.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_22_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.22.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.22.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_22_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.22.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.22.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_23_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.23.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.23.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_23_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.23.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.23.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_24_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.24.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.24.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_24_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.24.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.24.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_26_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.26.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.26.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_26_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.26.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.26.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_27_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.27.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.27.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_27_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.27.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.27.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_28_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.28.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.28.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_28_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.28.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.28.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_29_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.29.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.29.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_29_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.29.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.29.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_30_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.30.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.30.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_30_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.30.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.30.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_31_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.31.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.31.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_31_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.31.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.31.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 120,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_32_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.32.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.32.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_32_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.32.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.32.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_33_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.33.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.33.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 20,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_3_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.3.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.3.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 12,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_4_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.4.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.4.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 81,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_5_width_16k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.5.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.5.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 86,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_5_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.5.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.5.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 86,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_6_width_262k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.6.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.6.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 15,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_7_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.7.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.7.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 97,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_8_width_16k_l0_small/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.8.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.8.self_attn.o_proj.input",
+  "width": 16384,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 17,
+  "affine_connection": false,
+  "type": "sae"
+}

attn_out_all/layer_8_width_262k_l0_big/config.json

@@ -0,0 +1,10 @@
+{
+  "hf_hook_point_in": "model.layers.8.self_attn.o_proj.input",
+  "hf_hook_point_out": "model.layers.8.self_attn.o_proj.input",
+  "width": 262144,
+  "model_name": "google/gemma-3-4b-pt",
+  "architecture": "jump_relu",
+  "l0": 102,
+  "affine_connection": false,
+  "type": "sae"
+}