A developer pruned 85% of Gemma 4 26B-A4B and kept 92% agreement

A solo developer chopped most of the experts out of Gemma 4 26B-A4B, healed what was left overnight on a laptop, and ended up with a small dense model that still does his one job almost as well as the original.

The post hit r/LocalLLaMA with the title "I removed 85% of gemma's weights (and it is mostly fine)" and got 451 points and 146 comments. Behind it is a long blog post from the developer of Tolby, an app that watches your screen and tells you whether you're on task or off task. He's upfront that it's "an engineering side project, not research."

What's actually new

The problem was pretty ordinary. Tolby runs a local model 24/7 as a classifier. The 26B MoE worked well but it was too big for a lot of machines, and keeping it loaded all day pushed his computer into swap enough that he worried about his SSD. In his words, a 7 GB model just doesn't work on a system with 8 GB of RAM. The smaller Gemma 4 E2B did significantly worse on his task, and fine tuning it would have needed more data and compute than he had.

So he went the other direction. Gemma 4 26B-A4B has 30 layers with 128 experts each, and the router picks 8 per token. He picked a fixed set of 8 experts per layer and threw the rest away. Since the model already uses 8 experts per token and now there are only 8, the router has nothing left to choose, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M. A commenter asked how big that is in practice, and he said it went from 14.5 GB to 3.8 GB of memory. On his Mac it runs at 79 tokens/s, roughly double the speed of the 12B, at roughly half the memory.

The obvious way to choose experts failed. Keeping the ones that fire most often for the task "just dies," he said in the thread. What worked was removing an expert and measuring how much the hidden state moves away from the original model, then keeping the ones that matter most.

Cutting is the easy part though. The healing is the interesting bit. He calls it block-local training. The layers group into 5 blocks, and for each block he feeds in the original model's clean input and trains the pruned block to match the original block's output. "I'm not training on the answer. I'm training on the thought process," he wrote. He only needed prompts from Tolby, no labels.

The numbers from his blog:

• Before healing, hidden-state error was 0.6858. The best block-local run got it to 0.2638.
• At the same rank and steps, block-local scored 0.2995 against 0.3312 for training the whole model end to end.
• On 70 test prompts it gave the same top answer as the full model on all 70, up from 60 without healing.
• Final agreement on his classification task was 91.7%. For comparison, the 12B agrees with the 26B 90% of the time on the same data. His target was 98%, so by his own bar it missed.

He also flags these are single runs on 20 test prompts, so differences under about 0.01 are noise. And the blog has a whole section called "The clue was wrong," where he admits his theory about why the global attention layers helped came from a misleading measurement. Kinda refreshing tbh.

Why it matters if you build agents

A lot of agents have a step like Tolby's: a cheap, always-on check that runs way more often than the main model. Say your agent triages every incoming support message into "needs a human" or "auto-reply" before anything else happens. You could send that to a frontier API every time, run a full mid-size model locally, or train a classic classifier if you have labeled data.

This post adds a fourth option. Take a big MoE model that already does the task well with your prompts, keep only the experts your task needs, and train the leftover blocks to think like the original. You need prompts not labels, and he trained it overnight on a laptop. He says the method is light enough to run on a user's own device in about 5 GB of memory, so the model could adapt to that user's data. He hasn't tested that on a real user's device yet.

The cost is real and he says it plainly. Knowledge is gone. In the thread he shared that on MMLU biology it went from 97% to 68%, and on unrelated MMLU tasks it matched its original answer only 46% of the time. This is a one-task model and nothing more.

What people are saying

r/LocalLLaMA had fun with it ("Neural network lobotomy" was the top reply) but the useful pushback was about what you can't see. One commenter described an abliterated Qwen model that looked the same as the base on coding benchmarks and then turned out to be very bad at Flutter, the kind of thing you don't notice until you hit it. Another pointed out that a model agreeing 92% of the time might be just as confident on the 8% it gets wrong, and suggested looking at the logprobs on those misses. The author himself said he's not sure how it handles prompt injection, since none was in his training prompts.

Several people asked why not just use a small classifier. Suggestions included XGBoost and CatBoost, or using Gemma to generate training data for a smaller model. He said he doesn't have the data for that, but he already uses a small tree model to decide when to run the LLM so the app saves battery, and he'll look harder at that.

Others pointed to existing work. Someone shared a pruned 26B coder variant on Hugging Face that they use as their daily driver, and another commenter noted LiteRT-LM can lazily load E4B's per-layer embeddings, which the author admitted he'd missed when he started.

Open questions

The task-level results come from one developer's own data and prompts, with small test sets. Nobody has reproduced them yet. The on-device training idea is untested, and so is robustness to inputs his prompts didn't cover. If you want to poke at it, he says there's a notebook at the top of the blog post. Just test the exact edge cases your agent will hit, because the damage won't show up anywhere else.

Sources

• Frankensteining Gemma's experts, Tolby blog, https://tolby.app/blog/frankensteining-gemmas-experts
• I removed 85% of gemma's weights (and it is mostly fine), r/LocalLLaMA, https://www.reddit.com/r/LocalLLaMA/comments/1xnoqyy/i_removed_85_of_gemmas_weights_and_it_is_mostly/