← All work

Research, in progress

Training mixture-of-experts language models on one 6 GB GPU

Testing what everyone else assumes.

A from-scratch research environment for small mixture-of-experts language models. Quiet for now — open source later.

Problem

Everyone building with AI uses the same handful of models and inherits their assumptions. The questions worth asking — how experts should route, whether backprop is the only learning rule worth using — sit behind compute nobody can afford.

What I did

Built a lab instead of a wrapper. One spike per idea, each proving itself before the next builds on it: custom routing, alternative learning rules, a dashboard measuring perplexity and routing behaviour on every run — trained locally, from scratch. I direct the research; my team of AI agents does much of the building.

What happened

Four spikes through to working models. Nothing released yet. It stays quiet until it is worth using rather than worth admiring, and then it goes open source.

4research spikes
6 GBconsumer VRAM
0hosted APIs
AIagent research team