Research, in progress
Training mixture-of-experts language models on one 6 GB GPU
Testing what everyone else assumes.
A from-scratch research environment for small mixture-of-experts language models. Quiet for now — open source later.
Problem
Everyone building with AI uses the same handful of models and inherits their assumptions. The questions worth asking — how experts should route, whether backprop is the only learning rule worth using — sit behind compute nobody can afford.
What I did
Built a lab instead of a wrapper. One spike per idea, each proving itself before the next builds on it: custom routing, alternative learning rules, a dashboard measuring perplexity and routing behaviour on every run — trained locally, from scratch. I direct the research; my team of AI agents does much of the building.
What happened
Four spikes through to working models. Nothing released yet. It stays quiet until it is worth using rather than worth admiring, and then it goes open source.