Projects / Built here
A model on your own machine
Alive Running here now. You can look at it, and there is something to join.
A capable model now runs on a home computer. The limit is video memory. Quantisation shrinks a model until it fits, at a cost in quality that can be measured.
A GPU in the rack here does this, so the area is written from a running machine.
What decides it
Three things decide it: how much video memory you have, what quantisation you can accept, and how long you are willing to wait for the first token.
What this area sets out to record is which tasks survive heavy quantisation and which fail.
The job a local model does best
A local model is rarely the strongest one available. Its value is that it is always available.
The agent described in the agents area uses a local model on the GPU in this house as the last link in a chain: a paid model first, then five free ones, then this. It has no tools and answers in plain text, which makes it the weakest link. It also has no rate limit and no bill, so the agent still answers when every cloud quota is spent.
This use needs far less hardware than matching a frontier model.
What is not known here
The open items for this project. Work on it starts here.
- Specific model and quantisation recommendations are deliberately absent until they have been measured on the machine here rather than repeated from elsewhere.
- The cheap end of the hardware question, what the least money is that gets a usable local model, is not costed yet.
- Which tasks survive heavy quantisation has not been measured here. That measurement is the most useful result this area could produce, and the most work.