In this article: what the build is · why twelve cards · what had to be built · how fast it is · several requests at once · documents it has already read · the hardware · what did not work · what is next · where the engineering is
What the build is
The cards are Nvidia Tesla V100s, sold to data centers from 2017 and retired from them years ago. Each has 32 GB of very fast memory and they come four to a board, joined inside the board by a fast link. The model is DeepSeek V4.1 Flash, released in October 2026. It is a mixture of experts: for every word it writes, the model consults a handful of small specialist networks, the experts, chosen from a pool of 15,360. Only a few run for any one word, but all of them have to be somewhere the cards can reach quickly.
Nvidia publishes a version of this model whose experts are stored in a compact four-bit form it calls NVFP4. The expert values in that pack equal the ones in DeepSeek’s own release. Our aim with this build was simple to state: serve every expert from that pack, from card memory, with nothing compressed further and nothing read from the slower memory of the host computer. On eight cards that is not possible. On twelve it is.
The model runs as three sections, one per board, each section split across the four cards of its board. A request passes through the first board, then the second, then the third, like stations on a line. The build keeps four conversations ready at once, each with up to 524,288 tokens of context, and it is tuned for the way we use it: one long piece of agent work at a time, on prompts of sixty thousand to two hundred thousand tokens that grow by small turns. It has been our default DeepSeek server since November 17, 2026.
Speed is counted in tokens. A token is roughly three quarters of an English word. A “128K” prompt is about 128,000 tokens, roughly 96,000 words.
Why twelve cards and not eight
The arithmetic decides it. One expert in NVFP4 takes 4,874,240 bytes on each of the four cards that share it. The forty layers of the model hold 278.9 GiB of expert records. Eight cards of 32 GB hold 254 GiB. So an eight-card machine has to compress some experts further or read some of them from host memory, and either choice costs something. Twelve cards hold every expert with room to spare for the conversations.
Where to cut the model into three sections is also decided by arithmetic. Twenty layers of experts are 34.86 GiB per card, more than a card holds, so the first cut cannot fall at the middle. A section has to begin at a layer of a particular kind, and the last section also has to carry the model’s drafting network and its output stage. That leaves one legal cut that fits: layers 0 to 13, 14 to 27 and 28 to 39. It is the layout a pull request of ours to the shared engine had already run on twelve cards, at about 3 tokens a second on an unoptimized engine. Our starting point was that someone else had shown it could load.
What had to be built
The engine we started from accepted exactly one arrangement of this model: two sections, cut at layer 20, and nothing else. The work of this build was a set of changes, published as install kits, that teach it a third section, fix its scheduler for a line of three, cut the middle section’s memory to fit, and add a memory limit on every card so that a boot that would not fit fails at once instead of failing under load. After that came three days of test windows, one change per boot, each with its pass-or-fail rule written before the boot ran. Eleven of those changes are switched on in what serves today. The rest are recorded with their numbers, including the ones that failed.
How fast it is
All of the figures below were measured on November 19, 2026, on the configuration that serves today, with every card limited to 175 watts and a clock ceiling of 1,452 MHz, four conversation slots open, the model’s reasoning mode off, 1,024 tokens written per request, each prompt read cold (nothing of it in the cache), and each cell the mean of two runs. The prompts are real files of three kinds: a source-code file, a legal brief, and literary prose.
Three columns matter most. Decode is how fast the model writes once it has read the prompt. Prompt read is how fast it takes in the prompt before the first word. First token is the wait before the first word appears, which for a long document is almost entirely the read.
Code (a source file as the prompt)
| Context (prompt tokens) | Decode, one request (tok/s) | Prompt read (tok/s) | First token, cold | Three requests together (tok/s, summed) | First token, document already cached |
|---|---|---|---|---|---|
| 1K | 184 | 1,567 | 0.7 s | 443 | 0.2 s* |
| 8K | 176 | 3,267 | 2.5 s | 421 | 0.2 s |
| 32K | 176 | 4,478 | 7.3 s | 404 | 0.3 s |
| 64K | 157 | 4,697 | 13.9 s | 364 | 0.3 s |
| 128K | 157 | 4,670 | 28.1 s | 360 | 0.4 s |
| 256K | 153 | 4,518 | 58.0 s | 344 | 0.6 s |
Legal (a brief as the prompt)
| Context (prompt tokens) | Decode, one request (tok/s) | Prompt read (tok/s) | First token, cold | Three requests together (tok/s, summed) | First token, document already cached |
|---|---|---|---|---|---|
| 1K | 102 | 1,526 | 0.7 s | 249 | 0.2 s* |
| 8K | 93 | 3,253 | 2.5 s | 219 | 0.2 s |
| 32K | 92 | 4,354 | 7.5 s | 213 | 0.3 s |
| 64K | 93 | 4,561 | 14.4 s | 215 | 0.3 s |
| 128K | 93 | 4,538 | 28.9 s | 213 | 0.4 s |
| 256K | 92 | 4,422 | 59.3 s | 207 | 0.7 s |
* At 1K the request’s fixed cost is most of the read, so the cache cannot show. Literary prose, the third kind, writes at 86 tokens a second at 1K, 86 at 128K and 85 at 256K, and reads at 4,586 tokens a second at 128K and 4,456 at 256K. Every cell of all three kinds, with its minimum and maximum, is in the build’s measurements file, linked below.
Three things to read from the tables.
- Writing speed barely falls with context. One code request writes at 184 tokens a second on a short prompt and 153 at 256,000 tokens; a legal request at 102 and 92. Code is faster than legal or literary text because it is more predictable, and the model’s drafting network guesses more of it right.
- Reading is about 4,500 tokens a second at any length past 32K. A 128,000-token file is read in 28 seconds, a 256,000-token file in a minute. Short prompts read more slowly per token because each request carries a fixed cost.
- A writing figure is the figure of the text that was written. We ran this same bench twice on November 19 after a change that alters rounding, and the model wrote different text in every cell; the writing speed of a cell moved with its text by up to about 10 percent either way, while two repeats of one text agree within 2 percent. So for one code request at 128K the honest statement is a band, 157 to 175 tokens a second across the two benches of the day, not a single number.
Several requests at once
The build keeps four conversations ready and works on three at once. Because the three sections of the model form a line, each step of the line carries one request, so the three requests take turns rather than batching. Two consequences follow, and both are measured.
First, long prompts are read one after another. Three cold 128K code requests released together got their first words after 77, 90 and 90 seconds, where one alone takes 28. Second, once every prompt is in, the three write together: 360 tokens a second summed on code at 128K, 213 on legal, against 157 and 93 for one request. That is about 2.2 to 2.5 times one stream, not three, and it is the throughput of this machine, not its latency. The “three requests together” column in the tables is that sum. Counted from the first request’s first word to the last request’s end, including the waiting, the three give 138 tokens a second on code and 111 on legal. This is not a batched-decode engine; more requests than sections gain little.
Documents it has already read
The largest gain for daily use is not in either table’s writing column. A request on a document the machine has already read gets its first word in under a second at every length: 0.4 seconds at 128K and 0.6 to 0.7 seconds at 256K, against 28 and 58 seconds cold. In a working session the prompt grows by a question and an answer each turn, and nearly all of it has been read before, so the wait before each answer is the new part only.
The same holds when several people work on one document: the first time a second or third stream shares a file the machine reads it once more, and after that every stream hits. Three code streams on one cached 128K document wrote 427 tokens a second over their window, within 2 percent of the sum of the three alone.
The hardware
Three boards of four Tesla V100 SXM2 cards, 32 GB each. Inside a board the four cards share Nvidia’s fast link and split each section of the model between them; between boards the hand-off from one section to the next goes over the host’s PCIe lanes, through two different PCIe switches, so the second hand-off crosses host bridges and costs more than the first. Each card is limited to 175 watts and a clock ceiling of 1,452 MHz. We ran at 200 watts for a day and the host tripped once under four long chats at once, so 175 is where it stays; long prompts read about a third slower at 150 watts than at 200, and the figures above are 175-watt figures.
These cards have no hardware for four-bit or eight-bit arithmetic. Every compact number is unpacked in software into the half-precision form the cards can multiply. The compact experts pay off because the cards read fewer bytes, not because they compute faster, and the reading is what a model waits on when it writes.
What did not work
Every window of this build is recorded with its rule, its result and, where the prediction was wrong, the prediction. A few that matter:
- The first three-section boot answered wrongly, and the diagnostic window after it ended with four cards of the first section faulting. A spacing rule fixed it.
- Our own checker was wrong. The check that confirms two answers are identical compared them only as far as the shorter one, so an answer cut off halfway passed. An independent review found it; the checks were made strict and the earlier results re-run on whole answers. They held.
- Four optimizations passed every correctness check and failed their speed line: a store that saves conversations to disk so a restart loses nothing (its first word on large turns came a second later than without it), a leaner hand-off between sections, a tighter packing of the expert records (less memory, but slower rounds, the opposite of the prediction), and an exact arithmetic for the four-card sums (2.2 times the cost of the library’s own). Each stays in the record as a switch that is off.
- Several streams do not read their prompts together. Released at once, their reads interleave, and a read that shares the cards with a writing stream runs at about half its lone rate. The cause is not established.
- Predictions were wrong on both sides. One change gave twice its predicted gain; another was predicted to fail on speed and failed on closeness first. Each prediction stays recorded as written.
- What these measurements do not establish: answer quality in use. How closely a build follows a reference version is a measurement about the model, not a promise about any particular answer.
What is next
As this is published, two things are being switched on in the serving configuration: the conversation store, re-cut so that a matter’s documents can be pinned in memory and survive a restart, and a count of cached tokens in every reply, so that a hit can be told from a re-read without a stopwatch. The next study is batched decoding on the line, whose first question, written down before any data, is whether rows written together round the way single ones do. After that, the measurements file and this article will be updated again.
Where the engineering is
- Volta 12X AmPLIfy on GitHub: the build itself: the install kits, the full measurements with their dates and conditions, the architecture, the history of every test window, and the list of what is not yet published.
- The Volta research hub: every build, every improvement that worked or failed, the testing method, and the source research.
- PLI Volta Builder: the same method packaged so that an AI coding assistant can repeat it on other cards.
- A current AI model on used 2018 graphics cards: our earlier article, written for firm owners, on what the machines cost, what they cannot do, and the lesson about checking.
- Local models: what this looks like as a service for a firm.
- Back to the research index