Bonsai 27 B on old GPU’s

I mostly use Qwen3:8b for my analysis because it uses about 8 GB of VRAM and fits within the 12 GB on my 3080 Ti GPU, along with the KV Store, runtime, and context. I want to use larger models that I can code with at decent speeds. I added an old 1080 Ti with 11 GB of VRAM and sharded a larger model across both cards, but the older card’s memory bandwidth and the PCIe bandwidth between the two cards were too low to achieve an acceptable tokens-per-second rate.

I’ve been watching a few videos by Codacus, who seems to have a similar setup to mine, but with a GTX 10 series and RTX 3060 GPU in his AI server, plus a MacBook on his network. I just don’t have the same system RAM as he does to exploit some of his techniques. He’s been doing some interesting optimizations.

I saw that he tested a 3.5 GB model (Bonsai:27B) comparable to the 54 GB Qwen3 FP16 with 27 B parameters. My 3080 only has 12 GB, and when I shard with the 1080 with 11 GB, it’s too tight of a fit the quantized 18 GB version to also handle context and runtime, and the speed goes down drastically, so I’ve been staying away from the larger models for now. But with a comparable 3.5 GB model based on the 21 GB model, I now have new hope of using something more powerful than the 8 B parameter model.

Bonsai is not available to install with Ollama. I found that I had to jump through a few hoops. I downloaded the model from Hugging Face, but found that llama.cpp only supported 2- to 8-bit models. I had to download a custom version of llama .cpp to support 1-bit models. The worst part was the build – or multiple builds where I discovered that I also needed to install the NVIDIA CUDA Toolkit and libssl, restarting the build each time. Somewhere along the way I ended up with a few services terminating including ollama, hermes dashboard, and a agent orchestration server I was working on. The last build took so long that I headed to sleep.

Install Bonsai & Custom Llama.cpp on x86 server

sudo apt update
sudo apt install -y \
  build-essential \
  cmake \
  git \
  python3-pip \
  nvidia-cuda-toolkit \
  libssl-dev
git clone
  https://github.com/PrismML-Eng/llama.cpp.git \
  ~/src/bonsai-llama
cd ~/src/bonsai-llama
cmake -B build \
  -DGGML_CUDA=ON \
  -DGGML_CCACHE=OFF \
  -DGGML_CUDA_NCCL=OFF \
  -DCMAKE_CUDA_ARCHITECTURES=86,61 \
  -DLLAMA_SERVER_WEBUI=OFF
cmake --build build \
  --target llama-cli llama-server \
  --parallel 1
python3 -m venv .hf-venv
./.hf-venv/bin/pip install -U huggingface_hub
mkdir -p ~/models/bonsai-27b
 
./.hf-venv/bin/hf download \
  prism-ml/Bonsai-27B-gguf \
  Bonsai-27B-Q1_0.gguf \
  --local-dir ~/models/bonsai-27b
./build/bin/llama-cli \
  -m ~/models/bonsai-27b/Bonsai-27B-Q1_0.gguf \
  -ngl 99 -c 8192 \
  -p "Explain why a 27-billion-parameter model
    can fit in 4 GB with one-bit weights." \
  -n 250
./build/bin/llama-server \
  -m ~/models/bonsai-27b/Bonsai-27B-Q1_0.gguf \
  --device CUDA0 \
  -ngl 99 -c 8192 \
  --reasoning-budget 0 \
  --host 0.0.0.0 \
  --port 8080

I woke up and tried it out. I found it was thinking so much that it never got around to answering because of the limit I imposed on its response. I kept trying to increase the limit, and just got more thinking out of it. I eventually setup the web server and talked to it. I got an answer after 3200 tokens at 72 tokens/s.

Prompt Test

Explain why a 27-billion-parameter model can fit in 4 GB with one-bit weights.

353 tokens 0.5s 670.48 tokens/s

Reasoning

Here’s a thinking process that leads to the explanation of why a 27-billion-parameter model can fit in 4 GB with one-bit weights:

Analyze the Request:
- Subject: A 27-billion-parameter model.
- Condition: Using “one-bit weights” (Quantization).
- Constraint: Fitting into 4 GB of memory.
- Goal: Explain the math and logic behind this specific scenario.
Initial Assessment & Math Check:
- Total parameters: 27 billion ( $27 \times 10^{9}$ 27×109).
- Size per parameter: 1 bit.
- Total size in bits: $27 \times 10^{9}$ 27×109 bits.
- Conversion to Bytes (8 bits per byte): $(27 \times 10^{9}) / 8 \approx 3.375 \times 10^{9}$ (27×109)/8≈3.375×109 bytes.
- Conversion to Gigabytes (1 GB = $10^{9}$ 109 bytes or $2^{30}$ 230 bytes? Usually, in storage/compression contexts, $10^{9}$ 109 or 1 GiB ( $1073741824$ 1073741824) is used. Let’s calculate both to see if it fits 4 GB).
- Decimal GB: $\approx 3.375$ ≈3.375 GB.
- Binary GiB: $3.375$ 3.375 GB / $1.074$ 1.074 $\approx 3.14$ ≈3.14 GiB.
- Result: It fits comfortably within 4 GB.
Contextualizing “One-Bit Weights”:
- What is the default? Usually 16-bit floating point (FP16/BF16) or 32-bit (FP32).
- Size of standard weight:
  - FP16: 2 bytes (16 bits).
  - FP32: 4 bytes (32 bits).
- Size of 1-bit weight: 1 bit (0.125 bytes).
- Compression ratio: $2$ 2 bytes / 0.125 bytes = 16x. (Or 4x if comparing to FP32).
Addressing the “Missing” Data:
- A neural network isn’t just the weights.
- There are Activations (hidden states) and Gradients (if training).
- Crucial distinction: The prompt implies inference (running the model). If it were training, 27B parameters in 4GB is impossible regardless of weight precision because activations and gradients would bloat the memory requirement significantly (often 16x-20x model size). I need to clarify this distinction.
Structuring the Explanation:
- Introduction: Acknowledge the claim (it’s mathematically sound for storage/inference) but add the necessary caveat (it’s not training).
- The Math (The “How”):
  - Step 1: Calculate raw bits.
  - Step 2: Convert to bytes.
  - Step 3: Compare to 4 GB.
- The Mechanism (The “Why”):
  - Explain Bitwise Quantization (INT8 vs. 1-bit).
  - Explain the compression ratio relative to standard FP16/FP32.
- The Caveats (The “But”):
  - Overhead: Model metadata, optimizer states (if applicable), and most importantly, Activations.
  - Speed: 1-bit is often slower or requires specific hardware (sparse tensor cores).
  - Accuracy: Loss of precision (though modern “ZeroQuant” or “AWQ” methods mitigate this).
  - Training vs. Inference: Reiterate that this only applies to inference.
Drafting the Content (Iterative refinement):
- Math check:
  - $27, 000, 000, 000 params$ 27,000,000,000 params
  - $1 bit/param = 0.125 bytes/param$ 1 bit/param=0.125 bytes/param
  - $27, 000, 000, 000 \times 0.125 = 3, 375, 000, 000 bytes$ 27,000,000,000×0.125=3,375,000,000 bytes
  - $3.375 GB$ 3.375 GB (decimal).
  - $3.375 GB \approx 3.14 GiB$ 3.375 GB≈3.14 GiB.
  - $3.14 < 4$ 3.14<4.
  - It fits.
- Comparison:
  - FP16 (standard for AI): 2 bytes.
  - FP16 size: $27 B \times 2 = 54$ 27B×2=54 GB.
  - Reduction factor: 54 GB / 3.375 GB = 16x.
- Technical Nuance (One-Bit):
  - True 1-bit often requires a sign bit (sign-magnitude) or is handled via binary matrices (e.g., binary neural networks).
  - However, often “1-bit” in modern quantization (like GPTQ or AWQ) implies a scaling factor per channel/activation group, which takes up some space. Self-correction: Keep it simple first. The prompt asks “Why it can fit,” so the math is the primary answer. Mentioning the overhead is a good “pro-tip” section.
- The “Memory Wall”:
  - Mention that while weights fit, the data passing through (activations) takes up space.
  - Example: If context window is 4096, activations might equal or exceed the model size depending on architecture.
Final Polish:
- Use clear headings.
- Use bold text for key numbers.
- Ensure the tone is educational.
Output Generation: (This matches the provided good response.)
- Check against the tool definitions: The user asked an explanation question. I don’t strictly need the get_datetime or get_info tools here, as this is general math and computer science knowledge. I will answer directly.
Refining the “Training vs Inference” point:
- If I train this, I need gradients. If weights are 1 bit, gradients are usually stored in 16/32 bits to maintain precision. So during training, it would need ~4GB for weights + 4GB+ for gradients + optimizer states (Adam needs more). It would definitely not fit in 4GB. I must emphasize Inference.
Final Review of the Explanation:
- 27B params.
- 1 bit = 1/8 byte.
- 27B / 8 = 3.375 GB.
- 3.375 GB < 4 GB.
- Q.E.D.
(Self-correction: Is it exactly 1 bit? Sometimes binary weights need a scale factor. If there is a scale factor stored in 16-bit for every 1-bit weight, that’s 2 bytes per 1-bit weight -> 27 * 2 = 54GB. That doesn’t fit. However, usually, scale factors are shared per channel or layer, or computed on the fly. I will assume the prompt refers to the raw weight memory footprint without the overhead of per-parameter scale factors, or that the scale factors are negligible compared to the 4GB constraint.)Let’s stick to the raw math first, then mention overhead as a caveat.(Double check: 27 billion is 2.7×10102.7×1010.
2.7×1010/8=3.375×1092.7×1010/8=3.375×109 bytes = 3.375 GB.
It is very close to 4GB. It fits.)Wait, 27 billion is often cited for models like Llama-2 7B, 13B, 30B… Llama-2 30B is the closest 27B analog. Llama-2 7B is 14GB in FP16. 7B * 2 = 14GB.
27B * 2 = 54GB.
54GB / 16 = 3.375GB.
Yes, the math holds up.Final Plan:
1. The Math (The core answer).
2. Comparison to standard formats.
3. The “Catch” (Inference vs. Training & Overhead).
4. Technical implementation (Quantization).
No tools needed.

This is a feat of quantization—the process of reducing the numerical precision of floating-point numbers to improve efficiency.

Here is the step-by-step mathematical breakdown of why a 27-billion-parameter model can fit into 4 GB of memory, along with the technical caveats involved.

1. The Raw Math

To understand the fit, we must calculate the storage size of the weights (the model’s brain).

The Standard (Baseline):
Modern AI models are typically trained in 16-bit floating point (FP16) or 32-bit floating point (FP32).

FP16: Each weight takes up 2 bytes.
FP32: Each weight takes up 4 bytes.

The One-Bit Constraint:
The prompt specifies 1-bit weights.

1 bit: Each weight takes up $1/8$ of a byte (0.125 bytes).

The Calculation:

Total Parameters: 27,000,000,000 (27 billion)
Size per Parameter: 0.125 bytes (1 bit)
Total Memory (Bytes): $27,000,000,000 \times 0.125 = 3,375,000,000$ bytes.
Convert to Gigabytes:
- Decimal GB (standard storage): $\approx 3.375$ GB.
- Binary GiB (RAM standard, $1024^3$): $\approx 3.14$ GiB.

Result: $3.375\text{ GB} < 4\text{ GB}$.
The raw weights fit comfortably into 4 GB.

2. The Compression Ratio

If this model were stored in standard FP16 format (the industry standard for inference), it would require:
$$27\text{ Billion} \times 2\text{ bytes} = 54\text{ GB}$$

By using 1-bit quantization, you are compressing the model by a factor of 16x.

3. Critical Caveats: “Inference” vs. “Training”

While the weights fit in 4 GB, this explanation only applies to inference (running the model on text/data).

1. The “Hidden” Memory Usage (Activations)
A model is not just a static list of numbers (weights). When you run the model, it creates temporary data called activations (representing the state of the neurons as it processes a prompt).

Activations are usually stored in FP16 (2 bytes).
If the context window is large (e.g., reading a long book), the memory required for activations can equal or even exceed the memory required for the weights.
Note: The 4GB limit provided in your prompt applies specifically to the model weights, not the total RAM required to run a conversation with the model.

2. Training vs. Inference
If you were training this model (teaching it), you would need:

Gradients: The rates of change for the weights (usually stored in FP32 to be precise, or FP16).
Optimizer States: Algorithms like Adam require storing the history of weights, often doubling or tripling the memory usage.
Verdict: It is impossible to train a 27B model in 4GB RAM. It is only possible to run (inference) it there.

3. Implementation Overhead
Pure 1-bit math is rarely implemented as literally as “1 bit per number” because the computer cannot process binary matrices efficiently without translation overhead.

To make it work, the data is often grouped (e.g., 128 bits at a time).
Scaling Factors: To convert 1-bit numbers back to usable numbers, the model usually requires a “scale factor” (a multiplier) stored in 16-bit or 32-bit floating point.
Correction: While the raw math says it fits 4GB, the metadata required to manage the bits might push it slightly over 4GB (e.g., 3.8 GB).

Summary

A 27-billion-parameter model fits in 4 GB with one-bit weights because 1 bit equals 1/8th of a byte.

$$ \frac{27 \text{ billion}}{8} \approx 3.375 \text{ GB} $$

This represents a 16x reduction in size compared to standard AI training formats, making it possible to run massive models on consumer-grade hardware (like a Raspberry Pi or a mid-range laptop) for reading and generating text.

Bonsai 27B Q1_0 3,113 tokens 39s 79.25 t/s

At 72 tokens per second, it’s a reasonable model to work with, but waiting 40 seconds for a long-winded reply (and thinking) has a few concerns with using Agentic Agents. It’s time to do some benchmarks. For comparison, these are the results I had the other day:

_{Model / layout}	GPUs	_Context	_{Prompt tok/s}	_{Gen tok/s}
Qwen 3.8: 27B	Both	8k	405.697	21.431
Qwen 3.8: 27B	Both	40k	265.146	4.003
Qwen 3.8: 27B	Both	64k	214.235	1.738
Laguna XS 2.1 Q4	3080 Ti	8k	314.202	50.331
Laguna XS 2.1 Q4	3080 Ti	40k	478.677	45.269
Laguna XS 2.1 Q4	3080 Ti	64k	448.441	41.098
Qwen3-Coder 30B	Both	8k	1,742.119	66.337
Qwen3-Coder 30B	Both	40k	449.675	42.277
Qwen3-Coder 30B	Both	64k	364.088	23.058

You can see that I often rely on both GPUs to run the larger Qwen models. Laguna is small enough to fit in the RTX 3080 Ti VRAM and avoid the memory-bandwidth bottleneck between the two GPUs. It was the most promising option for speed alone. The target context is 40k to use Hermes, and at least 64k to work with agentic coding agents. Ideally, I would want 100 tokens per second. I can use 41 t/s, but it will feel slow.

As the context grows, the KV cache in VRAM grows too and can potentially start spilling into system RAM, which can slow it down. Another issue I realized later is that when you shard models across multiple GPUs, you don’t want the last layer on a GPU with low memory bandwidth, because it also has to return the output in addition to the layer values.

So with Bonsai’s small size, it leaves plenty of room for the KV cache and context to grow. But can it perform?

I kept trying to run tests, but none of my GPUs showed up as CUDA-capable. I was getting driver initialization failures.

			
NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x31:0xffff:2780)
NVRM: GPU 0000:02:00.0: rm_init_adapter failed, device minor number 1

I rebooted the server twice. I finally shut down the server, turned off the power supply, unblocked the cable, and attempted to reseat the riser ribbon. Somehow, it had slipped out a little, but was still firmly connected. I pushed it in and turned the server on. I booted into the login screen in about two seconds. I headed back upstairs, SSH’d into the server, and confirmed the driver could detect both cards as supporting CUDA.

Hermes started running the performance tests and came back after a few minutes. The Bonsai version of Qwen always performed much better.

Context	Bonsai 27B Q1	Qwen 3.8 27B	Laguna XS 2.1 Q4	Qwen3-Coder 30B
8k	1391.876 / 78.879	405.697 / 21.431	314.202 / 50.331	1742.119 / 66.337
40k	1181.102 / 65.736	265.146 / 4.003	478.677 / 45.269	449.675 / 42.277
64k	1034.801 / 57.864	214.235 / 1.738	448.441 / 41.098	364.088 / 23.058

An “intelligent” model is now within reach for an agentic agent to use locally. The ability to improve quality by running a 27B model, even with such aggressive compression, is a game-changer.

I started thinking – can the 1080 Ti run this as well? Theoretically, yes, but currently, no. I only targeted the SM 8.6 architecture when I built the custom llama, and I would need to rebuild for Pascal SM 6.1. (The prior code above was updated to target both cards)

Building takes a long time. Should I bother if I’m just testing theoretical limits? Would it even be usable? Yes. The dogs need a walk and some playtime. To be safe, I created a separate build for 6.1, so if I ran into problems, I could still use the 8.6 version.

cd ~/src/bonsai-llama
cmake -B build-sm61 \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=61 \
  -DGGML_CCACHE=OFF \
  -DGGML_CUDA_NCCL=OFF
  
cmake --build build-sm61 \
  --config Release \
  -j"$(nproc)"

What a surprise – the GTX 1080 Ti runs Bonsai 27B Q1 at every context size.

Context	RTX 3080 Ti	GTX 1080 Ti
8k	1391.876 / 78.879	255.627 / 29.193
40k	1181.102 / 65.736	195.853 / 22.378
64k	1034.801 / 57.864	162.888 / 19.131

At 64k, the GTX used 8,491 MiB of its 11,264 MiB and sustained 19.131 generation tok/s. It is roughly 6.35x slower at prompt ingestion and 3.03x slower at generation than the RTX at that window, but it is viable as independent background capacity. I now have options…

Models with more parameters, around 27B, can do more advanced processing and reasoning than smaller 8B models. The first issue is how much memory you need, quickly followed by speed. I’m past that now and into the territory where I still want to improve speed, but moving to improving quality. After that, it’s concurrency for multiple agents using the same models at once, and the efficiency of electricity usage for producing work. There are also many other tangents to go down.

Each target has a different set of tests to verify improvements. Tokens per second is only an initial test to compare models for improvement. Even though you may have a high throughput of tokens per second, if a slower model can perform the same amount of work in less time and with less token usage, then that metric goes out the window.

Quality is a metric you build tests to measure: which models pass, fail, or struggle. After watching many Codacus videos, I’ve concluded that if all models pass a test, the test has failed to expose problems and help determine each model’s quality.

Working with local AI models is a constant struggle on small budgets and limited hardware options. Some people suggest using the power’s waste heat to heat homes. I find this wishful thinking. People made the same claims for bitcoin mining, and little came of it. With high-power equipment, you start running into additional costs to manage the heat and the noise from fans. Our summers are hot where no one would think of turning on an electric heater. My basement is cool, but the server warms it up a little and isn’t at full power. If I get better equipment to run bigger models, I’d have to handle more heat and noise.

Looking at the Energy consumption specs on Prism ML’s page, they show mWh/ token is about 1.0 on an RTX 4090. Of course, these are just numbers they provide, and they compare only their own energy consumption across different model sizes rather than other models like Qwen3, which they are based on. I don’t have a basis for comparison, and the other part of that test would be to compare the average tokens used per prompt rather than how much one token consumes – because, after all, we are chasing an overall reduction in energy usage and increased speed to produce an improved-quality result.

Prism ML – Bonsai Energy Consumption

For comparison, my setup includes an RTX 3080 Ti, a GTX 1080 Ti, an M1 Pro, and an iPhone XS Max. I haven’t tested on the MacBook or iPhone.

In their PDF white paper, they report 122.8 tokens per second on an RTX 4090, 162.6 on an RTX 5090, and 146.2 on an RTX Pro 6000 with 512 input tokens, whereas I was targeting 8k, 40k, and 64k contexts. More tokens create a larger KV cache and slower token generation, but this reflects my intended use case, where people often start to feel pain as context grows.

Given that my card is from a generation prior to what they tested and my context windows are larger, I believe I’ve gotten solid results: 78, 65, and 57 tokens per second at 8k, 40k, and 64k windows.

Prism ML Bonsai White Paper – Tokens Per Second