I learned a bit more about my setup with the 1080 Ti and 3080 Ti and the unexpected speed problems when sharding larger models. I thought the GPU was the problem, but now I believe the slowdown is directly related to memory bandwidth.

I also asked Hermes to check the job status to compare it with the initial results now that eight hours have passed. With more data, we could estimate the 1080 Ti’s performance more accurately instead of sticking to conservative estimates.

Based on the production evidence, I increased the 1080 Ti’s share of work to 30%. I considered increasing the 3080’s share to process more during quiet cycles, but the safety margin would shrink, and the Facebook batch would conflict more often. This tweak primarily improves the balance so that the 1080 Ti is less likely to finish its fixed partition early and sit idle. Contention with the Facebook job remains, as it shares the 3080 Ti with the email job.

I started getting a bit confused on how the jobs were actually setup vs what I asked. So Hermes started explaining.

Why do the video cards have different time budgets?

Well, there is my answer. It leaves 35 seconds for Facebook, but Facebook only has a possible collision 20% of the time because we have a 3-minute job vs. a 15-minute job overlap. I’m wasting two minutes when I could be processing email on the 3080.

From here, I instructed Hermes to extend the time budget for the 3080 Ti if the email job ran when the Facebook job wasn’t expected. For long-running batch jobs, every second counts – and I just found 140 seconds available in a 15-minute window.

Hermes didn’t do exactly as I asked – it took an initiative and decided not to run the mail job if the Facebook job was about to run, it would skip the email job. If the Facebook job was about to run soon, it would stop 20 seconds before that job kicked off.

The adaptive fix protects Facebook, but it shortens or skips the email run near the Facebook deadline and doesn’t guarantee more aggregate email time than I had. I thought I would gain 140 seconds of processing time, but it was closer to 45 seconds. That’s 3 more emails processed per 15-minute window, instead of the 10 I thought I was getting while improving the Facebook jobs’ reliability.

We were reaching a point where we were trading off protecting the two jobs against maximum processing throughput, and with a cron job, reliability was hard to manage when two separate jobs occasionally competed for the same GPU.

To maximize throughput with what I have, I need to focus on the distributed coordinator agent runners and stop relying on cron jobs. Once that is up and running, it also lets jobs run on other machines that may have models that can process data.

Broken 3090 FTW3

A bit of a context switch. 3090 cards with 24 GB of VRAM start around $800 on eBay. I found one (item 298651176498) with 25 bids up to $208.50, with three days to go. However, it had some problems.

I wasn’t bothered by the video output, as this is just for running headless AI models.

The red lights could mean anything. The easiest fix is that one of the three 8-pin PCIe power inputs might not be seated properly or might be missing. I asked the seller, but if they say all three power inputs were connected correctly, I’d have to pass. If they didn’t test it, then it’s really a gamble. Am I okay with losing money if it doesn’t work?

I started looking at the different pictures. It was covered in dust, and the warrenty/model/barcode sticker was fairly worn. It’s definitely seen some action rubbing against something. The warranty sticker also had cuts that looked like someone intentionally crossed out the model number and barcode. I think it was thrown out. This is getting to be something were its only worth what I would knowingly spend on an electronics-repair project or parts donor. I need something dependable, so I’ll pass.

I also looked at the NVIDIA M40S, which seemed cheap at $170 each, with 24 GB of VRAM and low power draw, but compared to my current setup and problems, it was a hard pass. I started looking at a variety of affordable cards and comparing what they could do. Memory alone was not enough – I also needed bandwidth. I kept feeling like I was being pushed toward the RTX 30 series, and only the 3090 had double the VRAM capacity. One interesting contender was the Tesla V100 at $649 (Item 327316565001) that had the memory bandwidth and 32 GB of VRAM. The TFLOPS at half the speed was a little concerning, and I wasn’t familiar with HBM2 VRAM.

With 36 GB, the Tesla V100 can run Qwen3.8 27b or Laguna XS Q4 on a single card without the overhead of communicating with another GPU, and its memory bandwidth is roughly comparable to the existing 3080 Ti. It’s passively cooled and needs a fan/duct.

I was looking at the RTX 3090 for its memory bandwidth, but used cards are $800+, so picking up the Tesla V100 can save $150 and add more VRAM with negligible impact on memory bandwidth.

I’ve been looking at older video cards on a budget, but how do the latest cards compare with my MacBook and Raspberry Pi? I did a bit of digging to compare.

A special note on the Raspberry Pi 5: it can run a separate GPU via a PCIe connector, so the 1080 Ti can move to a Raspberry Pi 5 once Kaylee has its GPU swapped for something better.

Then I started wondering about other stuff in the house and how the tech compares.

Hmm… I’ve hardly touched the PlayStation since I got it. If there were a way to make it useful, I’d be happy to run models on it. I used to run Red Hat on the PS2 and PS3, but someone stole my PS2, and Sony removed Linux support on the PS3 after a firmware update. The funny thing is, they promoted this, saying their systems were powerful enough to run as desktop computers. I’d love to have a Linux game console again and have it chugging away running AI models. The Ps5 seems comparable to my 1080 Ti, but with more memory available.

I wonder whether Sony would ever consider releasing a firmware update that lets Linux run on the PS3 again, since it no longer sells the console or games. Its tech is simply too old. The VRAM seems too small to run even the smallest models. Something like smollm2-135m is 271 mb before you consider context, runtime, and llama.cpp overhead. At best, I can run simple parsing jobs on deterministic data without AI, or use it as a remote agent to a frontier model. However, the power draw simply isn’t worth it when a Raspberry Pi can do the same. It would be better to use its GPU for rendering 3D images – but again, we’ve got outdated tech.

Batching Distributed Agents

I asked Hermes to pick up where we left off on yesterday’s distributed batch work before I added the GPU and started running benchmark tests. It was close to being ready for testing, but with an additional GPU that needed its own dedicated agent and ran slower, I asked it to add features to allow me to see the performance of agents over time and to see when each one was next due to report its results based on past performance, and allow agents to register with various models that they can run. We also setup the site for a login so it would be ready to deploy once it’s stable.

I then started setting up an agent on my MacBook, planning to hold off until the last moment, when we would need to stop the existing cron jobs. I had Hermes add the ability to register a new agent so we wouldn’t end up with random agents connecting willy-nilly without anyone logging in first to register them.

I realized that if I used only the registration token for agent authentication, I would run into problems if I set up multiple Raspberry Pis with the same image, since they would all process the same work. I needed to allow registration only once while still letting the token be regenerated in case it was ever compromised or needed to be reissued. The one-time enrollment code would have the agent generate their own credentials.

I also wanted to display agent hardware and provider capability inventory on the dashboard so I could better understand the fleet. Instead of depending on every agent to have Ollama installed, I also wanted to support agents that were simply remote dispatchers that could call OpenAI Codex, Agentic Claude, etc.

Hermes wanted to know what I had on my MacBook – well, nothing. I was thinking about calling Ollama over the network, but now I was having second thoughts.

system_profiler SPHardwareDataType
# 2026-09-10 20:45:41.591 system_profiler[8653:17864822] hw.cpufamily: 0x1b588bb3
# Hardware:
# 
#     Hardware Overview:
# 
#       Model Name: MacBook Pro
#       Model Identifier: MacBookPro18,3
#       Model Number: MKGQ3LL/A
#       Chip: Apple M1 Pro
#       Total Number of Cores: 10 (8 Performance and 2 Efficiency)
#       Memory: 16 GB
#       System Firmware Version: 18000.120.36
#       OS Loader Version: 18000.120.36
#       Serial Number (system): [REDACTED]
#       Hardware UUID: [REDACTED]
#       Provisioning UDID: [REDACTED]
#       Activation Lock Status: Enabled
tailscale status
# 100.66.58.61    lewie-macbook  lewismoten@  macOS  -                                                                 
# 100.124.159.40  codejamboree   lewismoten@  linux  -                                                                 
# 100.89.122.105  iphone-xs-max  lewismoten@  iOS    -                                                                 
# 100.84.224.31   kaylee         lewismoten@  linux  active; direct 204.111.226.149:41641, tx 143234832 rx 2880405648  
python3 --version
# Python 3.13.2
ollama --version
# zsh: command not found: ollama
ollama list
# zsh: command not found: ollama
curl -fsS http://127.0.0.1:11434/api/tags
# curl: (7) Failed to connect to 127.0.0.1 port 11434 after 1 ms: Couldn't connect to server
ollama pull qwen3:8b
# pulling manifest 
# pulling a3de86cd1c13: 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 5.2 GB
# verifying sha256 digest 
# writing manifest 
# success 
ollama list
# NAME        ID              SIZE      MODIFIED       
# qwen3:8b    500a1f067a9f    5.2 GB    15 seconds ago    
curl -fsS http://127.0.0.1:11434/api/tags
# {
#   "models": [
#   {
#     "name":"qwen3:8b",
#     "model":"qwen3:8b",
#     "modified_at":"2026-09-10T20:39:47.997418119-04:00",
#     "size":5225388164,
#     "digest":"500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41",
#     "details": {
#       "parent_model":"",
#       "format":"gguf",
#       "family":"qwen3",
#       "families":["qwen3"],
#       "parameter_size":"8.2B",
#       "quantization_level":"Q4_K_M",
#       "context_length":40960,
#       "embedding_length":4096
#       },
#     "capabilities":["completion","tools","thinking"]
#   }
#   ]
#}%            

Eventually, we got to a point where I wanted to simplify things and have the worker code downloadable from the dashboard, along with a setup script to simplify the configuration process. I knew I’d eventually want to set this up without reading a bunch of documents. Just show me a quick interface to set it up.

I ran through the setup script a few times and noted a few additional features. For one, the setup script needed to be dynamically generated based on the website it was downloaded from, so the controller URL would be pre-populated.

Ok – everything’s connected. Now I need a “real” job. I started with a small job to check the sitemap.xml for my 1976 Sebring-Vanguard CitiCar restoration/modernization blog. Then I ran another job to compile a list of transcripts from local government meetings.

I started to realize that the worker agents needed to report back with artifacts so that other jobs that depended on the first could access that information. In addition to that, some of these tasks may result in downloading a video file of a few GB in size – especially if it’s a long meeting where everyone wants to talk and/or is held at the school auditorium with a higher video resolution that quadruples the file size.

I was essentially creating something similar to build agents at this point. With all these manual updates for each test, I was getting tired of going through the full installation process each time. I also had Hermes wire in an auto-update.

At first, Hermes made it an argument to pass to the worker to check for and perform an update. I needed the worker to update itself. I wasn’t going to bother updating 20 or so Raspberry Pis every time a new update went out. Instead, I told Hermes to have the worker provide its version each time it asks for new work or reports a completion with artifacts, so that the server can respond with an instruction to update the worker.

The further I progressed through the steps, the more problems I ran into that I had to solve. One job to download a transcript depended on another job to discover the transcript. Another job would then parse transcripts. All these jobs depended on each other, but they weren’t updating based on the outcomes. I originally moved toward artifacts, but now I was starting to see a need for hooks after work was completed and project data that all jobs could access in one form or another.

I now had a way to see the data on the dashboard that was coming back and what was pending. Things were starting to become visual.

Other problems started popping up when worker agents reported back too often, which could look like a denial-of-service (DOS) attack and pollute the web server’s logs with unnecessary requests. The workers were changed to throttle themselves to report back no more than once every 10 seconds, or 6 times per mission. To do this while still processing more than 6 units of work per minute, the server was set up to accept multiple units of work in a single request. The throttle to limit the agents reporting back to the server was also applied to the agents making requests to external websites as well. After all, these are long running tasks that may take days to process. In the meantime, the local model can be used to analyze documents until the window opens again.

Eventually, after all the back-and-forth, I finally had something working where the MacBook could fetch lists of government meetings, download a transcript, process it, and then download the next one.

I realized the work allocation with the distributed weights differed significantly from the actual time spent, so I created a separate pie chart to show the difference. Part of this is because some jobs run only once a day. Web request throttling also reduces the time that job spends fetching data. The main focus of worker weights is distributing the longer-running analysis with the AI models. Currently, I have only one analysis job set up, so it takes most of the time compared to the other jobs.

The other thing I wanted to see was metrics over time. I had Hermes chart the token processing rate (tokens per second) and jobs completed. I had it add filters so that I could limit the results per agent and/or model. This would let me see whether using different models affects job completion speed or tokens per second for long-winded models.

Another problem I ran into was a model that failed to respond to a specific job properly 15 times.

Job #11 failed: Ollama transcript response quarantined: Expecting ‘,’ delimiter: line 158 column 2 (char 352)

After that, the worker did nothing but keep claiming the same job repeatedly and did nothing about it. I asked Hermes to move failed jobs to the end of the queue so we don’t waste time repeating the same job. Hermes set up the failed job units of work to be deferred by 15 minutes. All agents skip the unit of work until the window passes.

With that, I heard the fans in my laptop spin up again as it moved on to the next unit of work.

The facts needed a bit of formatting rather than showing raw JSON responses, so Hermes made them more presentable in the web interface while also allowing them to be downloaded as well.

In Closing

Much progress has been made today. A setup script allows a quick installation of a worker agent. It’s now downloading transcripts and parsing out facts about what was discussed during government meetings. Jobs that depend on each other are now being populated as each one brings in its own data. We now have artifacts as a result of jobs. We also have projects where I can view the results. There is an additional pie chart of the actual time being allotted to different workers for different jobs, and we also have a timeline chart that can filter results by model, agent, and time period.

Tomorrow is 9/11. I’ve been asked to attend a ceremony at the town square, and then I’m off to feed the seniors afterward.