While I was splitting off accounts today and setting up subdomains, I did some research beforehand and had an agent running a task in the background.

I often see people on YouTube Videos offering some kind of secret tip to speed up Ollama by running models in parallel, but with only 12 GB of GPU memory and most of it occupied, parallel requests would multiply context memory usage and risk GPU contention, which I mitigated the other day when changing all jobs to use the same model, qwen3:8b, and ditching models that spilled over into the system RAM.

Using a model that is partially in system RAM is very slow and introduces bottlenecks through the CPU and limited bandwidth en route to the GPU. Swapping between multiple models that occupy more than half of the available GPU memory caused problems, with the GPU spending a minute per request swapping models in and out.

I worked with Hermes and ran some benchmarks before and after configuring OLLAMA_FLASH_ATTENTION.

Benchmark result on qwen3:8b, 8,192-token context, uncached ~4,100-token prompts:

Setting	Prompt eval	Generation	Mean request time
Flash Attention off	4,781 tok/s	111.6 tok/s	3.42 s
Flash Attention on	4,775 tok/s	107.8 tok/s	3.54 s

It produced no measurable speedup for the present workload; the small ~3–4% difference is normal benchmark variance, not evidence of a regression.

Hermes also suggested a few other ways to improve speed.

1. Eliminate scheduling gaps. Email batches run up to 145 seconds on a 3-minute schedule, leaving up to ~35 seconds idle each cycle; Facebook can also skip when email owns the shared inference lock. A single continuous queue worker would keep Qwen busy and choose email or Facebook work without conflict. This is likely the largest safe throughput improvement.
2. Benchmark a shorter response ceiling. Email extraction allows num_predict: 1800, despite the output schema being fairly bounded. Testing 900 or 1200 against a representative sample could reduce long-tail response time—only adopt it if validation quality stays unchanged.
3. Benchmark smaller source excerpts. Current email prompts send up to 12,000 source characters. Reducing that selectively for receipts, notices, and short messages can improve prompt evaluation time without losing useful facts.

I decided to eliminate the scheduling gaps. Instead of running a cron job every few minutes, I wanted to set up a service that would keep running and pick up the next task once it received a response. However, I wanted a little bit more. I wanted a web interface where I could set job priorities so the next batch picked up was weighted accordingly. So in a given hour, maybe 70% of jobs would be for processing email, 20% for Facebook, and 10% for media transcripts. I wanted the ability to temporary toggle some batches on/off, and support for a separate Ollama server to pickup work if I had set one up. I also wanted to be able to add new jobs via the web interface. In a sense, I was preparing it for future expansion. There are other documents that I would like to have processed, and doing anything locally takes a great deal of time on limited hardware. Perhaps I could even setup my MacBook to process some jobs overnight when I’m not using it.

So Hermes went off to work, creating a mockup of a website running on PHP and SQLite, with the plan to eventually host it on my virtual private server in a data center and distribute jobs among multiple clients requesting work.

I was troubleshooting a few problems throughout the day, putting out a few fires. By the end of the day, I realized I had left Hermes alone and that it was ready for me to evaluate. I set up an SSH tunnel to the server and opened the website in my browser.

It’s simple enough. I set up a test job through the interface. I may need a pie chart to better understand the work based on weighted priority and enabled state, but it’s a good first pass. It could use some localization for formatting large numbers.

I decided that it was good enough to keep running and let it continue. However, I had a bit of a predicament. Not only was I about to head to sleep, but I had a medical appointment tomorrow. I didn’t want to have it waiting for me to review something before proceeding.

We decided on an action plan. Hermes would keep the existing archive workers and data untouched and improve the controller’s worker/lease safety and usable metrics. To keep the existing cron workers safe, I tasked it with starting to set up jobs for a few tasks. One was to start processing a blog for my 1976 Sebring-Vanguard CitiCar. The other was a bigger one – mainly to start monitoring a few local government sites for video transcripts of board meetings and start extrapolating facts and summaries, while throttling its requests. When I get home, I’ll review the code and decide whether to enable it.

I do plenty of research on government documents, and it would be nice to automate some of the background processing to help connect the dots more easily moving forward. However, Facebook and email processing are higher priorities. Documents are spread across many places on the various sites, and I often spend time going back and forth to locate information or request it.

In response to requests, it started adding more features. You can now select a target model so an Agent Runner can find jobs it can run. This adds the flexibility to use different models, such as Gemma4 for image processing.