Forgejo Became My Server Canary

I had another server outage, but this one was different. I did not discover it because someone emailed me, because a visitor reported a broken website, or because I happened to check one of my pages.

I discovered it because I tried to push code to Forgejo.

That was the first sign that something was wrong. Forgejo is one of the bigger things I recently added to the server, and it has also started becoming part of my daily workflow. When a git push failed, it immediately got my attention.

The other major recent addition was Nextcloud. I had installed it, but I had not really started using it yet. My plan was to clean up Dropbox first before transferring files over. Still, because Nextcloud was one of the largest new applications added to the server recently, it seemed like a possible suspect. Anything that syncs files, indexes data, or performs background jobs has the potential to pull at CPU, memory, disk, or database resources if it gets busy at the wrong time.

So my first question was not “what broke?” It was “what changed?”

Forgejo was new and actively being used. Nextcloud was new and large, even if I had not moved files into it yet. Either one seemed plausible enough to investigate.

From there, I checked my websites and confirmed they were offline. I also checked DNS because I had recently moved WordPress back to using my own nameservers. DNS still looked correct, so this was not a domain or nameserver issue. The server itself was not responding.

DigitalOcean showed that the droplet had been running at very high CPU and memory usage for roughly three hours. That explained the failed web requests, but it did not yet explain the cause. I tried connecting over SSH and got nowhere. I tried DigitalOcean’s web console, and that timed out too. At that point, the server was wedged badly enough that I had to do a hard restart.

Once the server came back, I moved quickly. My first step was to stop the DigitalOcean monitoring agent, do-agent, because I had recently run into a separate issue where it caused trouble on this same server. That made it an obvious early suspect, and disabling it gave me enough room to continue diagnostics.

The basic health checks looked good. Apache came back up. Disk space was fine. Inodes were fine. Swap was not being abused after the restart. Forgejo was running. Nextcloud did not appear to be the thing hammering the server.

Then the process list showed something else: ImunifyAV’s AI-Bolit / rustbolit scanner had started as a resident process with a --memory=2G option.

On a server with about 3.6 GiB of usable RAM, that is a lot.

The server normally runs at a manageable memory level. This was not a case where the machine had slowly outgrown its purpose. It was more like one security scanner suddenly brought a shop-vac into a dollhouse.

I killed the active rustbolit process, then found the systemd socket and service that could restart it:

aibolit-resident.socket
aibolit-resident.service

I disabled both:

systemctl disable --now aibolit-resident.socket
systemctl disable --now aibolit-resident.service

I also found the daily ImunifyAV cron job:

/etc/cron.daily/imunify-antivirus.cron

That cron file was not directly running the scan itself, but it handled ImunifyAV package updates and related maintenance. Given the timing and the resident scanner behavior, I made the emergency choice to disable its execute permission for now:

chmod -x /etc/cron.daily/imunify-antivirus.cron

After that, the server stabilized. Apache was healthy, the sites came back, disk usage was only around 30%, and inode usage was tiny. The immediate culprit was not storage, Forgejo, or Nextcloud. It was resource pressure from the security tooling.

Later, I found an automated Imunify notification from Saturday warning about outdated WordPress plugins and themes. That helped confirm that ImunifyAV was installed at the WHM/cPanel server level and was scanning WordPress installs. It was not something Nextcloud had installed. It was part of the server’s hosting and security stack.

Once the emergency was under control, I resized the droplet. I doubled the memory and increased the disk from 120 GB to 160 GB. The disk was not the problem, but the extra space came along with the resize. After rebooting, I confirmed the server saw the new resources:

Mem: 7.5 GiB total
Disk: 160G total, 124G available
Inodes: 1% used

I also confirmed the emergency fixes remained in place after the resize: do-agent was still disabled, the AI-Bolit resident socket and service were still disabled, and the ImunifyAV daily cron was still not executable.

The resize does not mean I will blindly re-enable everything right away. More RAM gives the server breathing room, but I still want to understand why ImunifyAV decided to launch such a heavy resident scanner in the first place. For now, I am leaving the aggressive scanner disabled while I manually update outdated WordPress components and monitor the server for a safe period.

The larger lesson is that Forgejo has become my server canary. Before, a website could be down for hours before I noticed. Now that I use the server as part of my daily work, I notice problems faster.

Nextcloud is still part of the plan, but I want the server to be stable before I start moving a large collection of files into it. That gives me time to clean up Dropbox, monitor the resized server under normal load, and make sure background jobs, backups, scans, and monitoring are not conflicting.

This outage was frustrating, but it made the server better. I found a hidden pressure point, ruled out my recent installs, added more breathing room, and now have a clearer plan for monitoring it going forward.