Lewis Moten

Unveiling the Code: Institutional Memory & Obsessions

Death by Metrics

DigitalOcean Monitoring Agent, CageFS, and a 95 GB Log Storm

When “Something Feels Off”

In the early morning hours of Friday, April 24, 2026, something changed on my server.

At first, it was subtle.

CPU usage suddenly jumped and remained elevated. The system normally idles comfortably, but it has now been hovering at 70–75% continuously. The server was still online, websites were responding, WHM was accessible, and email appeared functional, so the issue initially looked more like a nuisance than an emergency.

I had a heavy schedule over the past few days, including meeting with a client, a meeting as both a secretary and parade coordinator, breakfast at another organization, the library’s Taste for Books fundraiser, Church Business Meeting, Audio management for the church, Earth/Lone Pine Day, and preparing for a meeting at the Chamber of Commerce, and visiting another lodge later that evening. Suffice to say, I’ve been very busy.

I noticed the elevated resource usage yesterday, and even mentioned it during the conversation Monday morning at my Chamber checkup, but the server remained operational enough that I mentally categorized it as something to investigate later.

That turned out to be a dangerous assumption.

Over the next few hours, the problem quietly escalated.

The Warning Signs

The first major warning arrived through cPanel monitoring before my check-in.

At 9:50 AM on April 27, I received a warning notification indicating the server disk usage had reached 82.77%:

Disk usage warning
The filesystem “/var/log/apache2/domlogs” mounted at “/var/log/apache2/domlogs” reached “warn” status because you currently use 82.77% of its available blocks.

The notification reported:

  • Used: 98.75 GB
  • Available: 20.55 GB
  • Total: 119.29 GB

Later that evening, at 11:09 PM, another alert arrived.

This time the notification had escalated to critical status:

Disk usage warning
The filesystem “/var/log/apache2/domlogs” mounted at “/var/log/apache2/domlogs” reached “critical” status because you currently use 92.56% of its available blocks.

The system now reported:

  • Used: 110.42 GB
  • Available: 8.87 GB
  • Total: 119.29 GB

At the time, the message implied Apache domlogs were responsible, but later investigation would reveal the true culprit was elsewhere.

The server was consuming approximately 11.67 GB in roughly 13 hours:

110.42 GB – 98.75 GB = 11.67 GB

That worked out to nearly 11.67 GB in 13 hours, or approximately 0.9 GB/hour

which is an enormous amount of sustained log growth.

Services Begin Failing

By the morning of April 28, the situation had worsened dramatically.

When I finally sat down to investigate seriously, I discovered:

  • Websites were unresponsive
  • WHM was inaccessible
  • cPanel logins failed
  • Services such as queueprocd were failing
  • Statistics processing lagged behind
  • CPU usage remained abnormally high
  • The disk usage graph was missing
Image 1. CPU Usage abrupt change

One of the first service alerts I received involved queueprocd:

The service “queueprocd” appears to be down.

Additional service instability appeared around:

  • cphulkd
  • statistics processing
  • monitoring systems
  • various cPanel background services

At this point, only SSH access remained functional.

That detail became extremely important.

Recovering Through SSH

Because WHM and the websites were inaccessible, recovery had to be done entirely via terminal access.

The first step was verifying disk free space. The result immediately confirmed the root filesystem was effectively full:

df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/vda1  120G 120G   48K 100% /
...
Image 2. File system 100% used

Next, I checked the disk free space of the inode. The server showed inode usage at approximately 98%, confirming the system was under significant filesystem stress.

df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
...
/dev/vda1    681K  664K   17K   98% /
...
Image 3. inode usage at 98%

Home Directory

I initially checked the home directory to see if there were any issues with the large file size, but didn’t find anything meaningful.

[root@server ~]# du -xh | sort -h | tail -50
...
6.5M	./.spamassassin
6.5M	./.spamassassin/sa-compile.cache
12M	./.cpanm
12M	./.cpanm/work
12M	./.cpanm/work/1735591090.23983
13M	./.cache/composer/files
18M	./.cache/composer
19M	./.cache
37M	.

Clear Cached Metadata

I initially cleared some cached metadata from package installations, hoping it would free up resources since the system was exhausted and the data would no longer be needed. The system was so exhausted that it couldn’t even create a lock file, as there were no blocks available to write to.

[root@server ~]# dnf clean all
This system is receiving updates from CloudLinux Network server.
[Errno 28] No space left on device: '/var/cache/dnf/metadata_lock.pid'

Finding the Largest Directories

To determine where the space was being consumed, I ran a disk usage check on the log folder. The output quickly narrowed the problem down to the root of that folder. I then drilled deeper and deeper.

du -xhd1 / 2>/dev/null | sort -h
...
98G  /var
133G /
du -xhd1 /var 2>/dev/null | sort -h
...
96G /var/log
98G /var
du -xhd1 /var/log 2>/dev/null | sort -h
...
116M /var/log/apache2
96G  /var/log
Image 4. Large top-level directories
Image 5. Tracing large directory to /var/log

That was the breakthrough moment. The problem was not Apache domlogs at all. It was the system logs.

Identifying the Giant Log Files

To identify the specific files consuming large amounts of space in the folder, I used the find command. The output revealed that the true source of the disk exhaustion was the two most recent message logs, which were much larger than normal. Together, these two files accounted for approximately 95 GB of usage.

find /var/log -maxdepth 1 -type f -exec ls -lh {} \; | sort -k5 -h
...
-rw------- 1 root root 51M Apr  5 00:00 /var/log/messages-20260405
-rw------- 1 root root 51M Apr 12 00:00 /var/log/messages-28260412
-rw------- 1 root root 51M Apr 19 00:00 /var/log/messages-28260419
-rw------- 1 root root 40G Apr 25 23:58 /var/log/messages-20260426
-rw------- 1 root root 55G Apr 28 13:33 /var/log/messages
Image 6. Find large files

Inspecting the Log Before Deleting Anything

Rather than immediately deleting the files, I first inspected their contents. Initially, I looked at the last 100 entries, but they were large and repetitive. I was finding that every 18 minutes, 94 log entries, each truncated to 8 KB, were added, all of which looked the same. There were also two journal failures.

I then went and looked at an individual log. The result pointed directly at the DigitalOcean monitoring agent and often repeated “cagefs-skeleton”, along with numerous CageFS mountpoints.

Log Details

Apr 28 13:31:45 server rsyslogd[792245]: imjournal: open() failed for path: ‘/var/lib/rsyslog/imjournal.state.tmp’: No space left on device [v8.2506.0-2.el9 try https://www.rsyslog.com/e/2433 ]

rsyslogd

Apr 28 13:31:45 server /opt/digitalocean/bin/do-agent[708042]: /home/do-agent/cmd/do-agent/run.go:59: failed to gather metrics: 28 error(s) occurred:#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/local”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/local”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/local/bin”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/local/bin”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton“} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton“} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/opt”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/opt”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/libexec/passenger”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/libexec/passenger”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.python”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.python”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/imunify360/wp-plugins”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/imunify360/wp-plugins”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.nodejs”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.nodejs”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/passenger”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/usr/share/passenger”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/lib/mysql”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/lib/mysql”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/lib/proxyexec/cagefs.sock”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/lib/proxyexec/cagefs.sock”} gauge:{value:49152}} was collected before with the same name and label values#012* collected metric “node_filesystem_size_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/www/cgi-bin”} gauge:{value:1.28089829376e+11}} was collected before with the same name and label values#012* collected metric “node_filesystem_free_bytes” { label:{name:”device” value:”/dev/vda1″} label:{name:”device_error” value:””} label:{name:”fstype” value:”xfs”} label:{name:”mountpoint” value:”/usr/share/cagefs-skeleton/var/www/cgi-bin”} gauge:{value:491

do-agent

[root@server ~]# tail -100 /var/log/messages
...
lots of repetative entries
...
[root@server ~]# tail -1 /var/log/messages
/opt/digitalocean/bin/do-agent run.go
failed to gather metrics
collected metric "node_filesystem_size_bytes"
{
  device: /dev/vda1
  device_error: 
  fstype: xfs
  mountpoint: /usr/share/cagefs-skeleton/usr/local
  gauge: 1.28089829376e+11
}
was collected before with the same name and label values
...
... repeat again, and again
Image 7. Reviewing the last entry in 55 GB log

This immediately suggested an interaction between:

  • DigitalOcean’s monitoring agent
  • CloudLinux CageFS
  • Prometheus filesystem metric collection

Stopping the Log Storm

I went from investigation to containment. I stopped the runaway logging immediately by stopping the service and preventing it from starting again on VPS restart. Afterward, I truncated the excessive log files.

systemctl stop do-agent
systemctl disable do-agent
truncate -s 0 /var/log/messages
truncate -s 0 /var/log/messages-20260426
Image 8. Stop Digital Ocean Agent and truncate logs

This instantly recovered approximately 95 GB of disk space. A fresh disk usage check confirmed recovery. The server dropped from nearly 100% utilization to approximately 21%.

df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/vda1  120G  25G   95G  21% /
...
Image 9. Review disk usage

CPU Usage Collapses Back to Normal

One of the most revealing observations came immediately afterward. Before cleanup, the server CPU remained around 70–75% continuously. After stopping the DigitalOcean agent and clearing the logs, CPU usage dropped to roughly 4–6% and eventually stabilized near idle.

Image 10. CPU utilization returns to normal

A later top output showed the server was effectively back to normal operation, with 99.3% CPU idle, load averages below 1%, and healthy memory. Swap was in use, but not exhausted, and no runaway processes were consuming the CPU. This strongly confirmed that the runaway logging itself had become the dominant system load.

top
top - ... load average: 0.63, 0.74, 0.97
...
%Cpu(s): ... 99.3 id, ...
MiB Mem: ... 436.5 free ... 1954.7 buff/cache
MiB Swap: ... 2202.2 avail Mem
...   USER ...  %CPU %MEM   TIME+ COMMAND
... mongod ...   0.7  4.2 0:13.91 mongod
... cpanelc+ ... 0.3  0.1 0:01.70 p0f
... root ...     0.3  0.6 8:54.00 agent360
... root ...     0.0  0.4 47:17.36 systemd
...
Image 11. System load looks nominal

Can You Hear Me Now?

At this point, the problem was contained, and I needed to get web services back online. I started with a simple cURL request to access the WHM/cPanel. I met with an error indicating they had not recovered. I restarted the service, checked its status, and verified that it was responding to HTTP requests afterward.

[root@server ~]# curl -k https://127.0.0.1:2087
curl: (7) Failed to connect to 127.0.0.1 port 2087: Connection refused
[root@server ~]# /scripts/restartsrv_cpsrvd
Waiting for “cpsrvd” to start ……waiting for “cpsrvd” to initialize ………finished.
Service Status
	cpanel (cpsrvd (SSL) - dormant mode - accepting connections) is running as root with PID 1720812 (systemd+/proc check method).
Startup Log
	Apr 28 13:42:32 server.codejamboree.com systemd[1]: Starting cPanel services...
	Apr 28 13:42:32 server.codejamboree.com restartsrv_cpsrvd[1720828]: License is valid and has already updated recently.
	Apr 28 13:42:32 server.codejamboree.com restartsrv_cpsrvd[1720812]: Starting PID 1720812: /usr/local/cpanel/libexec/cpsrvd-dormant
	Apr 28 13:42:32 server.codejamboree.com systemd[1]: Started cPanel services.
cpsrvd started successfully.
[root@server ~]# systemctl status cpanel
● cpanel.service - cPanel services
     Loaded: loaded (/etc/systemd/system/cpanel.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-04-28 13:42:32 UTC; 17s ago
   Main PID: 1720812 (cpsrvd (SSL) - )
     Status: "Ready"
      Tasks: 2 (limit: 22864)
     Memory: 25.4M
     CGroup: /system.slice/cpanel.service
             └─1720812 "cpsrvd (SSL) - waiting for connections                    "
Apr 28 13:42:32 server.codejamboree.com systemd[1]: Starting cPanel services...
Apr 28 13:42:32 server.codejamboree.com restartsrv_cpsrvd[1720828]: License is valid and has already up>
Apr 28 13:42:32 server.codejamboree.com restartsrv_cpsrvd[1720812]: Starting PID 1720812: /usr/local/cp>
Apr 28 13:42:32 server.codejamboree.com systemd[1]: Started cPanel services.
[root@server ~]# curl -k https://127.0.0.1:2087
<!DOCTYPE html>
...

Bringing cPanel Back Online

With free space restored, I restarted critical hosting services with cPanel that integrate with WHM/cPanels monitoring, logging, dependency checks, and service management system.

  • cpsrvd for the primary daemon behind cPanel/WHM forces fresh sockets, fresh PID files, cleans memory state, and restore listener bindings
  • queueprocd for deferred background tasks inside cPanel, restores cPanel’s internal housekeeping operations
  • httpd for Apache, which serves web pages, rebuilds listeners, clears hung workers, reopens log files, and restores a clean state to serve web pages
  • Exim for mail transport to restore queue processing, spool handling, rebind SMTP listeners and resume mail delivery

Following an earlier message about queueprocd failing, I restarted it after checking its status to confirm it was running. I also restarted additional WHM/cPanel-related services.

[root@server ~]# systemctl status queueprocd
● queueprocd.service - cPanel Queue services
     Loaded: loaded (/etc/systemd/system/queueprocd.service; disabled; preset: disabled)
     Active: active (running) since Tue 2026-04-28 13:42:09 UTC; 9min ago
   Main PID: 1720679 (queueprocd - wa)
     Status: "Ready"
      Tasks: 1 (limit: 22864)
     Memory: 7.1M
     CGroup: /system.slice/queueprocd.service
             └─1720679 "queueprocd - waiting up to 60s to process a task"
Apr 28 13:42:09 server.codejamboree.com systemd[1]: Starting cPanel Queue services...
Apr 28 13:42:09 server.codejamboree.com startqueueprocd[1720679]: ==> cPanel TaskQueue Processing Daemo>
Apr 28 13:42:09 server.codejamboree.com systemd[1]: Started cPanel Queue services.
[root@server ~]# /scripts/restartsrv_cpsrvd
...
Service Status
	cpanel (cpsrvd (SSL) - dormant mode ...
...
cpsrvd restarted successfully.
[root@server ~]# /scripts/restartsrv_httpd
...
Startup Log
	Notice: journal has been rotated since unit was started, output may be incomplete.
httpd restarted successfully.
[root@server ~]# /scripts/restartsrv_exim
...
exim restarted successfully.
Image 12. Restart cPanel
Image 13. Restart cPanel Background Services
Image 14. Restart Apache
Image 15. Restart Email

I also checked the cPanel service status:

systemctl status cpanel
...
Active: active (running) ...
...
Status: "Ready"
... cpsrvd (SSL) - wating for connections
...
... Started cPanel Services.
Image 16. Check cPanel status

Once services restarted successfully:

  • WHM became accessible again
  • websites began responding
  • cPanel recovered
  • queue processing resumed
  • system load normalized

Investigating the Timeline

With the server stable, I began reconstructing what triggered the issue.

I checked the package update history and found that do-agent was upgraded from 3.18.10-1 to 3.18.12-1 on April 24, 2026, around 4 am UTC (midnight EST), just as the CPU usage jumped to 75% and has stayed there ever since. The timing suggested a regression in 3.18.12-1.

grep do-agent /var/log/dnf.log*
...
2026-04-24 03:49 ... Package do-agent.x86_64 3.18.10-1 will be upgraded
2026-04-24 03:49 ... Package do-agent.x86_64 3.18.12-1 will be an upgrade
...
... Upgraded: do-agent-3.18.12-1...
...
Image 17. Review package updates for Digital Ocean Agent
Image 18. Compare the timeline when the CPU usage changed

Verifying the Installed Version

I confirmed the installed agent version:

/opt/digitalocean/bin/do-agent --version
...
Version: 3.18.12
...
Build Date: Tue Apr 21 22:02:26 UTC 2026
...
Image 19. Verify installed version

Reviewing the problem, everything was surrounding metrics in /usr/share/cagefs-skeleton

[root@server ~]# journalctl -u do-agent -n 50 --no-pager
Apr 28 13:40:24 server.codejamboree.com /opt/digitalocean/bin/do-agent[708042]: /home/do-agent/cmd/do-agent/run.go:59: failed to gather metrics: 28 error(s) occurred:
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/local"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/local"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/local/bin"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/local/bin"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/opt"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/opt"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/libexec/passenger"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/libexec/passenger"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/imunify360/wp-plugins"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/imunify360/wp-plugins"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.nodejs"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.nodejs"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/passenger"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/passenger"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.python"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.python"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/lib/proxyexec/cagefs.sock"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/lib/proxyexec/cagefs.sock"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/lib/mysql"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/lib/mysql"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/spool/at"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/spool/at"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/www/cgi-bin"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/www/cgi-bin"}  gauge:{value:20480}} was collected before with the same name and label values
* collected metric "node_filesystem_size_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/www/html"}  gauge:{value:1.28089829376e+11}} was collected before with the same name and label values
* collected metric "node_filesystem_free_bytes" { label:{name:"device"  value:"/dev/vda1"}  label:{name:"device_error"  value:""}  label:{name:"fstype"  value:"xfs"}  label:{name:"mountpoint"  value:"/usr/share/cagefs-skeleton/var/www/html"}  gauge:{value:20480}} was collected before with the same name and label values
...

Attempting to Restore Filesystem Metrics

At first, I attempted to exclude CageFS mountpoints from collection using different flags. However, the agent repeatedly failed to start. The service repeatedly failed stating that ignored-mount-points and mount-points-exclude are mutually exclusive.

--collector.filesystem.mount-points-exclude
--collector.filesystem.ignored-mount-points
journalctl -u do-agent -n 30 --no-page
...
...failed to create NodeCollector:
--collector.filesystem.ignored-mount-points and
--collector.filesystem.mount-points-exclude are 
mutually exclusive
...
Image 20. Review Journal for Digital Ocean Agent errors

This suggested that the agent automatically injected one of the filesystem exclusion flags.

Stable Mitigation

Ultimately, the stable workaround was to disable only the filesystem collector by overriding the configuration

systemctl edit do-agent
...
[Service]
ExecStart=
ExecStart=/opt/digitalocean/bin/do-agent --syslog --no-collector.filesystem
...
Image 21. Digital Ocean Agent Configuration

After reloading systemd and restarting the service:

systemctl daemon-reload
systemctl reset-failed do-agent
systemctl restart do-agent
Image 22. Restart Digital Ocean Agent

This restored metrics for CPU, memory, network, and load while disabling only filesystem statistics collection.

Filing a Support Ticket

Once the system stabilized, I submitted a detailed report to DigitalOcean support.

The ticket included:

  • the exact version upgrade timeline
  • screenshots of CPU graphs
  • disk usage growth
  • service failures
  • the duplicate metric errors
  • and the interaction with CloudLinux CageFS mountpoints
Ticket #12093409
do-agent 3.18.12 causes runaway logging and high CPU on CloudLinux CageFS Droplet

April 28, 2026 10:40 AM

Type: Compute, Sub Type: Droplet, Tertiary Type: Graphs & Monitoring

Environment:

* CloudLinux + cPanel/WHM
* CageFS enabled
* DigitalOcean Droplet
* do-agent upgraded automatically from 3.18.10-1 to 3.18.12-1 on 2026-04-24 around 03:49 UTC

Issue:
Immediately after the upgrade, CPU usage rose from normal levels to ~75% continuously. /var/log/messages grew to approximately 55 GB and a rotated messages log reached approximately 40 GB, eventually filling the disk and causing WHM/cPanel services to fail.

The logs repeatedly contained errors similar to:

failed to gather metrics: collected metric “node_filesystem_size_bytes” … was collected before with the same name and label values

The repeated mountpoints were under:
/usr/share/cagefs-skeleton/

This appears related to CloudLinux CageFS bind mounts interacting with the filesystem collector in do-agent 3.18.12.

Findings:

* Disabling do-agent immediately stopped the log flood and CPU usage dropped back to normal.
* Re-enabling the agent with:
–no-collector.filesystem
allows the agent to run normally.
* Attempting to use:
–collector.filesystem.mount-points-exclude
fails because the agent reports:
“–collector.filesystem.ignored-mount-points and –collector.filesystem.mount-points-exclude are mutually exclusive”

Impact:

* ~95 GB of runaway logs
* sustained high CPU
* filesystem exhaustion
* WHM/cPanel service interruption

Request:
Can you advise the correct supported method to exclude CageFS mountpoints from filesystem collection in do-agent 3.18.12, or confirm whether this is a known regression?

Image 23. Excessive CPU
Image 24. Storage Full
Image 25. Duplicate Stats on CageFS
Image 26. Mutually Exclusive Mount Point Configuration

At this point, as a secretary, I needed to run to the bank to meet a treasurer to open a new business account. I had addressed the problem, found a workaround, and informed the package’s authors. After the visit to the bank, I had lunch with a group of people, headed off to a client, then visited another client, and then another client who happened to be visiting before heading home. I checked the status of the ticket and was pleased with it.

DigitalOcean support responded quickly and escalated the issue to engineering.

Their response acknowledged that the issue appeared related to:

  • the filesystem collector
  • interacting with CageFS mount points
Digital Ocean Response
do-agent 3.18.12 causes runaway logging and high CPU on CloudLinux CageFS Droplet

April 28, 2026 12:43 PM

Hello,

Thank you for contacting DigitalOcean. Hope you are doing well.

Based on the behaviour you’ve described, it does appear that the issue is related to the filesystem collector in do-agent 3.18.12 interacting with CageFS mount points. 

I’ve escalated this issue to our Engineering team and am currently awaiting their update. I’ll let you know as soon as we hear back.

We appreciate your patience and understanding in this regard. 

Swimmingly,
Anoop V
Senior Cloud Support Engineer
DigitalOcean

—–
We value your feedback! After your ticket closes we will send you an email survey, please let us know how we did.

Click here to learn more about how Paperspace can help you build production-ready AI/ML applications with GPUs!

Click to update your account information or your payment method
Here’s a link to our helpful documentation
Need faster support? Check out our DigitalOcean Support Plans
—–

ref:!00Df2018t5m.!500QP01QvKFR:ref

Lessons Learned

Several important lessons came out of this incident.

1. Elevated CPU Without Obvious Cause Matters

The server remained operational for days while CPU usage stayed abnormally high.

That should have been treated as a stronger warning sign earlier.

2. “Disk Usage” Alerts May Point to Symptoms, Not Root Cause

The alerts referenced /var/log/apache2/domlogs but the true issue was /var/log/messages

Monitoring systems often associate warnings with monitored paths rather than the actual largest files.

3. SSH Access Can Save a Server

Even after websites, WHM, and cPanel failed, SSH still worked. That terminal access made the entire recovery possible.

4. Log Storms Can Become Resource Storms

The issue was not just disk usage.

The constant logging activity also generated:

  • sustained CPU load
  • inode pressure
  • filesystem overhead
  • service instability
  • queue failures
  • degraded cPanel performance

5. CloudLinux CageFS Creates Unique Monitoring Challenges

The repeated references to /usr/share/cagefs-skeleton strongly suggest that monitoring tools not explicitly designed for CageFS environments may encounter duplicate filesystem metric problems.

6. Backblaze Unaffected

Backblaze, a remote service I use to store off-site backups, was unaffected. The JetBackup plugin does not back up the entire disk or the log files. It backs up only the user accounts and configuration data necessary for WHM/cPanel. If it had backed up the entire disc remotely, there would have been a financial impact on both the transfer and storage to the remote service.

7. No VPS Backup Impact

The entire image of the VPS is backed up on Digital Ocean’s servers daily for 7 days, and would have been considered if SSH was not possible, the cause could not be diagnosed, or the system could not be brought back to a normal state. The financial cost of the backup is plan-based on a fixed fee of $9.60 per month, rather than usage-based at $0.03/GiB.

Possible Cost Savings

This exercise identified an opportunity to assess whether high costs can be reduced. I worked out the estimated costs for normal usage, maximum usage, and 4 days at maximum capacity. Under normal usage, it works out to around $5.25 per month, and $25.20 if the full space is consumed. This hiccup would have added $2.56, bringing the total to $7.81 – still significantly under the $9.60 monthly plan.

Normal Usage:
25 GB file system * 7 day retention = 175 GB retained
175 GB * $0.03 = $5.25 per month ($0.17/day)

Max Usage:
120 GB file system * 7 day retention = 840 GB retained
840 GB * $0.03 = $25.20 per month ($0.81/day)

4 Day Log Storm:
$0.81 max usage – $0.17 normal usage = $0.64 per day
$0.64 per day * 4 days = $2.56
$5.25 + $2.56 = $7.81

Daily Backup Plan Threshold Savings:
$9.60 Plan / $0.03 GiB = 320 GB retained
320 GB / 7-day retention = 45.7 GB usage
45.7 GB / 120 GB = 38% capacity

At almost half the cost for normal usage, it seems lucrative to make the switch. However, my goal is to grow. Once the data on the server exceeds 38% (45 GB) capacity, usage-based backups exceed the plan’s $9.60 cost limit.

Image 27. Backup Plans
Image 28. Backup Costs on Usage

Looking at the Usage-Based backups, there are different backup frequencies to choose from, ranging from every 4 hours to weekly, which affect the overall price. Retention can be set from 7 to 30 days. The flexibility is quite interesting and would become more important later on if the need for the latest data becomes critical, especially when a website or data is updated frequently.

One small frustration is that the daily backups do not specify their size or show charts. The activity reports that the backup execution went from 45 seconds to 1 minute and 20 seconds. I’m not in a position to see any data in the VPS insight dashboard once the metrics were cleaned during recovery, so I’m now unaware of what the hard drive displayed in the lead-up to the failure.

Image 29. Droplet Backups & Snapshots
Image 30. Droplet Activity
Image 31. Digital Ocean Droplet Insights

Final State

After recovery:

  • CPU returned to normal
  • disk usage stabilized
  • WHM/cPanel recovered
  • websites resumed functioning
  • backups remained intact
  • and monitoring stabilized with filesystem collection disabled

The issue appears strongly tied to:

  • DigitalOcean do-agent 3.18.12
  • filesystem metric collection
  • CloudLinux CageFS mount structures

At the moment, the server remains stable using --no-collector.filesystem while awaiting further feedback from DigitalOcean engineering.

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Discover more from Lewis Moten

Subscribe now to keep reading and get access to the full archive.

Continue reading