Still Blind After the Log Storm: When “Unsupported” Isn’t the End of the Story
In April, I ran into a server problem that turned a monitoring tool into part of the outage.
The short version is this: the DigitalOcean monitoring agent began producing repeated filesystem metric errors on a CloudLinux/cPanel server using CageFS. The logs grew until /var/log consumed roughly 96 GB, CPU stayed high, and WHM/cPanel services became unavailable.
I wrote about the original incident here:
Death by Metrics: DigitalOcean Monitoring Agent, CageFS, and a 95 GB Log Storm
That earlier post explains the failure itself: the full disk, the log flood, the duplicate node_filesystem_* errors, and the CageFS paths that kept appearing in the logs.
This post is about what happened afterward.
A month later: unsupported
I opened a DigitalOcean support ticket in April. The first response seemed to recognize the issue as an interaction between the do-agent filesystem collector and CageFS mount points. The ticket was escalated to Engineering.
Then I waited.
At the end of May, about a month after I reported the incident, I received the final answer: CloudLinux/CageFS environments are not supported by the DigitalOcean Agent.
That may be technically true. I understand that companies draw support boundaries. CloudLinux and CageFS create unusual filesystem layouts, and I am not asking DigitalOcean to provide full support for every possible server configuration.
But the answer still left me with a practical problem.
My only safe workaround was to disable the filesystem collector entirely.
/opt/digitalocean/bin/do-agent --syslog --no-collector.filesystem
That stopped the dangerous behavior, but it also removed the filesystem metrics I needed most.
After an outage involving disk exhaustion, being blind to disk usage is not exactly reassuring.
Support Ticket #12093409 Thread
#12093409 do-agent 3.18.12 causes runaway logging and high CPU on CloudLinux CageFS Droplet
Type: Computer; Sub-Type: Droplet; Tertiary Type: Graphs & Monitoring; Created Apr 28, 2026, 10:40 AM; Last Updated: Jun 03, 2026, 10:32 AM
Initial Report: April 28, 2026, 10:40 AM
Environment:
* CloudLinux + cPanel/WHM
* CageFS enabled
* DigitalOcean Droplet
* do-agent upgraded automatically from 3.18.10-1 to 3.18.12-1 on 2026-04-24 around 03:49 UTC
Issue:
Immediately after the upgrade, CPU usage rose from normal levels to ~75% continuously. /var/log/messages grew to approximately 55 GB and a rotated messages log reached approximately 40 GB, eventually filling the disk and causing WHM/cPanel services to fail.
The logs repeatedly contained errors similar to:
failed to gather metrics: collected metric “node_filesystem_size_bytes” … was collected before with the same name and label values
The repeated mountpoints were under:
/usr/share/cagefs-skeleton/
This appears related to CloudLinux CageFS bind mounts interacting with the filesystem collector in do-agent 3.18.12.
Findings:
* Disabling do-agent immediately stopped the log flood and CPU usage dropped back to normal.
* Re-enabling the agent with:
–no-collector.filesystem
allows the agent to run normally.
* Attempting to use:
–collector.filesystem.mount-points-exclude
fails because the agent reports:
“–collector.filesystem.ignored-mount-points and –collector.filesystem.mount-points-exclude are mutually exclusive”
Impact:
* ~95 GB of runaway logs
* sustained high CPU
* filesystem exhaustion
* WHM/cPanel service interruption
Request:
Can you advise the correct supported method to exclude CageFS mountpoints from filesystem collection in do-agent 3.18.12, or confirm whether this is a known regression?
Escalated to the Engineering team: April 28, 2026, 12:43 PM
Hello,
Thank you for contacting DigitalOcean. Hope you are doing well.
Based on the behaviour you’ve described, it does appear that the issue is related to the filesystem collector in do-agent 3.18.12 interacting with CageFS mount points.
I’ve escalated this issue to our Engineering team and am currently awaiting their update. I’ll let you know as soon as we hear back.
We appreciate your patience and understanding in this regard.
Swimmingly,
Anoop V
Senior Cloud Support Engineer
DigitalOcean
Still Investigating: April 30, 2026, 11:01 AM
Hello Lewis,
Thank you for your patience. Our engineering team is still investigating this issue. I’ll continue to monitor them and update you as things progress.
Warm Currents,
Whitney J
Still Investigating: May 19, 2026, 3:03 AM
Hello Lewis,
Thank you for waiting patiently. I sincerely apologize for the delay here.
Our Engineering team is still reviewing the issue and we are monitoring the same internally. I would request you to kindly hold on for sometime. We will get back to you as soon as we have more updates.
Your patience is highly appreciated.
Swimmingly,
DipankarC
Senior Cloud Support Engineer II
DigitalOcean
Unsupported: May 29, 2026, 9:32 AM
Hello,
Thank you for your patience. We have an update from our Engineering team.
Please note that the DigitalOcean Agent is not supported on CloudLinux CageFS environments, which is causing the runaway logging and high CPU usage you are experiencing. You can find the list of officially supported operating systems and distributions directly in the DigitalOcean Agent GitHub README.
Please feel free to let us know if you have any other questions. We’re here and happy to help you anytime.
Swimmingly,
Anoop V
Senior Cloud Support Engineer
DigitalOcean
Feature Request: June 3, 2026, 10:32 AM
Hello,
I understand that CloudLinux/CageFS is not officially supported by the DigitalOcean Agent. I wanted to suggest a possible feature that may help customers with unsupported or unusual filesystem layouts without requiring DigitalOcean to fully support those environments.
Would DigitalOcean consider adding an alternative filesystem collector mode in which the do-agent can read filesystem metrics from a local file rather than walking the mounted filesystems directly?
For example:
–collector.filesystem.source=file
–collector.filesystem.file=/var/lib/do-agent/filesystem.prom
In this model, do-agent would remain the trusted process that sends metrics to DigitalOcean, but customers could provide sanitized filesystem metrics generated by their own script or monitoring tool. The file could use a documented Prometheus-style text format or another simple documented schema.
This would allow customers with CloudLinux CageFS, cPanel/WHM, heavy bind mounts, containers, chroots, or other unusual mount layouts to avoid duplicate filesystem metrics and runaway logging, while still reporting basic filesystem usage to DigitalOcean.
A file-based approach may also be safer than an exec-based plugin, since the agent would not need to run customer-provided commands. Customers would be responsible for generating the file, and the agent would read it only.
The original issue I experienced was that after the do-agent update from 3.18.10-1 to 3.18.12-1, the filesystem collector encountered CageFS mountpoints and repeatedly logged duplicate node_filesystem_size_bytes metric errors. This resulted in sustained high CPU usage, approximately 95 GB of logs, disk exhaustion, and WHM/cPanel service interruption.
Even if CloudLinux remains unsupported, an alternative filesystem metric source could provide a safer escape hatch for customers who otherwise must disable the filesystem collector entirely or uninstall the agent.
Thank you for considering it.
The server was not unable to report disk space
What bothered me was that the server itself could still show disk space just fine.
From the terminal, these commands worked:
df -hdf -ih
The output gave a clean and useful view of free space and inode usage. It showed the main filesystems I cared about: /, /boot, /boot/efi, and /tmp.


So the issue was not that the server was incapable of reporting disk usage.
The issue appeared to be that the DigitalOcean agent’s normal filesystem collector was walking and interpreting the mount layout in a way that encountered CageFS bind mounts and duplicate filesystem metrics.
That distinction matters.
If a basic terminal command can safely show useful disk information, then maybe the agent should have a simpler fallback mode that does not require full mount-table discovery.
Unsupported should not mean blind

I am not asking DigitalOcean to fully support CloudLinux, CageFS, cPanel, or every unusual filesystem layout.
I am asking for safer fallback options.
The current choice appears to be:
- Leave filesystem collection enabled and risk duplicate metric errors, runaway logging, high CPU, and disk exhaustion; or
- Disable filesystem collection and lose DigitalOcean disk/inode metrics entirely.
That is a bad tradeoff.
Unsupported may mean “we cannot troubleshoot every detail of this environment.” That is fair.
But unsupported should not have to mean “your only safe choice is to turn off the very metrics that would warn you about disk exhaustion.”
The feature request
Instead of only complaining about the support answer, I opened a feature request:
The request proposes three possible approaches.
1. A df-based filesystem collector mode
The simplest option would be for the agent to collect disk and inode metrics using something equivalent to:
df -Pdf -Pi
That would provide the same practical information administrators already trust from the terminal:
- total space;
- used space;
- available space;
- percent used;
- total inodes;
- used inodes;
- available inodes;
- inode percent used;
- filesystem/device;
- mountpoint.
This would not require the agent to reason through every bind mount or virtualized filesystem path.
2. An explicit path mode
Another option would let administrators tell the agent exactly which paths matter.
For example:
--collector.filesystem.paths=/,/boot,/boot/efi,/tmp
That would allow the agent to collect metrics only for those paths, using something like df or statvfs internally.
For many servers, this is enough. I do not need DigitalOcean to understand every CageFS bind mount. I need to know whether /, /boot, /boot/efi, or /tmp is running out of space.
3. A customer-provided metrics file
The most flexible option would let the administrator generate filesystem metrics however they prefer and write them to a local file.
For example:
--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
The agent would remain the trusted process that submits metrics to DigitalOcean, but the customer would control how the filesystem numbers are gathered.
That matters because there may be a trust issue with allowing customers to push arbitrary filesystem metrics directly into DigitalOcean’s API. From what I can tell, DigitalOcean’s monitoring API is mainly for reading metrics and managing alert policies, not for customers to submit replacement disk metrics themselves.
That makes sense. Allowing direct metric submission would raise questions about validation, trust, and abuse.
But a local file mode would be different. The do-agent would still be the thing sending metrics to DigitalOcean. It would simply read sanitized filesystem numbers that the administrator generated locally.
Why this is not just about CloudLinux
CloudLinux/CageFS is the example that exposed the problem for me, but the larger issue is not limited to CloudLinux.
Any environment with unusual mount behavior could run into similar problems:
- cPanel/WHM systems;
- chroot-heavy systems;
- bind-mount-heavy systems;
- container-heavy hosts;
- Docker or LXC hosts;
- systems with duplicated or virtualized mountpoints.
The more complex the mount layout, the more useful a simple fallback becomes.
Sometimes an administrator does not need a monitoring agent to understand every mountpoint. Sometimes they just need the same answer df already gives.
The support answer was technically clear, but operationally incomplete
I do appreciate that DigitalOcean eventually gave a clear support boundary: CloudLinux/CageFS is not supported by do-agent.
But it took about a month to get there, and the answer did not leave me with a safe way to keep disk metrics visible in DigitalOcean.
That is the part that still feels unresolved.
The issue was not simply that an unsupported configuration had a minor bug. The agent produced a severe failure mode: repeated logging, high CPU, and disk exhaustion. The only workaround that stopped it also removed the filesystem visibility I needed after the outage.
That is why the feature request matters.
It is not asking DigitalOcean to support my exact setup. It is asking for a safer middle ground between “full filesystem collector” and “no filesystem metrics at all.”
Even one fallback option would help.
What I hope changes
I hope DigitalOcean considers adding one or more safer filesystem collection modes:
--collector.filesystem.mode=df
--collector.filesystem.paths=/,/boot,/boot/efi,/tmp
--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
I would also like to see the agent handle repeated collector failures more defensively. A metrics collector should not be able to fill /var/log/messages until the server runs out of space.
If a collector starts producing the same duplicate metric error repeatedly, the agent could rate-limit the message, disable only the affected collector, and emit one clear warning.
Monitoring should reduce surprises, not create them.
Closing thought
The original incident taught me that observability tools can become part of the outage.
The support follow-up taught me something else: even when a platform is unsupported, there may still be a reasonable product improvement hiding inside the failure.
I do not need DigitalOcean to support every detail of CloudLinux/CageFS.
I would just like a way to safely report basic disk usage without having to ask the agent to navigate a filesystem maze.
The following is the support ticket thread, included for context.
Feature request: https://github.com/digitalocean/do-agent/issues/357
The issue asks DigitalOcean to consider alternate filesystem collection modes for do-agent, including:
df-based collection;- explicit path-based collection;
- customer-provided filesystem metrics file;
- safer handling of repeated filesystem collector failures.
GitHub @digitalocean/do-agent#357
digitalocean / do-agent
Issue #357
Feature request: safer alternate filesystem collection modes for do-agent
June 3, 2026, 11:09 AM
I would like to request one or more alternate filesystem collection modes for do-agent that do not require the agent to walk and interpret every mounted filesystem on the host.
Some environments have unusual, duplicated, virtualized, or bind-mounted filesystem layouts. In those environments, the current filesystem collector can encounter duplicate or misleading mount data. A safer alternative would allow administrators to collect only the basic disk and inode metrics they actually need.
This request is not for full support of every unusual environment. Instead, it is a request for safer fallback collection modes that could help administrators avoid full mount-table discovery when it is not appropriate for their server.
The three possible approaches are:
- A
df-based filesystem collector mode; - An explicit path-based filesystem collector mode;
- A customer-provided filesystem metrics file.
Any one of these would help. Supporting more than one would give administrators flexibility.
Proposed option 1: df-based filesystem collector mode
Please consider adding a filesystem collector mode that gathers disk and inode metrics using the equivalent of:
df -P df -Pi
Possible option names:
--collector.filesystem.mode=df
or:
--collector.filesystem.use-df
When enabled, do-agent would collect filesystem space and inode metrics using df-style output instead of walking and interpreting the full mount table through the current filesystem collector.
This would provide the same type of filesystem information administrators already trust from the terminal:
- total space;
- used space;
- available space;
- percent used;
- total inodes;
- used inodes;
- available inodes;
- inode percent used;
- filesystem/device;
- mountpoint.
Possible advanced options:
--collector.filesystem.df-path=/usr/bin/df
--collector.filesystem.df-args="-P"
--collector.filesystem.df-inode-args="-Pi"
If parsing fails, the agent could disable only the df filesystem collector and emit a single warning, rather than repeatedly logging the same failure.
Why a df-based mode may be enough
On the affected server, standard df output provides a clean and practical filesystem view without exposing the large CageFS bind-mount layout that caused problems for the normal collector.
Example df -h output:
Filesystem Size Used Avail Use% Mounted ondevtmpfs 4.0M 0 4.0M 0% /devtmpfs 1.8G 0 1.8G 0% /dev/shmtmpfs 732M 74M 658M 11% /runtmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup/dev/vda1 120G 30G 90G 26% //dev/vda3 507M 316M 191M 63% /boot/dev/vda2 200M 7.5M 193M 4% /boot/efi/dev/loop0 3.9G 204K 3.7G 1% /tmpnone 1.8G 4.0K 1.8G 1% /var/lve/dbgovernor-shm
Example df -ih output:
Filesystem Inodes IUsed IFree IUse% Mounted ondevtmpfs 447K 344 447K 1% /devtmpfs 457K 1 457K 1% /dev/shmtmpfs 800K 924 800K 1% /runtmpfs 1.0K 18 1006 2% /sys/fs/cgroup/dev/vda1 60M 663K 60M 2% //dev/vda3 256K 327 256K 1% /boot/dev/vda2 0 0 0 - /boot/efi/dev/loop0 256K 49 256K 1% /tmpnone 457K 2 457K 1% /var/lve/dbgovernor-shmtmpfs 92K 22 92K 1% /run/user/1002
This suggests that the issue is not that filesystem usage cannot be reported on this host. The issue is that the current collector appears to inspect the mount layout in a way that encounters CageFS bind mounts and duplicate filesystem metrics.
A df-based fallback mode could collect the basic disk and inode information administrators already use from the terminal, while avoiding deeper mount-table discovery.
For many servers, this would be sufficient. In this example, the useful monitored filesystems would primarily be:
//boot/boot/efi/tmp
and possibly /var/lve/dbgovernor-shm only if the administrator chooses to include tmpfs-style filesystems.
The agent could optionally ignore common virtual filesystems by default, such as:
devtmpfstmpfscgroupcgroup2procsysfsdebugfstracefsoverlaysquashfs
This would give do-agent a safer fallback for unusual mount layouts without requiring full CloudLinux/CageFS support.
Proposed option 2: explicit path-based filesystem checks
Please consider an option that collects filesystem metrics only for specific administrator-provided paths.
For example:
--collector.filesystem.paths=/,/boot,/boot/efi,/tmp
or:
--collector.filesystem.paths-file=/etc/do-agent/filesystem-paths.conf
Example paths file:
//boot/boot/efi/tmp
When this option is used, do-agent would skip full mountpoint discovery and collect filesystem metrics only for the listed paths.
The behavior could be similar to running:
df -P / df -P /boot df -P /boot/efi df -P /tmp df -Pi / df -Pi /boot df -Pi /boot/efi df -Pi /tmp
or using equivalent statfs / statvfs calls internally.
This would allow administrators to say:
Only report disk and inode usage for these important paths.
That is often all that is needed for practical alerting.
This would also avoid requiring administrators to craft complex mountpoint exclusion regular expressions for bind-mount-heavy systems.
Proposed option 3: customer-provided filesystem metrics file
Please also consider allowing do-agent to read filesystem metrics from a local file.
For example:
--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
or:
--collector.filesystem.source=file --collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
In this model, the customer could generate the file however they prefer:
df;stat;- a shell script;
- a cron job;
- a monitoring tool;
- a custom parser with environment-specific exclusions.
do-agent would remain the trusted process that submits metrics to DigitalOcean, but the customer would control how filesystem metrics are gathered.
A file-based approach may be safer than an exec-based plugin because do-agent would not need to run arbitrary customer commands. It would only read a documented local file format.
Example conceptual format:
mountpoint=/ size_bytes=128849018880 used_bytes=32212254720 avail_bytes=96636764160 used_percent=26 inode_total=62914560 inode_used=663000 inode_avail=62251560 inode_used_percent=2mountpoint=/boot size_bytes=531628032 used_bytes=331350016 avail_bytes=200278016 used_percent=63 inode_total=262144 inode_used=327 inode_avail=261817 inode_used_percent=1mountpoint=/tmp size_bytes=4187593113 used_bytes=208896 avail_bytes=3972844748 used_percent=1 inode_total=262144 inode_used=49 inode_avail=262095 inode_used_percent=1
Or, if preferred, the file could use a documented Prometheus-style text format.
Why this is useful
Some environments have mount tables that are technically valid but difficult for a general-purpose filesystem collector to interpret safely.
Examples include:
- CloudLinux CageFS;
- cPanel/WHM systems;
- chroot-heavy systems;
- bind-mount-heavy systems;
- container-heavy hosts;
- Docker/LXC environments;
- systems with duplicated or virtualized mountpoints.
In these environments, the administrator may not need the agent to understand every mountpoint. They may only need reliable metrics for a few filesystems or paths, such as:
//boot/boot/efi/tmp
A df-style mode, explicit path mode, or customer-provided file mode would avoid unnecessary full mount discovery and reduce the risk of duplicate filesystem metrics.
Example use case: CloudLinux / CageFS / cPanel
I understand that CloudLinux/CageFS is not officially supported by do-agent. This feature request is not asking for full CloudLinux support.
However, this environment is a good example of why safer alternate filesystem collection modes would be useful.
Environment:
- DigitalOcean Droplet;
- CloudLinux + cPanel/WHM;
- CageFS enabled;
do-agentupgraded automatically from3.18.10-1to3.18.12-1;- Upgrade occurred around
2026-04-24 03:49 UTC.
After the upgrade, the filesystem collector began repeatedly logging duplicate metric errors related to CageFS bind mounts.
The repeated mountpoints were under:
/usr/share/cagefs-skeleton/
The logs repeatedly contained errors similar to:
failed to gather metrics: collected metric "node_filesystem_size_bytes" ... was collected before with the same name and label values
The impact was significant:
- sustained high CPU usage, around 75%;
- approximately 55 GB
/var/log/messages; - approximately 40 GB rotated messages log;
- disk exhaustion;
- WHM/cPanel service interruption.
Disabling do-agent immediately stopped the log flood and CPU returned to normal.
In this case, I did not need the agent to inspect CageFS mountpoints. I only needed basic disk and inode metrics for the main filesystems. Commands such as the following were sufficient to show the information I needed:
df -P df -Pi
or, for specific paths:
df -P / df -P /boot df -P /boot/efi df -P /tmp df -Pi / df -Pi /boot df -Pi /boot/efi df -Pi /tmp
Current workaround
The only safe workaround I currently have is to disable the filesystem collector entirely:
/opt/digitalocean/bin/do-agent --syslog --no-collector.filesystem
That prevents the runaway filesystem collector behavior, but it also removes the DigitalOcean filesystem metrics I actually need for this Droplet.
This creates an unfortunate tradeoff:
- leave filesystem collection enabled and risk duplicate metric errors, runaway logging, high CPU usage, and disk exhaustion;
- disable filesystem collection and lose the disk/inode metrics that would help detect or prevent disk exhaustion.
A safer alternate collection mode would avoid this tradeoff by allowing do-agent to report basic filesystem usage without walking the full mount layout.
Why mountpoint exclusion rules are not always enough
Mountpoint exclusion rules are useful, but they still require the agent to discover and reason about the host’s mount layout.
In bind-mount-heavy or CageFS-style environments, that discovery process can be fragile. Administrators may also have to write complex regular expressions to exclude paths the agent did not need to inspect in the first place.
A df-based, path-based, or file-based mode would be simpler and more predictable:
- do not walk every mountpoint;
- do not inspect CageFS bind mounts unnecessarily;
- do not require complex mountpoint exclusion regular expressions;
- collect only the filesystems or paths the administrator explicitly cares about;
- allow administrators to generate clean filesystem metrics themselves when needed.
Requested features
Please consider adding one or more of the following options.
df-based mode
--collector.filesystem.mode=df
or:
--collector.filesystem.use-df
This would collect filesystem space and inode metrics using the equivalent of df -P and df -Pi.
Explicit path mode
--collector.filesystem.paths=/,/boot,/boot/efi,/tmp
or:
--collector.filesystem.paths-file=/etc/do-agent/filesystem-paths.conf
This would collect filesystem metrics only for explicitly configured paths.
Customer-provided file mode
--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
or:
--collector.filesystem.source=file --collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt
This would allow customers to generate filesystem metrics themselves and let do-agent read and submit them.
Additional defensive behavior
Even when an environment is unsupported, it may also be helpful for the agent to handle repeated filesystem collector failures more defensively.
For example:
- rate-limit repeated duplicate metric errors;
- disable only the affected collector after repeated failures;
- emit one clear warning instead of repeatedly logging the same error;
- avoid filling system logs when the metrics collector is unhealthy.
A metrics issue should not be able to fill /var/log/messages, exhaust disk space, and contribute to a production service outage.
Related issues
This request may also help with or relate to other reports involving CloudLinux support, duplicate metric collection, or high CPU from filesystem metric collection:
- [CloudLinux] This script does not support the OS/Distribution on this machine #129 — CloudLinux install/support issue for CloudLinux + WHM/cPanel.
- Support to Cloudlinux droplet-agent#131 — request for CloudLinux support.
- Unable to gather advanced metrics, ERROR “was collected before with the same name and label values” #228 — duplicate metric errors with “was collected before with the same name and label values.”
- do-agent process constant high CPU usage #233 — high CPU and repeated duplicate
node_filesystem_*metric errors.
This feature request is more specific: provide safer alternate filesystem collection modes, such as df-based collection, explicit path collection, or customer-provided filesystem metrics, so that users do not have to choose between unsafe full mount discovery and disabling filesystem metrics entirely.
Trouble Ticket
I also opened a DigitalOcean Support ticket for this incident in April. Support confirmed that CloudLinux/CageFS is not officially supported by do-agent.
This request is not for full CloudLinux support, but for a safer alternative filesystem collection mode that could help unsupported or unusual mount layouts avoid a full mount-table discovery.
[#12093409](https://cloudsupport.digitalocean.com/s/case-detail?recordId=500QP00001QvKFRYA3) do-agent 3.18.12 causes runaway logging and high CPU on CloudLinux CageFS Droplet
June 3, 2026, 11:12 AM
Adding one more piece of context: I had previously written up the incident in more detail here:
The post includes the timeline, symptoms, recovery steps, log growth, and examples of the duplicate node_filesystem_* metric errors involving CageFS paths.
I’m sharing it only as additional background. The main request here remains the same: a safer alternate filesystem collection mode so users do not have to choose between disabling filesystem metrics entirely and risking full mount-table discovery in unusual or unsupported environments.
