Unsupported Should Not Mean Blind

Still Blind After the Log Storm: When “Unsupported” Isn’t the End of the Story

In April, I ran into a server problem that turned a monitoring tool into part of the outage.

The short version is this: the DigitalOcean monitoring agent began producing repeated filesystem metric errors on a CloudLinux/cPanel server using CageFS. The logs grew until /var/log consumed roughly 96 GB, CPU stayed high, and WHM/cPanel services became unavailable.

I wrote about the original incident here:

Death by Metrics: DigitalOcean Monitoring Agent, CageFS, and a 95 GB Log Storm

That earlier post explains the failure itself: the full disk, the log flood, the duplicate node_filesystem_* errors, and the CageFS paths that kept appearing in the logs.

This post is about what happened afterward.

A month later: unsupported

I opened a DigitalOcean support ticket in April. The first response seemed to recognize the issue as an interaction between the do-agent filesystem collector and CageFS mount points. The ticket was escalated to Engineering.

Then I waited.

At the end of May, about a month after I reported the incident, I received the final answer: CloudLinux/CageFS environments are not supported by the DigitalOcean Agent.

That may be technically true. I understand that companies draw support boundaries. CloudLinux and CageFS create unusual filesystem layouts, and I am not asking DigitalOcean to provide full support for every possible server configuration.

But the answer still left me with a practical problem.

My only safe workaround was to disable the filesystem collector entirely.

/opt/digitalocean/bin/do-agent --syslog --no-collector.filesystem

That stopped the dangerous behavior, but it also removed the filesystem metrics I needed most.

After an outage involving disk exhaustion, being blind to disk usage is not exactly reassuring.

Support Ticket #12093409 Thread
#12093409 do-agent 3.18.12 causes runaway logging and high CPU on CloudLinux CageFS Droplet

Type: Computer; Sub-Type: Droplet; Tertiary Type: Graphs & Monitoring; Created Apr 28, 2026, 10:40 AM; Last Updated: Jun 03, 2026, 10:32 AM

Initial Report: April 28, 2026, 10:40 AM
Escalated to the Engineering team: April 28, 2026, 12:43 PM
Still Investigating: April 30, 2026, 11:01 AM
Still Investigating: May 19, 2026, 3:03 AM
Unsupported: May 29, 2026, 9:32 AM
Feature Request: June 3, 2026, 10:32 AM

The server was not unable to report disk space

What bothered me was that the server itself could still show disk space just fine.

From the terminal, these commands worked:

df -h
df -ih

The output gave a clean and useful view of free space and inode usage. It showed the main filesystems I cared about: /, /boot, /boot/efi, and /tmp.

Image 1. Free Space
Image 2. iNode Usage

So the issue was not that the server was incapable of reporting disk usage.

The issue appeared to be that the DigitalOcean agent’s normal filesystem collector was walking and interpreting the mount layout in a way that encountered CageFS bind mounts and duplicate filesystem metrics.

That distinction matters.

If a basic terminal command can safely show useful disk information, then maybe the agent should have a simpler fallback mode that does not require full mount-table discovery.

Unsupported should not mean blind

I am not asking DigitalOcean to fully support CloudLinux, CageFS, cPanel, or every unusual filesystem layout.

I am asking for safer fallback options.

The current choice appears to be:

  1. Leave filesystem collection enabled and risk duplicate metric errors, runaway logging, high CPU, and disk exhaustion; or
  2. Disable filesystem collection and lose DigitalOcean disk/inode metrics entirely.

That is a bad tradeoff.

Unsupported may mean “we cannot troubleshoot every detail of this environment.” That is fair.

But unsupported should not have to mean “your only safe choice is to turn off the very metrics that would warn you about disk exhaustion.”

The feature request

Instead of only complaining about the support answer, I opened a feature request:

Feature request: alternate filesystem collector modes for df, explicit paths, or customer-provided files

The request proposes three possible approaches.

1. A df-based filesystem collector mode

The simplest option would be for the agent to collect disk and inode metrics using something equivalent to:

df -P
df -Pi

That would provide the same practical information administrators already trust from the terminal:

  • total space;
  • used space;
  • available space;
  • percent used;
  • total inodes;
  • used inodes;
  • available inodes;
  • inode percent used;
  • filesystem/device;
  • mountpoint.

This would not require the agent to reason through every bind mount or virtualized filesystem path.

2. An explicit path mode

Another option would let administrators tell the agent exactly which paths matter.

For example:

--collector.filesystem.paths=/,/boot,/boot/efi,/tmp

That would allow the agent to collect metrics only for those paths, using something like df or statvfs internally.

For many servers, this is enough. I do not need DigitalOcean to understand every CageFS bind mount. I need to know whether /, /boot, /boot/efi, or /tmp is running out of space.

3. A customer-provided metrics file

The most flexible option would let the administrator generate filesystem metrics however they prefer and write them to a local file.

For example:

--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt

The agent would remain the trusted process that submits metrics to DigitalOcean, but the customer would control how the filesystem numbers are gathered.

That matters because there may be a trust issue with allowing customers to push arbitrary filesystem metrics directly into DigitalOcean’s API. From what I can tell, DigitalOcean’s monitoring API is mainly for reading metrics and managing alert policies, not for customers to submit replacement disk metrics themselves.

That makes sense. Allowing direct metric submission would raise questions about validation, trust, and abuse.

But a local file mode would be different. The do-agent would still be the thing sending metrics to DigitalOcean. It would simply read sanitized filesystem numbers that the administrator generated locally.

Why this is not just about CloudLinux

CloudLinux/CageFS is the example that exposed the problem for me, but the larger issue is not limited to CloudLinux.

Any environment with unusual mount behavior could run into similar problems:

  • cPanel/WHM systems;
  • chroot-heavy systems;
  • bind-mount-heavy systems;
  • container-heavy hosts;
  • Docker or LXC hosts;
  • systems with duplicated or virtualized mountpoints.

The more complex the mount layout, the more useful a simple fallback becomes.

Sometimes an administrator does not need a monitoring agent to understand every mountpoint. Sometimes they just need the same answer df already gives.

The support answer was technically clear, but operationally incomplete

I do appreciate that DigitalOcean eventually gave a clear support boundary: CloudLinux/CageFS is not supported by do-agent.

But it took about a month to get there, and the answer did not leave me with a safe way to keep disk metrics visible in DigitalOcean.

That is the part that still feels unresolved.

The issue was not simply that an unsupported configuration had a minor bug. The agent produced a severe failure mode: repeated logging, high CPU, and disk exhaustion. The only workaround that stopped it also removed the filesystem visibility I needed after the outage.

That is why the feature request matters.

It is not asking DigitalOcean to support my exact setup. It is asking for a safer middle ground between “full filesystem collector” and “no filesystem metrics at all.”

Even one fallback option would help.

What I hope changes

I hope DigitalOcean considers adding one or more safer filesystem collection modes:

--collector.filesystem.mode=df
--collector.filesystem.paths=/,/boot,/boot/efi,/tmp
--collector.filesystem.file=/var/lib/do-agent/filesystem-metrics.txt

I would also like to see the agent handle repeated collector failures more defensively. A metrics collector should not be able to fill /var/log/messages until the server runs out of space.

If a collector starts producing the same duplicate metric error repeatedly, the agent could rate-limit the message, disable only the affected collector, and emit one clear warning.

Monitoring should reduce surprises, not create them.

Closing thought

The original incident taught me that observability tools can become part of the outage.

The support follow-up taught me something else: even when a platform is unsupported, there may still be a reasonable product improvement hiding inside the failure.

I do not need DigitalOcean to support every detail of CloudLinux/CageFS.

I would just like a way to safely report basic disk usage without having to ask the agent to navigate a filesystem maze.

The following is the support ticket thread, included for context.

Feature request: https://github.com/digitalocean/do-agent/issues/357

The issue asks DigitalOcean to consider alternate filesystem collection modes for do-agent, including:

  • df-based collection;
  • explicit path-based collection;
  • customer-provided filesystem metrics file;
  • safer handling of repeated filesystem collector failures.
GitHub @digitalocean/do-agent#357

Discover more from Lewis Moten

Subscribe now to keep reading and get access to the full archive.

Continue reading