warn about low disk space

[garrett]

Origin story

When talking with a friend who was using Cockpit on a test machine, he noticed things weren't working correctly. After poking around, he discovered that there was no free space left on the system (at least in /). Once that was fixed, things worked as expected.

Details

When storage is completely full, all sorts of problems start happening. Files can't be saved, processes stop working reliably, etc.

Even when space is "low", there are issues — filesystems slow down, heavy fragmentation occurs (even on fragmentation-resistant filesystems), and some file save actions do not work (if the size exceeds the space).

There are several reasons why space can be an issue:

• PackageKit not cleaning up after itself
• ever-growing logs (especially when there's a flood of entries)
• old, small disks
• not enough enough space allocated (either on bare metal's partitions, or in a VM's config)

While Cockpit isn't a janitor, it should report something as important as emergency levels of free space as a warning.

Open questions

So, now the questions are:

• How little is too little? How do we determine when to warn?
• How would this work with VMs? and Containers?
  • (Read-only disk images and partitions should be exempt)
• Can we provide suggestions?
• What actions can we perform?
  • Vacuum journal
  • Remove old logs
  • Clean system caches
  • Provide suggestions (mainly based on which filesystems are full)

[davidmalcolm]

Is the question "what's using all the space?" in-scope for Cockpit? On the desktop I'd launch
Baobab, but I'm guessing that having something similar in cockpit would be feature creep.

(edited: I just hit this; my laptop just ran out of disk space. It turned out to be 25GB taken by a "mock" build in a subdir below /var/lib/mock - which wouldn't have been my first guess)

[garrett]

I meant "What's using all the space?" with a few heuristics, not a filesystem-wide scan.

I've found it's almost always /var/logs/, something in /var/cache/ (usually PackageKit, which needs to be fixed to not keep around so much cache), or system-wide VMs or containers (which Cockpit already knows about) that eat up space in the root filesystem.

I should've specified that it's not a server Baobab — thanks for bringing it up!

[martinpitt]

We have a notification mechanism now, and our front page redesign has this kind of "system health", so this gets into scope again.

[marusak]

I am a bit on fence here.
I think having server "Baobab"-kind-of-tool would be very useful. That is however for a separate page (from starter kit) that can be preloaded and post status if some partition is running out of space.
For me when I used to have storage issues always worked top-down approach with listing directory (starting in /) and identifying which sub-directories are the biggest ones and then go into those and repeat until I got to something that I knew I could remove (like packagekit cache, journal logs, old VMs, some SW I was not using anymore...)

I find suggested "heuristics" to be a somewhat shoot in the dark. Sure, on our locale machines it is usually journal or packagekit, but on real servers you are probably going to deal with running apps data, containers and such which can have non-standard locations.

On metrics page we show disks utilization which can help server users and for filesystem/directory utilization I would suggest just having separate page (that I don't think we will get to any time soon).

[garrett]

I find suggested "heuristics" to be a somewhat shoot in the dark. Sure, on our locale machines it is usually journal or packagekit, but on real servers you are probably going to deal with running apps data, containers and such which can have non-standard locations.

Right, but every system will have logs, package caches, and other similar items. On a first pass, we could definitely handle things that affect all systems.

These things happen to be isolated from the task at hand, and are more invisible. Someone dealing with their own custom storage will have set that up and will be working with that more on a day-to-day basis.

Nobody sets up logs, manually downloads packages, and so on. These are things a system does and there should be a way to gain insight on them and clear up any issues with them.

As far as storage space is concerned, we do have easy ways of showing how full a volume is. I know for a fact that df is super-fast, so it's possible to get this information on a volume-level basis. Doing something with it is a bit different. A baobab-like starter kit page could have a root passed to it to scan that volume, for example. Same for low disk space warnings.

But, again, all of that is great, but out of scope of this issue. This issue is about system-level stuff that can be done to fix issues and ease burdens. It's not about someone cleaning up their data storage for their app. Nor is it about scanning a filesystem for large sources (although, if that existed, it could be an optional action to take).

[davidmalcolm]

At risk of scope-creep again: is a server more likely to run into the classic time-sharing setup of multiple users, and one of the users is hogging all the space? I can imagine a view showing user's home directory sizes, how big their quotas are, and how much of their quotas they're using. Things like: "you don't have quotas set up, do you want to do that?" etc. (I'm just lurking here; not sure to what extent this reflects the goals of the project)

[martinpitt]

I know for a fact that df is super-fast, so it's possible to get this information on a volume-level basis.

The metrics page is doing exactly that, FTR, and shows you red bars for the mounts that are low on space.

Indeed I would start with showing an alert in the Overview's Health box. That does not need a lot of guesswork, and is already a good indicator for "hey, your machine is going to act up soon". Implementing a file browser etc. is a huge pile of work for comparatively little gain, and I don't want to do that. Some standard actions like "clear /var/cache", "vacuum the journal" and so on are fine of course, and we can add more of them as we go. But even just being told that there is an upcoming problem is the biggest win.