system, health: "Janitor"

[garrett]

Cockpit could/should have a system "Janitor" that ties in with the health card.

The main idea is to show an persistent overview of upcoming issues before they hit a critical enough level to display in the health card and to provide a means to address these issues.

It could be a new page, linked from the health card, that would show the amount of available storage — among other things — for critical, growing filesystem points, including:

/
/var/log/
/var/cache/
/tmp/
/home/

(In many systems, these are the same. In some, they're not. Where they're the same, they'd collapse down to /.)

Critically, there should be a way to vacuum logs and set policies to automatically trim the log size.

For manual compacting, it would be the eqivalent of:

journalctl --vacuum-time=20d
journalctl --vacuum-size=500M

For automatic compacting, we could have an interface that adjusts /etc/systemd/journald.conf with flags like SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=

In addition to logs, a janitor could show the size of and clean various caches, /tmp/, and so on.

It should be arranged in a human logical format instead of a filesystem format, so it could have groups on the page that say: "Log storage", "Cache", "Quota" (for users, including implied quotas) and mechanisms to free up space for each.

[garrett]

A lot is kind of storage related, but this is more about the data (and mainly the metadata) than the drives.

[garrett]

All of this already exists in system services, in the form of systemd and journalctl; I'm suggesting that Cockpit provides a friendly interface for these little-known things.

Currently, a lot of people put /var/ and /tmp/ on a separate filesystem to make sure their space do not overflow the root / partition. They shouldn't have to, unless they want to. But they're trying to make sure their disks aren't completely full, as that will cause a lot of very bad things to happen (such as apps not opening properly, files not being written, logs not being added, and other corruption issues.)

It would be ideal if they could set a policy (and share this policy — so perhaps a bit of Ansible and a shell script exporting is desired here too, much like SELinux) and have the system handle it for them.

It's also good if they could clear these items much like cleaning the cache and cookies in a browser.

[marusak]

This is mostly duplicate of #8787
And I feel the same about it. It would be a cool separate page. That can be preloaded and post statuses.
If we just pick subset of locations to watch for we are going to get request for some additional location, then for configs to set up location and of course config for locations to be ignored :)

Closing in favor of #8787 . I realize this one is trying to be more broad, but what it mentions in details is already mentioned in the other issue.

[garrett]

show an persistent overview of upcoming issues before they hit a critical enough level to display in the health card

available storage — among other things — for

Emphasis on the above quotes.

While freeing up space is a big part of this, it's not everything. For example:

• zombie processes
• a user process hogging resources
• constant network saturation

But the point of differentiation between this and the other is: One is proactive, the other is reactive. That is, this would let you clean up messes. The other would let you fix problems when they arise. There's a bit of a difference, even if they overlap.

This issue proactive; it's for an always-available UI for performing system clean-up tasks whenever someone might feel like it.

The other issue is reactive; for "oh crap" moments that arise when there's a problem or when there is very nearly a problem.

If we only had the other issue implemented, there wouldn't be a way to get to the journal vacuuming and other tools. We'd need a persistent UI for this, which is what this issue is for.

[garrett]

I described this in a Jira ticket, speciflcally https://issues.redhat.com/browse/COCKPIT-741.

I posted the first comment, related to showing the disks (especially in error states) @ #8787 (comment)

My second comment, about cleaning up the disks (related to this issue) is, as follows...

---

It'd be great if we had a journal truncation feature when needed.

If /var/ is getting pretty full:

journalctl --disk-usage

The journal can be cleaned up too:

journalctl
  --vacuum-size=BYTES Reduce disk usage below specified size
  --vacuum-files=INT Leave only the specified number of journal files
  --vacuum-time=TIME Remove journal files older than specified time

And /var is where yum/dnf stores package cache:

yum clean {metadata,packages,dbcache,expire-cache,all}

It would be nice to clean PackageKit as well, but I'm not sure if there's a way to instruct it to clean its cache. There is:

sudo pkcon refresh force -c -1

...but that only works for enabled repos and it forces a re-fetch of the metadata. (If repos are disabled, it doesn't work. And re-fetching takes more time.)

This might be growing into a "free up space" app. 😢

I guess we could have a free up space dialog instead where it abstracts a lot of this away. It could look as simple as this:

---

Clean cache and journal

☑ Clear package cache
☑ Trim journal
    [ size dropdown ⏷ ][ time dropdown ⏷ ]

(Clean) Cancel

---

(This should be simple enough, no need for a customizable app or page or whatever.)

Clearing the package cache (old metadata, previously downloaded packages, etc.) would run the appropriate command(s).

As for the journal: I'm pretty sure size and time can be used together. And if the size is "unlimited" or time is "complete history" (or somesuch), then it's a no-op for either (or both). If both are a no-op, we should probably have a notice. Perhaps we should default to a generous size and/or time.

If we did have a dialog like this, people may want to active it on their own. So it might need to be a component that's automatically suggested by the overview page's health area and (somehow) able to be manually shown from the Storage page?

[garrett]

So what I'm suggesting here is a dialog that would allow someone to clear the cache and trim the journal. It would be available when needed and possibly as an action from somewhere else, such as actions on the storage page somewhere (possibly associated with the system partition which has cache and logs).

In the context of this issue, I'm more concerned about having this reusable component existing for the usage of when #8787 alerts about space not being available. Having it available elsewhere (even when not needed) is less of a priority.

[garrett]

I tagged this as the Overview page, but it's really a dialog component that will be mainly featured on the overview page (and might need to be called from the Metrics and/or Storage page too).