Kubelet kubelet_volume_stats_* wrong metrics values under cri-o

[atillaqb]

Describe the bug: CSI Driver reporting wrong prometheus metrics to the Kubelet

Capacity of the all volumes: 88 GB (when I created PVC only for 1Gb volume).

df -h
Filesystem                  Size  Used Avail Use% Mounted on
devtmpfs                     12G     0   12G   0% /dev
tmpfs                        12G  376K   12G   1% /dev/shm
tmpfs                        12G   12M   12G   1% /run
tmpfs                        12G     0   12G   0% /sys/fs/cgroup
/dev/mapper/ocivolume-root   89G   58G   32G  65% /

Most likely CSI driver using values for the metrics:

• kubelet_volume_stats_used_bytes
• kubelet_volume_stats_capacity_bytes
• kubelet_volume_* family

From the host directly. It is not calculating the size of the folders, used for the PV.

Expected behaviour: True size of the Volumes based on the size of the Hostpath folder used to provision volume. Is it possible?

Steps to reproduce the bug:

1. Install Kubernetes on the OKE worker node.
2. Install openebs from the helm chart
3. Use openebs-hostpath in the storageclass of the PVC.

The output of the following commands will help us better understand what's going on:

• kubectl get pods -n <openebs_namespace> --show-labels :

app=localpv-provisioner,chart=localpv-provisioner-4.2.0,component=localpv-provisioner,heritage=Helm,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,pod-template-hash=6455c4d767,release=openebs

Anything else we need to know?:
Add any other context about the problem here.

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels): localpv-provisioner,chart=localpv-provisioner-4.2.0
• Kubernetes version (use kubectl version): 1.31.1
• Cloud provider or hardware configuration: OCI OKE self managed
• OS (e.g: cat /etc/os-release): Oracle Linux 8.10
• kernel (e.g: uname -a): 5.15.0-207.156.6.el8uek.x86_64
• others:

[tiagolobocastro]

hmm not sure how we export metrics today (btw this is not a csi driver).
But I don't think we can easily do this here, since hostpath is simply a directory within the filesystem, we'd have to do some kind of du command, but that's expensive.
@niladrih wdyt?

[avishnu]

This has been a known behavior with hostpath pvc. There was a prior effort to come up with a "du" based approach, but that was deemed expensive operation, as @tiagolobocastro points out above.

[atillaqb]

Hello! Maybe this can be useful block of code.
I started with a question how to efficiently report directory size via NodeExporter and found this script
https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/directory-size.sh
It is uses bash and du.
But for CSI driver it is not usable. So I decided to find a CSI specs for the VolumeSize / Stats
https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats
Then I asked LLM for the implementation of directory size in CSI driver:

package main

import (
	"fmt"
	"os"
	"path/filepath"
	"syscall"
)

// getFilesystemStats returns total and available space for the filesystem at path.
func getFilesystemStats(path string) (total, available int64, err error) {
	var stat syscall.Statfs_t
	if err = syscall.Statfs(path, &stat); err != nil {
		return
	}
	total = int64(stat.Blocks) * int64(stat.Bsize)
	available = int64(stat.Bavail) * int64(stat.Bsize)
	return
}

// getDirectoryUsage recursively sums up the sizes of files in the given directory.
func getDirectoryUsage(path string) (int64, error) {
	var total int64
	err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		}
		if !info.IsDir() {
			total += info.Size()
		}
		return nil
	})
	return total, err
}

func main() {
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "Usage: %s <volume_path>\n", os.Args[0])
		os.Exit(1)
	}
	volumePath := os.Args[1]

	// Check if the volumePath exists and is a directory.
	info, err := os.Stat(volumePath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
		os.Exit(1)
	}
	if !info.IsDir() {
		fmt.Fprintf(os.Stderr, "Error: %s is not a directory\n", volumePath)
		os.Exit(1)
	}

	// Option 1: Use statfs to get filesystem-wide stats.
	totalFS, availableFS, err := getFilesystemStats(volumePath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error retrieving filesystem stats: %v\n", err)
		os.Exit(1)
	}
	usedFS := totalFS - availableFS

	// Option 2: Calculate directory usage exactly (this might be slower).
	dirUsed, err := getDirectoryUsage(volumePath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error calculating directory usage: %v\n", err)
		os.Exit(1)
	}

	// For demonstration, we print both.
	// In a production CSI driver, choose the method that best suits your volume semantics.
	fmt.Println("# HELP node_directory_size_bytes Disk space used by the volume")
	fmt.Println("# TYPE node_directory_size_bytes gauge")
	// Reporting filesystem stats (if volumePath is a dedicated mount)
	fmt.Printf("node_directory_size_bytes{volume=%q,method=%q} %d\n", volumePath, "statfs", usedFS)
	// Reporting exact directory usage (if volumePath is a subdirectory)
	fmt.Printf("node_directory_size_bytes{volume=%q,method=%q} %d\n", volumePath, "du", dirUsed)

	// For NodeGetVolumeStatsResponse, you’d wrap these values in a proper gRPC response:
	//
	// VolumeUsage{
	//    Available: availableFS, // if using statfs
	//    Total:     totalFS,       // if using statfs
	//    Used:      usedFS,        // or dirUsed, depending on your use case
	//    Unit:      VolumeUsage_BYTES,
	// }
}

[geier]

We build https://github.com/prokube/local-storage-exporter for better monitoring of local storage classes such as openebs-hostpath.

[tiagolobocastro]

Thanks @geier didn't know about that

We also an alternative to the hostpath, the rawfile-localpv which tackles some of the issues of hostpath by creating a new filesystem for each volume rather than sharing the host path directly