Add ability to check whether PVC storage requirements are fulfilled by the node

[maunavarro]

Describe the problem/challenge you have

It seems that a PVC can claim and be provisioned more storage than the disk size of a particular node.
For example, the following PVC requesting 5G should be denied if the available capacity of nodes in cluster is 1G

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-hostpath-pvc
  namespace: openebs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5G

As it currently stands, the above PVC will be provisioned and bound to a PV .

Describe the solution you'd like

We would like to limit the storage policy to existing free storage in node.

Environment:

• OpenEBS version:

$ kubectl get po -n openebs --show-labels
NAME                                           READY   STATUS    RESTARTS   AGE    LABELS
openebs-localpv-provisioner-b559bc6c4-k87hl    1/1     Running   0          4h4m   name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.0.0,pod-template-hash=b559bc6c4
openebs-ndm-cluster-exporter-b48f4c59d-f24d9   1/1     Running   0          4h4m   name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.0.0,pod-template-hash=b48f4c59d
openebs-ndm-node-exporter-22mv4                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-2g78z                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-4h526                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-9p2xv                1/1     Running   3          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-c999w                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-d5xsq                1/1     Running   3          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-fnl8m                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-h2q88                1/1     Running   4          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-kcrs7                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-vclcz                1/1     Running   2          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-x6ww5                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-node-exporter-xczq2                1/1     Running   1          14d    controller-revision-hash=5fc6c5df65,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.0.0,pod-template-generation=1
openebs-ndm-operator-77d54c5c69-pt85n          1/1     Running   0          89m    name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.0.0,pod-template-hash=77d54c5c69

• Kubernetes version :

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-4+2076e5554ee7d2", GitCommit:"2076e5554ee7d2e0ac57857d161db7fd6fad915a", GitTreeState:"clean", BuildDate:"2021-01-05T22:46:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g: cat /etc/os-release):

~# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• kernel:
  linux

[trathborne]

As mentioned in a Slack #openebs thread:

There are a few dimensions to the problem:

• (any filesystem) Current available space ← naïve solution to this would be a great start!
• (XFS) Quotas & their usage (and maybe an overcommit percentage)
• (LVM) Space available in VG

Affinity-by-IOPS might be another way to schedule, and as I understand it affinities can have some ranking metric so I can say 'Allocate this PV where:' ... 'it just fits' or 'there is the most space' or 'where other local PVs are generating fewer IOPS'.

[niladrih]

The hostpath provisioner volumes are thin-provisioned (basically how the filesystem handles it).
Taking an example...
Let's say there's a filesystem with 10 GiB storage capacity. A hostpath volume with 7 GiB capacity request is created on this filesystem. It succeeds. The application process consumes a small amount of space, say 1 GiB.
Going forward, if another volume request for say, 5 GiB is created, it will succeed. As the available capacity for the filesystem will be observed to be (10-1) GiB = 9 GiB!
The hostpath provisioner does not pre-allocate the requested capacity.

@maunavarro @trathborne -- Does the above hostpath provisioner behaviour suffice your use-case? Thick provisioning is beyond the scope of the hostpath provisioner's design limitations.

[trathborne]

This is working fine for me because I am using XFS quotas and I am ok with overcommitting.

I am more concerned about balance. Consider this case: I have 3 nodes, each with 10GiB of storage. 4 PVCs for 6GiB of storage come in. 2 nodes get 1 PVC, one node gets 2 PVCs. Now another PVC for 6GiB comes in. What is to guaranteed that it does not end up on the node that is already overcommitted?

[niladrih]

@trathborne -- The LVM-based LocalPV CSI-driver is better suited for capacity-aware scheduling use-cases. Here's a link: github.com/openebs/lvm-localpv

[trathborne]

@niladrih I was going to say that LVM doesn't let me overcommit, but then I found https://github.com/openebs/lvm-localpv/blob/2d5196996e97edb82d1840900416835b0b729328/design/monitoring/capacity_monitor.md#thin-logical-volume ... thanks!

[niladrih]

Hi @maunavarro, this issue has been open for a while. Are you are still facing this issue?

This feature is one which is typically present in a CSI driver based storage plugin (e.g. LocalPV ZFS, or LocalPV LVM). This is unfortunately beyond the scope of the out-of-tree provisioner library that this hostpath provisioner uses.

That being said, my viewpoint might be pigeonholed. Please feel free to open a design proposal or continue the discussion here. Marking this as 'wont-fix' for now.

[avishnu]

@niladrih if there is no possible way to achieve this in the external provisioner, should this be closed?

[niladrih]

A LocalPV Rawfile style pre-allocated volume comes to mind.

Another alternative could be filesystem quota.