WaitForAttach failed for azure disk: parsing "/dev/disk/azure/scsi1/lun1": invalid syntax

[andyzhangx]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:

Events:
  Type     Reason                  Age                 From                               Message
  ----     ------                  ----                ----                               -------
  Warning  FailedScheduling        25m (x26 over 31m)  default-scheduler                  0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 1 node(s) were not ready, 1 node(s) were out of disk space, 1 node(s) were unschedulable.
  Normal   SuccessfulMountVolume   23m                 kubelet, k8s-agentpool-66825246-0  MountVolume.SetUp succeeded for volume "default-token-cxk4v"
  Normal   SuccessfulAttachVolume  23m                 attachdetach-controller            AttachVolume.Attach succeeded for volume "pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967"
  Warning  FailedMount             1m (x11 over 22m)   kubelet, k8s-agentpool-66825246-0  MountVolume.WaitForAttach failed for volume "pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967" : azureDisk - Wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun1 (strconv.Atoi: parsing "/dev/disk/azure/scsi1/lun1": invalid syntax)
  Warning  FailedMount             1m (x10 over 21m)   kubelet, k8s-agentpool-66825246-0  Unable to mount volumes for pod "deployment-azuredisk1-68849b4cbf-k9blx_default(6e72ed25-3eb7-11e8-ab6b-000d3af9f967)": timeout expired waiting for volumes to attach or mount for pod "default"/"deployment-azuredisk1-68849b4cbf-k9blx". list of unmounted volumes=[azuredisk]. list of unattached volumes=[azuredisk default-token-cxk4v]
azureuser@k8s-master-66825246-0:~$ kubectl get no

  Warning  FailedMount             1m (x11 over 22m)   kubelet, k8s-agentpool-66825246-0  MountVolume.WaitForAttach failed for volume "pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967" : azureDisk - Wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun1 (strconv.Atoi: parsing "/dev/disk/azure/scsi1/lun1": invalid syntax)
  Warning  FailedMount             1m (x10 over 21m)   kubelet, k8s-agentpool-66825246-0  Unable to mount volumes for pod  

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
This was introduced by following code change introduced in v1.10.0:

kubernetes/pkg/volume/util/operationexecutor/operation_generator.go

Lines 517 to 518 in 568afb4

	markDeviceMountedErr := actualStateOfWorld.MarkDeviceAsMounted(
	volumeToMount.VolumeName, devicePath, deviceMountPath)

candidate fix is here(only works on Linux, not a good solution):
andyzhangx@e287390

This issue is caused by PR: #58177

Environment:

• Kubernetes version (use kubectl version): v1.10
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

/sig azure
/assign

[andyzhangx]

Update: above fix does not work on Windows, I am still investigating a better solution.
This issue is caused by devicePath updated after WaitForAttach, related code is here:

kubernetes/pkg/volume/util/operationexecutor/operation_generator.go

Lines 517 to 518 in ee4d90a

	markDeviceMountedErr := actualStateOfWorld.MarkDeviceAsMounted(
	volumeToMount.VolumeName, devicePath, deviceMountPath)

while devicePath should not be changed from LUN to real device path in azure_dd since

• AttachDisk happens on master, devicePath is a LUN(Logical Unit Number) number after AttachDisk process in the beginning
• WaitForAttach happens on agent node:
  • on Linux node, devicePath could be /dev/disk/azure/scsi1/lunx;
  • while on Windows node, devicePath would be a disk number, when WaitForAttach is recalled, it could not tell whether it's a LUN or a disk number (this is the key point).

[jingxu97]

@andyzhangx thanks for issuing this problem. Could you submit a PR with your fix so that we can review it? Thanks!

[andyzhangx]

@jingxu97 finally I realized devicePath should not be used in WaitForAttach func any more by referencing the gce_pd code, PTAL #62612, aws_ebs code should have same issue, I filed a aws_ebs issue here: #62613