Since Intel CPU Azure update, new Azure Disks are not mounting, very critical...

[theobolo]

Is this a request for help?:

New azure managed disks are not mouting on the Kubernetes nodes, Azure API slow.

Is this an ISSUE or FEATURE REQUEST? (choose one):

It's an ISSUE

What version of acs-engine?:

ACS Engine version : 0.7.0
Kubernetes version : 1.7.5

kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

After the big update yesterday, i'm seeing a lot of errors about attaching or detaching Azure Managed Disks on kubernetes, espacially the new created ones.

I recreated all my stacks (Elasticsearch/kafka/Zookeeper/...) using a PersistentVolumeClaim with my statefullsets after the big update yesterday.

All my new Azure Disks are not able to be mounted on my workers, but actually it should :

I'm posting the logs of the kube-controller manager :

W0105 10:24:00.133401       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.133439       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.133458       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.233803       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.233910       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.234034       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.234063       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.334429       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.334518       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.334462       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.334586       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.434953       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.434987       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.435103       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.435139       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.535432       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.535472       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.535692       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.535715       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.636100       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.636142       1 reconciler.go:267] Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a") from node "k8s-k8sworkers-20163042-3" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.636168       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"integration", Name:"grafana-1826545959-8qw5q", UID:"20425f47-f1f7-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22791080", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a" Volume is already exclusively attached to one node and can't be attached to another
I0105 10:24:00.636183       1 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"preproduction", Name:"mongo-0", UID:"5492929f-f1b4-11e7-92c5-000d3ab769d6", APIVersion:"v1", ResourceVersion:"22247369", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" Volume is already exclusively attached to one node and can't be attached to another
W0105 10:24:00.736504       1 reconciler.go:267] Multi-Attach error for volume "pvc-8743f77c-a472-11e7-b780-000d3ab769d6" (UniqueName: "kubernetes.io/azure-disk//subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-8743f77c-a472-11e7-b780-000d3ab769d6") from node "k8s-k8sworkers-20163042-2" Volume is already exclusively attached to one node and can't be attached to another

By the way it's an issue that the Azure community faced already and me aswell. Someone experienced the same kind of issues yesterday on an other Git thread. Azure/ACS#12

That's really critical, during the updates yesterday the Machines went down in a normal way but the Azure API was really slow and the mounting and detaching too.
Kubernetes was trying to balance my pods with their disks to another worker since the actual one was upgrading (more than 40min per machine), but it was impossible since the disks weren't able to detach from that worker.

I almost waited more than 1 hour to have my 3 mongoDb Azure Disks attached on my ReplicaSet after each machine reboot. I tought that Azure Disks where safe in production but basically it wasn't yesterday.

Conclusion, i'm not able to mount any disks on my statefullsets since yesterday on my Staging cluster.

Rebooted : kube-controller-manager, kubelet on the concerned workers, even restarted the virtual machines, it's always the same detach/attach error.

Please guys do you have any info on that, my Direct Professional Support plan mail is still not answered since 24h...

How to reproduce it (as minimally and precisely as possible):

Apparently someone else experienced that : Azure/ACS#12

[andyzhangx]

@theobolo could you run kubectl describe pvc PVC-NAME , it says "PersisentVolumeClaim is not bound" in the screenshot

[theobolo]

@andyzhangx Hi Andy, there it is :

Don't look at the log on the Kubernetes Dashboard, it's not always the last logs. It's happening on all the new PVC right now the real state is that the PVC is bound but :

The disks are up on the Azure Dashboard as i said and "Bound" on Kubernetes PVC, but they are not mounted even if they are marked "Unattached" on Azure.

And i paste also the log from kubelet from the worker-3 where Kafka-0 is supposed to be deployed :

[andyzhangx]

@theobolo
could you use azure-cli to run az vm show -g RESOURCE-GROUP -n AGENT-VM-NAME -d to get the agent VM details? you may need to run az login first. Thanks.

[theobolo]

@andyzhangx There it is (the kubernetes worker 3), but agents are running on workers i think, since they were before the update :

ghosty@workstation-ghosty:~/kubernetes-deployment/kafka_deployment$ az vm show -g k8s-fleeters-cluster-preproduction -n k8s-k8sworkers-20163042-3 -d
{
  "additionalProperties": {},
  "availabilitySet": {
    "additionalProperties": {},
    "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/availabilitySets/K8SWORKERS-AVAILABILITYSET-20163042",
    "resourceGroup": "k8s-fleeters-cluster-preproduction"
  },
  "diagnosticsProfile": null,
  "fqdns": "",
  "hardwareProfile": {
    "additionalProperties": {},
    "vmSize": "Standard_DS13_v2_Promo"
  },
  "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/virtualMachines/k8s-k8sworkers-20163042-3",
  "identity": null,
  "licenseType": null,
  "location": "northeurope",
  "macAddresses": "00-0D-3A-B7-64-8B",
  "name": "k8s-k8sworkers-20163042-3",
  "networkProfile": {
    "additionalProperties": {},
    "networkInterfaces": [
      {
        "additionalProperties": {},
        "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Network/networkInterfaces/k8s-k8sworkers-20163042-nic-3",
        "primary": null,
        "resourceGroup": "k8s-fleeters-cluster-preproduction"
      }
    ]
  },
  "osProfile": {
    "additionalProperties": {},
    "adminPassword": null,
    "adminUsername": "fleeters",
    "computerName": "k8s-k8sworkers-20163042-3",
    "customData": null,
    "linuxConfiguration": {
      "additionalProperties": {},
      "disablePasswordAuthentication": true,
      "ssh": {
        "additionalProperties": {},
        "publicKeys": [
          {
            "additionalProperties": {},
            "keyData": "HIDDEN",
            "path": "/home/fleeters/.ssh/authorized_keys"
          }
        ]
      }
    },
    "secrets": [],
    "windowsConfiguration": null
  },
  "plan": null,
  "powerState": "VM running",
  "privateIps": "10.240.0.5",
  "provisioningState": "Succeeded",
  "publicIps": "",
  "resourceGroup": "k8s-fleeters-cluster-preproduction",
  "resources": [
    {
      "additionalProperties": {},
      "autoUpgradeMinorVersion": true,
      "forceUpdateTag": null,
      "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/virtualMachines/k8s-k8sworkers-20163042-3/extensions/cse3",
      "instanceView": null,
      "location": "northeurope",
      "name": "cse3",
      "protectedSettings": null,
      "provisioningState": "Succeeded",
      "publisher": "Microsoft.Azure.Extensions",
      "resourceGroup": "k8s-fleeters-cluster-preproduction",
      "settings": {},
      "tags": null,
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "typeHandlerVersion": "2.0",
      "virtualMachineExtensionType": "CustomScript"
    }
  ],
  "storageProfile": {
    "additionalProperties": {},
    "dataDisks": [],
    "imageReference": {
      "additionalProperties": {},
      "id": null,
      "offer": "UbuntuServer",
      "publisher": "Canonical",
      "sku": "16.04-LTS",
      "version": "16.04.201706191"
    },
    "osDisk": {
      "additionalProperties": {},
      "caching": "ReadWrite",
      "createOption": "FromImage",
      "diskSizeGb": 128,
      "encryptionSettings": null,
      "image": null,
      "managedDisk": {
        "additionalProperties": {},
        "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/k8s-k8sworkers-20163042-3_OsDisk_1_b24ffe34a2504df2924985f9c9fd013f",
        "resourceGroup": "k8s-fleeters-cluster-preproduction",
        "storageAccountType": "Premium_LRS"
      },
      "name": "k8s-k8sworkers-20163042-3_OsDisk_1_b24ffe34a2504df2924985f9c9fd013f",
      "osType": "Linux",
      "vhd": null
    }
  },
  "tags": {
    "creationSource": "acsengine-k8s-k8sworkers-20163042-3",
    "orchestrator": "Kubernetes:1.7.5",
    "poolName": "k8sworkers",
    "resourceNameSuffix": "20163042"
  },
  "type": "Microsoft.Compute/virtualMachines",
  "vmId": "30c29441-6c41-4de0-b88b-1a50293fd257",
  "zones": null
}

[theobolo]

It seems that the worker-2 have a problem by the way :

{
  "additionalProperties": {},
  "availabilitySet": {
    "additionalProperties": {},
    "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/availabilitySets/K8SWORKERS-AVAILABILITYSET-20163042",
    "resourceGroup": "k8s-fleeters-cluster-preproduction"
  },
  "diagnosticsProfile": null,
  "fqdns": "",
  "hardwareProfile": {
    "additionalProperties": {},
    "vmSize": "Standard_DS13_v2_Promo"
  },
  "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/virtualMachines/k8s-k8sworkers-20163042-2",
  "identity": null,
  "licenseType": null,
  "location": "northeurope",
  "macAddresses": "00-0D-3A-B7-60-BC",
  "name": "k8s-k8sworkers-20163042-2",
  "networkProfile": {
    "additionalProperties": {},
    "networkInterfaces": [
      {
        "additionalProperties": {},
        "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Network/networkInterfaces/k8s-k8sworkers-20163042-nic-2",
        "primary": null,
        "resourceGroup": "k8s-fleeters-cluster-preproduction"
      }
    ]
  },
  "osProfile": {
    "additionalProperties": {},
    "adminPassword": null,
    "adminUsername": "fleeters",
    "computerName": "k8s-k8sworkers-20163042-2",
    "customData": null,
    "linuxConfiguration": {
      "additionalProperties": {},
      "disablePasswordAuthentication": true,
      "ssh": {
        "additionalProperties": {},
        "publicKeys": [
          {
            "additionalProperties": {},
            "keyData": "NOT THIS TIME",
            "path": "/home/fleeters/.ssh/authorized_keys"
          }
        ]
      }
    },
    "secrets": [],
    "windowsConfiguration": null
  },
  "plan": null,
  "powerState": "VM running",
  "privateIps": "10.240.0.4",
  "provisioningState": "Failed",
  "publicIps": "",
  "resourceGroup": "k8s-fleeters-cluster-preproduction",
  "resources": [
    {
      "additionalProperties": {},
      "autoUpgradeMinorVersion": true,
      "forceUpdateTag": null,
      "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/virtualMachines/k8s-k8sworkers-20163042-2/extensions/cse2",
      "instanceView": null,
      "location": "northeurope",
      "name": "cse2",
      "protectedSettings": null,
      "provisioningState": "Updating",
      "publisher": "Microsoft.Azure.Extensions",
      "resourceGroup": "k8s-fleeters-cluster-preproduction",
      "settings": {},
      "tags": null,
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "typeHandlerVersion": "2.0",
      "virtualMachineExtensionType": "CustomScript"
    }
  ],
  "storageProfile": {
    "additionalProperties": {},
    "dataDisks": [
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 20,
        "image": null,
        "lun": 2,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-287cfe7d-b999-11e7-a344-000d3ab7665a",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-287cfe7d-b999-11e7-a344-000d3ab7665a",
        "vhd": null
      },
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 30,
        "image": null,
        "lun": 3,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-d68e7210-bee9-11e7-a344-000d3ab7665a",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-d68e7210-bee9-11e7-a344-000d3ab7665a",
        "vhd": null
      },
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 64,
        "image": null,
        "lun": 0,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-c48ca983-b0d1-11e7-b9f4-000d3ab76964",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-c48ca983-b0d1-11e7-b9f4-000d3ab76964",
        "vhd": null
      },
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 32,
        "image": null,
        "lun": 4,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-4c0b9e2b-a48d-11e7-a344-000d3ab7665a",
        "vhd": null
      },
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 128,
        "image": null,
        "lun": 1,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-848a1391-a472-11e7-b780-000d3ab769d6",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-848a1391-a472-11e7-b780-000d3ab769d6",
        "vhd": null
      },
      {
        "additionalProperties": {},
        "caching": "ReadWrite",
        "createOption": "Attach",
        "diskSizeGb": 64,
        "image": null,
        "lun": 5,
        "managedDisk": {
          "additionalProperties": {},
          "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/fleeters-59cd2dcd-dynamic-pvc-984109f2-c9da-11e7-a941-000d3ab76964",
          "resourceGroup": "k8s-fleeters-cluster-preproduction",
          "storageAccountType": "Premium_LRS"
        },
        "name": "fleeters-59cd2dcd-dynamic-pvc-984109f2-c9da-11e7-a941-000d3ab76964",
        "vhd": null
      }
    ],
    "imageReference": {
      "additionalProperties": {},
      "id": null,
      "offer": "UbuntuServer",
      "publisher": "Canonical",
      "sku": "16.04-LTS",
      "version": "16.04.201706191"
    },
    "osDisk": {
      "additionalProperties": {},
      "caching": "ReadWrite",
      "createOption": "FromImage",
      "diskSizeGb": 128,
      "encryptionSettings": null,
      "image": null,
      "managedDisk": {
        "additionalProperties": {},
        "id": "/subscriptions/3770e200-bb58-4d3b-afcb-66c7ce083c3f/resourceGroups/k8s-fleeters-cluster-preproduction/providers/Microsoft.Compute/disks/k8s-k8sworkers-20163042-2_OsDisk_1_3bb94d8af0d449d38f1f19123975094a",
        "resourceGroup": "k8s-fleeters-cluster-preproduction",
        "storageAccountType": "Premium_LRS"
      },
      "name": "k8s-k8sworkers-20163042-2_OsDisk_1_3bb94d8af0d449d38f1f19123975094a",
      "osType": "Linux",
      "vhd": null
    }
  },
  "tags": {
    "creationSource": "acsengine-k8s-k8sworkers-20163042-2",
    "orchestrator": "Kubernetes:1.7.5",
    "poolName": "k8sworkers",
    "resourceNameSuffix": "20163042"
  },
  "type": "Microsoft.Compute/virtualMachines",
  "vmId": "9f8e9bb3-2152-4aca-8141-e41a31b39d35",
  "zones": null
}

[theobolo]

There is certainly something wrong with the Azure API, it says that my worker-2 and 4 have failed, on Azure Dashboard. But Kubernetes sees the nodes as "Ready" and i've some pods running on them ...

Azure status :

My pods on the worker-2 :

[andyzhangx]

thanks, so where is the data disk fleeters-59cd2dcd-dynamic-pvc-f4be8993-a472-11e7-b780-000d3ab769d6, error says it's attaching that disk to work-3, could you find it your resource group?
az disk list | grep f4be8993

[theobolo]

@andyzhangx Yep there it is :

[andyzhangx]

@theobolo then I think it's caused by the VM failed state, the error(attaching data disk fleeters-59cd2dcd-dynamic-pvc-f4be8993-a472-11e7-b780-000d3ab769d6 to work-3 failed) could be the last error long time ago. I think you should first resolve the VM failed state issue, I only got the Powershell solution here:
https://blogs.technet.microsoft.com/mckittrick/azure-vm-stuck-in-failed-state-arm/

[theobolo]

Yep but basically the worker-3 is in a Running State with no error and with a successfull agent running on it.
My disks are not either attached to a Failed machine since they are brand new : i created my statefulsets few hours ago, and my disks weren't mounted before anywhere else.

I'm gonna try to remove the "Failed" status using your method on the other workers, but i think that there is something more.

[andyzhangx]

you may update worker-3 VM as well if possible, the error said attach disk 1 failed because attach disk 2 to worker-3 VM failed, it's quite weired, there is possiblity that worker-3 VM is also in a wrong state.

[andyzhangx]

BTW, could you check the status of data disk fleeters-59cd2dcd-dynamic-pvc-f4be8993-a472-11e7-b780-000d3ab769d6, is it attached to a VM? that's important. Thanks.

[theobolo]

@andyzhangx Yep since yesterday i have a lot of really weird things on the Azure API, i saw a statement yesterday saying that it's not relevant anymore to look at the Azure messages on the dashboard, because they are delayed or laggy.

By the way i putted the disk status within my first message, it's absolutly not attached to any node and it was never been.

[theobolo]

Now all my nodes seems to be healthy using : az vm show -g k8s-fleeters-cluster-preproduction -n k8s-k8sworkers-20163042-0 -d command.

No more "Failed" state on Azure. I'll delete all the PVC and the statefulsets and just redeploy Kafka.