“Error syncing pod, skipping: timeout expired waiting for volumes to attach”

At Zegami we are working hard to make the ultimate experience for browsing large collections of images and visual data. For our upcoming cloud release (more details to follow soon) this means picking the right technology for the best possible user experience. To that end we’ve fully embraced container technology and have recently selected Google Container Engine as our hosting platform of choice.

There is a good amount of jargon in this post, if you are not familiar with Google Container Engine and Kubernetes they do have ever-improving documentation where you can learn about the tool. Some of that documentation comes in the form of notes when using the CLI or web client. Reading everything on the page or terminal is super helpful!

The error in the title is pretty commonplace when attaching storage to deployments/pods. For a little while I was scratching my head as to why I could create persistent storage as part of following the Jenkins tutorial but any other volumes that I created ended up failing with the same error, for example:

$kubectl describe pods postgres3-62097224-hbh95
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
3h 19s 102 {kubelet gke-cluster-1-default-pool-9e1dca4d-xl0d} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"postgres3-62097224-hbh95". list of unattached/unmounted volumes=[zegamiglobalportal-postgres-backup-claim zegamiglobalportal-postgres-data]

Error syncing pod, skipping: timeout expired waiting for volumes to attach

Why is this happening? A timeout error like this is pretty broad brush and it turned out that in my case I had simply not formatted the disk correctly.

When creating disks with the gcloud CLI you are warned about this:

gcloud compute --project "xxxxx" disks create "disk-1" --size "10" --zone "europe-west1-d" --type "pd-standard" 
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/master-booster-164412/zones/europe-west1-d/disks/disk-1].
disk-1 europe-west1-d 10 pd-standard READYNew disks are unformatted. You must format and mount a disk before it
can be used. You can find instructions on how to do this at:https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting

Following the instructions at https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting allows you to create a mounted folder in a VM.

You would then add necessary files to that folder and detach the folder from the VM. Following this you can create a new image based upon that disk and then use that disk as a gcePersistentDisk in Kubernetes.