Google Kubernetes Engine's auto-pilot mode does not grant nodes Google Storage write access
jeanid last edited by
I am trying to use the workflow orchestrator Snakemake with a Google Kubernetes Engine (GKE) https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview cluster. My workflow needs to upload files to Google Cloud Storage buckets within the same project. Both https://snakemake.readthedocs.io/en/v7.2.1/executing/cloud.html#setup-kubernetes-on-google-cloud-engine and https://cloud.google.com/deep-learning-containers/docs/kubernetes-container specify that a Google Storage read-write scope should be specified when the cluster is created, via
I tried doing so with an autopilot cluster:
gcloud container --project "project-name" clusters create-auto "default-cluster" \ --region "asia-southeast2" --release-channel "regular" \ --network "projects/project-name/global/networks/default" \ --subnetwork "projects/project-name/regions/asia-southeast2/subnetworks/default" \ --cluster-ipv4-cidr "/17" --services-ipv4-cidr "/22" --scopes storage-rw
(I generated this command using the Google Cloud web console, then added the
--scopes storage-rwargument at the end.)
But I still get a Snakemake error message related to lack of permission for Google Storage read-write:
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/kubernetes-nccs-npc2/o/whoami.txt?fields=name&prettyPrint=false: Caller does not have storage.objects.get access to the Google Cloud Storage object.
--scopes storage-rwdoes not work for the creation of autopilot clusters. Is this true? If so, how can I grant Google Storage read-write access to an autopilot cluster?
I think the error has more to do with the service account permissions running your cluster. I'd make sure all the permissions are included, like storage.ojjects.get. The error seems very permissions related.
Caller does not have storage.objects.get access to the Google Cloud Storage object.
You might be hitting this issue Google has reported:
I'd listen to their recommendations over mine.
When managing IAM roles, you can treat a service account either as a resource or as an identity. This https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/get-iam-policy is to get the iam policy of a service account resource. There are other gcloud commands to manage IAM policies for other types of resources. For example, to manage IAM policies on a project, use the $ gcloud projects commands and for services use $ gcloud iam service-accounts.
This will return your service running the cluster w/ role assignment (it's the robot one )
gcloud projects get-iam-policy
The problem is sometimes running these commands requires special permissions too, so it becomes fools progress. I think this link may be helpful. it shows how to set permissions on a service-account.
Also here is Google dictionary of IAM permissions including storage.ojjects.get.