Google Kubernetes Engine's auto-pilot mode does not grant nodes Google Storage write access

[jeanid]

I am trying to use the workflow orchestrator Snakemake with a Google Kubernetes Engine (GKE) https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview cluster. My workflow needs to upload files to Google Cloud Storage buckets within the same project. Both https://snakemake.readthedocs.io/en/v7.2.1/executing/cloud.html#setup-kubernetes-on-google-cloud-engine and https://cloud.google.com/deep-learning-containers/docs/kubernetes-container specify that a Google Storage read-write scope should be specified when the cluster is created, via --scopes storage-rw.

I tried doing so with an autopilot cluster:

gcloud container --project "project-name" clusters create-auto "default-cluster" \
    --region "asia-southeast2" --release-channel "regular" \
    --network "projects/project-name/global/networks/default" \
    --subnetwork "projects/project-name/regions/asia-southeast2/subnetworks/default" \
    --cluster-ipv4-cidr "/17" --services-ipv4-cidr "/22" --scopes storage-rw

(I generated this command using the Google Cloud web console, then added the --scopes storage-rw argument at the end.)

But I still get a Snakemake error message related to lack of permission for Google Storage read-write:

google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/kubernetes-nccs-npc2/o/whoami.txt?fields=name&prettyPrint=false: Caller does not have storage.objects.get access to the Google Cloud Storage object.

It seems --scopes storage-rw does not work for the creation of autopilot clusters. Is this true? If so, how can I grant Google Storage read-write access to an autopilot cluster?

[Amritpal]

I think the error has more to do with the service account permissions running your cluster. I'd make sure all the permissions are included, like storage.ojjects.get. The error seems very permissions related.

Caller does not have storage.objects.get access to the Google Cloud Storage object.

You might be hitting this issue Google has reported:
https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes

I'd listen to their recommendations over mine.

When managing IAM roles, you can treat a service account either as a resource or as an identity. This https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/get-iam-policy is to get the iam policy of a service account resource. There are other gcloud commands to manage IAM policies for other types of resources. For example, to manage IAM policies on a project, use the $ gcloud projects commands and for services use $ gcloud iam service-accounts.

This will return your service running the cluster w/ role assignment (it's the robot one )

gcloud projects get-iam-policy  

The problem is sometimes running these commands requires special permissions too, so it becomes fools progress. I think this link may be helpful. it shows how to set permissions on a service-account.

https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/add-iam-policy-binding

Also here is Google dictionary of IAM permissions including storage.ojjects.get.

https://cloud.google.com/iam/docs/understanding-roles