I am trying to run a simple gsutil rsync command in Airflow 1.10.10 using a BashOperator. I am getting the following error when executing the command:
gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket
I have tried placing the credential file in and using this:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/folder/path/credentials.json"
And I have also read through this document: https://cloud.google.com/storage/docs/gsutil_install#authenticate
The problem is that I cannot login using a browser or in any other interactive way because the Airflow setup code will run in a Docker container on AWS EKS. There's no feasible way to manually copy and paste verification codes. So I don't think any of the Google docs are helpful.
I have also tried the solutions here: gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket even though I'm loggedin in gcloud, specifically the solutions which asked me to run these two lines:
gcloud auth login
gcloud auth application-default login
So in my Dockerfile, I wrote RUN pip install gcloud and it installed correctly, but when I try to run the two commands above, it says bash: gcloud: command not found.
Why is gcloud installing, but not allowing me to run commands? Is there any way to get this working on a container where I cannot open a browser nor copy/paste verification codes?
