Topo data not loading from gcs

[peterdudfield]

Describe the bug
Topo data doesnt seem to load from GCS.

To Reproduce
I'll add a test https://github.com/openclimatefix/nowcasting_dataset/blob/main/tests/data_sources/test_topographic_data_source.py#L22

Expected behavior
Open data ok

Additional context
' rasterio.errors.RasterioIOError: GS_SECRET_ACCESS_KEY+GS_ACCESS_KEY_ID, GS_OAUTH2_REFRESH_TOKEN or GOOGLE_APPLICATION_CREDENTIALS or GS_OAUTH2_PRIVATE_KEY+GS_OAUTH2_CLIENT_EMAIL configuration options and /Users/peterdudfield/.boto not defined
'

• it may be i need to set some env variables

It should work fine

• https://gis.stackexchange.com/questions/364838/can-i-open-remote-datasets-with-rasterio

[peterdudfield]

@jacobbieker would you mind running the test with your setup and see if it works?

[jacobbieker]

Running it results in my having the same error as well

[peterdudfield]

corteva/rioxarray#246

[jacobbieker]

Changing it to this:

fs = fsspec.open(self.filename).fs
        with fs.open(self.filename) as f:
            self._data = rioxarray.open_rasterio(
                filename=f, parse_coordinates=True, masked=True
            )

results in

self = <gcsfs.core.GCSFileSystem object at 0x7fdd3f981640>, status = 401
content = b'{\n  "error": {\n    "code": 401,\n    "message": "Anonymous caller does not have storage.objects.get access to the ...reason": "required",\n        "locationType": "header",\n        "location": "Authorization"\n      }\n    ]\n  }\n}\n'
json = {'error': {'code': 401, 'errors': [{'domain': 'global', 'location': 'Authorization', 'locationType': 'header', 'messag....', ...}], 'message': 'Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.'}}
path = 'https://www.googleapis.com/storage/v1/b/solar-pv-nowcasting-data/o/Topographic%2Feurope_dem_1km_osgb.tif'

[JackKelly]

@jacobbieker Are you running this on your laptop? If so, have you run gcloud auth login at the command line?

[peterdudfield]

laptop, and yea ive used 'gcloud auth application-default login'

[jacobbieker]

I am, and I forgot to do that!

[peterdudfield]

latest error i get is

' # open the subdatasets if they exist

  if riods.subdatasets:

E AttributeError: '_GeneratorContextManager' object has no attribute 'subdatasets'

when using the method above

[JackKelly]

And, just to check, can you both access other OCF data on Google Cloud Storage from your laptops? Is this problem definitely specific to the topo data?

[peterdudfield]

filesystem = fsspec.open(self.filename).fs if not filesystem.exists(self.filename): raise RuntimeError(f"{self.filename} does not exist!")

I add this before, and this worked fine

[jacobbieker]

And, just to check, can you both access other OCF data on Google Cloud Storage from your laptops? Is this problem definitely specific to the topo data?

I can download the topo data using gsutil cp so I can access it, gcsfs just throws that error when trying to access the data when running in the same terminal

[jacobbieker]

I guess one way around it could be to download the file locally first, as it is fairly small, then use that cached file for it?

[JackKelly]

The issue that @peterdudfield linked to (corteva/rioxarray#246) seems to suggest that, yeah, if we want rioxarray to read data from a cloud storage bucket then we'll have to download the data locally before rioxarray will open it (because the underlying GDAL library is written in C++, and doesn't yet accept a Python file-like object... it needs a "real" file). I'll ask on that thread.

[JackKelly]

Hmm, yeah, I had a quick skim through the docs for rioxarray and for rasterio. I can't see any mention I could see of loading from cloud storage bucket. The only mention of loading over the network is loading cloud-optimised GeoTIFFs over HTTP(s).

So, yeah, I think we'll need to download the topo data locally and read from the local file (and only do this if we're reading from the cloud. When nowcasting_dataset is running on-prem, then it can happily read the topo data directly from disk).

But, I should add the caveat that I only skim-read the docs!