Can't list buckets

[jseabold]

I have some AWS credentials for which I can't list buckets. I almost never have bucket level permissions, but I do have permissions on prefixes. So something like

fs = s3fs.S3FileSystem(profile_name='user')
fs.ls('s3://home/sseabold/prefix')

Won't work because I can't list any buckets (S3Client.list_buckets fails), and I also don't generally have permissions at the bucket level.

Similar issues to those here [1, 2] for the default (old) boto behavior.

[1] conda/conda#2126
[2] blaze/odo#448

[martindurant]

I don't yet have a solution for this, we might indeed have to change the way that ls works.
In the meantime, you can still access details of individual files using info() and open them.

[jseabold]

It looks like info has the same issue with bucket permissions.

[martindurant]

Apologies, seems we only implemented the direct info() for the S3File object, i.e,. one you have already opened. Would info() on a given file, sidestepping the ls solve this problem for you?

[jseabold]

Ah, I suppose. I was just looking at info for the first time. My main use is for list. No worries. s3 ACLs are a pain.

[jseabold]

It seems that all of my errors that I've dug in to are coming down to things that assume I can list the bucket. The most recent dask error is because of the list_objects paginator [1]. It tries to list the objects on the bucket rather than the prefix that I asked for in dask.DataFrame.read_csv.

I don't have my head completely around things here yet and the paginator stuff is new to me, but my files are in a bucket that has a lot of objects and I'm not interested in any of them. I only want the keys in the prefix I'm asking for, which I have permissions for and there are only a few. The dask error comes because I called read_csv on an object from a single key and it's listing the whole bucket AFAICT.

[1] https://github.com/dask/s3fs/blob/master/s3fs/core.py#L256

[jseabold]

I wonder if you shouldn't also pop out the prefix from the path in the S3FileSystem._ls and pass that along for pagination. I assume this is why all of the AWS SDK stuff is very particular about trailing slashes to identify prefixes vs keys.

Happy to test any changes.

[martindurant]

It is a valid model.
The initial reason for wanting to list the whole bucket rather than the prefix/delimiter model, is that every list call is pretty slow, so caching once is much faster for nested key structures. Let me consider, and could either have using prefixes could be optional, or the caching scheme a bit smarter.

[jseabold]

Sure that makes sense.

Just as another data point, as you noted the list call is slow, and a typical bucket I use could take a very, very long time to list, assuming I even could. I've run into this before just on a prefix in this bucket, and I ended up killing the operation I was doing. I worry a bit about the cost of trying to make this key-value store too filesystem like, but that's up to you. I think users will understand that s3 is not a filesystem and that there are limitations here.

Sorry to bother about all of this. I generally get pretty far with boto(3) myself, but now that dask is and pandas is going to use s3fs, I'm having to rewrite utility functions to make things work for me, and I want to raise the issues I'm going to run into.

[martindurant]

Your help in identifying issues is totally appreciated, no apology required!

[kennethreitz]

I am also running into this, trying to use s3contents with a restricted bucket with an IAM that doesn't allow BucketListing.