Skip to content

> Not sure this is even supported by gcsfs - I don't see encoding as an available option there: #32391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
EgorBEremeev opened this issue Mar 2, 2020 · 1 comment

Comments

@EgorBEremeev
Copy link

Not sure this is even supported by gcsfs - I don't see encoding as an available option there:

https://github.com/dask/gcsfs/blob/523eb65b3e7feb05f9c10ce84523d1058716fecf/gcsfs/core.py#L1133

Might need to start upstream if wanted to make this possible

Hi, @WillAyd
In the current gcsf master GCSFileSystem.open() has been removed and fsspec.AbstractFileSystem.open() has works instead:

https://github.com/intake/filesystem_spec/blob/4c66e096d32dafe264e2d6707992ee6935685944/fsspec/spec.py#L717

where applying of passed encoding for the text reading\writing is now implemented:

        if "b" not in mode:
            mode = mode.replace("t", "") + "b"

            text_kwargs = {
                k: kwargs.pop(k)
                for k in ["encoding", "errors", "newline"]
                if k in kwargs
            }
            return io.TextIOWrapper(
                self.open(path, mode, block_size, **kwargs), **text_kwargs
            )

Note also this issue from gcsfs

So, it looks that in pandas ignoring of encoding parameter happens because in the pandas.io.gcs.get_filepath_or_buffer the mode = 'rb' is passed to call of GCSFileSystem.open(filepath_or_buffer, mode)

Tracing back to the moment of the first actual setting the mode parameter we have stop on this line:

pandas.io.common.py

def get_filepath_or_buffer(
    filepath_or_buffer, encoding=None, compression=None, mode=None
)

It is so, because in the call of get_filepath_or_buffer() performed from here

fp_or_buf, _, compression, should_close = get_filepath_or_buffer(

we do not pass value of mode and default mode=None works.

As I could suggest for read_csv() we need pass mode=r and for to_csv() we need pass mode=w in the call of get_filepath_or_buffer(). But I'm not sure where it's better to implement this change.

Originally posted by @EgorBEremeev in #26124 (comment)

@EgorBEremeev
Copy link
Author

Just open this by mistake

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant