add data_columns to doc string #13065

michaelaye · 2016-05-03T11:07:21Z

closes DOC: DataFrame.to_hdf() docstring does not mention data_columns #13061
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

How about this for #13061 ?
I don't think pandas has tests for each documented field, correct?
Do you want a whatsnew entry for this? If yes, how does that work? The guidelines don't talk about that.

jreback · 2016-05-03T12:59:02Z

I don't think pandas has tests for each documented field, correct?

not sure what you mean by this. pandas has tests for each option of course. doc-strings in the past were not audited as much as currently.

The doc-string should match for the same keyword args. Please change to match.

In [1]: pd.HDFStore.append?
Signature: pd.HDFStore.append(self, key, value, format=None, append=True, columns=None, dropna=None, **kwargs)
Docstring:
Append to Table in file. Node must already exist and be Table
format.

Parameters
----------
key : object
value : {Series, DataFrame, Panel, Panel4D}
format: 'table' is the default
    table(t) : table format
               Write as a PyTables Table structure which may perform
               worse but allow more flexible operations like searching
               / selecting subsets of the data
append       : boolean, default True, append the input data to the
    existing
data_columns : list of columns to create as data columns, or True to
    use all columns
min_itemsize : dict of columns that specify minimum string sizes
nan_rep      : string to use as string nan represenation
chunksize    : size to chunk the writing
expectedrows : expected TOTAL row size of this table
encoding     : default None, provide an encoding for strings
dropna       : boolean, default False, do not write an ALL nan row to
    the store settable by the option 'io.hdf.dropna_table'
Notes
-----
Does *not* check if data being appended overlaps with existing
data in the table, so be careful
File:      ~/pandas/pandas/io/pytables.py
Type:      instancemethod

jorisvandenbossche · 2016-05-03T13:11:03Z

pandas/core/generic.py

@@ -1084,6 +1084,9 @@ def to_hdf(self, path_or_buf, key, **kwargs):
                       / selecting subsets of the data
        append   : boolean, default False
            For Table formats, append the input data to the existing
+        data_columns : list of columns to create as data columns, or True to
+            use all columns. This will create additional indexed columns for
+            on-disk queries, by default only 'index' and 'columns' are indexed.


What is 'columns'?

The same columns as written in your docs? See here: http://pandas.pydata.org/pandas-docs/stable/io.html#querying-a-table

by default the axes of the object are indexed (e.g. index for Series, index and columns for DataFrame)

I see, it's just not really clear from the docstring (but it would also take to much space there to explain).

maybe it should be improved in the io docs?

michaelaye · 2016-05-04T22:30:55Z

Do you want me to expand the docstring in .append()? Or want it in .to_hdf() as short as it is in .append()
I cannot find anywhere to_hdf tests that test for it's individual options. Only mentioning of to_hdf I can find is here.

jreback · 2016-05-04T22:33:42Z

to_hdf just calls the underlying functions it's all tested directly on HDFStore. to_hdf is just a wrapper around this.

you can expand the .append doc string if u want and add that to to_hdf

michaelaye · 2016-05-04T22:52:51Z

I just noticed that columns in HDFStore.append() is not supported anymore:

        if columns is not None:
            raise TypeError("columns is not a supported keyword in append, "
                            "try data_columns")

Should the append=None not be removed then from the call signature?

jreback · 2016-05-04T22:56:05Z

@michaelaye what does columns have to do with append? (that is an older / deprecated kw)

michaelaye · 2016-05-04T22:58:14Z

That's how it looks like, yes. But it's still there, in the method signature, see above, where you copied it in yourself.

codecov-io · 2016-05-04T23:05:35Z

Current coverage is 84.14%

Merging #13065 into master will decrease coverage by <.01%

@@             master     #13065   diff @@
==========================================
  Files           138        137     -1   
  Lines         50587      50261   -326   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42593      42288   -305   
+ Misses         7994       7973    -21   
  Partials          0          0

Powered by Codecov. Last updated by b4e2d34...8a36c36

jreback · 2016-05-06T12:00:46Z

pandas/core/generic.py

@@ -1084,6 +1084,9 @@ def to_hdf(self, path_or_buf, key, **kwargs):
                       / selecting subsets of the data
        append   : boolean, default False
            For Table formats, append the input data to the existing
+        data_columns : list of columns to create as data columns, or True to


data_columns : list of columns, or True, default None ...explanation on next line

michaelaye · 2016-05-06T16:29:20Z

What about the obsolete "columns" keyword in the Store.append call()? Different problem I guess, but I don't really understand why to keep it? Not at least for backwards compatibility, because it raises a TypeError now, and so it would if the keyword would be removed and somebody still would try to use it.

jreback · 2016-05-06T16:34:07Z

its to provide a custom message with a hint, meaning columns was passed, but maybe you meant data_columns.

columns is a valid kw for .select, so people might try it with append.

jorisvandenbossche · 2016-05-11T11:58:41Z

This looks good to me. @jreback ?

jreback · 2016-05-11T12:28:28Z

pandas/core/generic.py

+        data_columns : list of columns, or True, default None
+            This will create additional indexed columns for on-disk queries,
+            by default only 'index' and 'columns' are indexed. True will index
+            all columns.


this is only true for a data frame (and not true for other objects); by default the axes are indexes. e.g. index and columns might be better.

i'm sorry, it's not clear what exactly you want changed how.

this is a generic doc-string that applies when you are looknig at Series,DataFrame, or Panel. so it should either be somewhat generic or specific to that type (which is harder and not worth it). You are saying 'index' and 'columns', which don't apply to Series/Panel.

How can we make this specific to Series,DataFrame and Panel? Because considering the complexity of pandas I find it highly important to have the docstrings as useful as possible.

I think in this case it might not be easy to do this because someone is doing:

``HDFStore.append, there isn't any object at that point, so you can just give an example e.g. Series -> 'index' is defined, DataFrame -> 'index' and 'columns' ,Panel -> 'items', 'major_axis','minor_axis'` (put them in bullet points).

So, for a Panel I actually can not add columns to the index?
If that's true, how about then I add the docstring like so:

data_columns : only applicable to DataFrames, see examples

and then I add the usage examples with the suggestions you made above?

jreback · 2016-05-25T16:01:38Z

can you rebase / update?

…pend()`

michaelaye · 2016-05-26T04:43:58Z

I see the problem now, maybe this is the reason why the docstring was left out in the first place? What should I do? I guess the data_columns option as actually being ignored if it would be used with a Series.to_hdf? I just tried to use it with a non-existing column name and no error appeared.

jreback · 2016-06-30T10:17:42Z

can you update

michaelaye · 2016-06-30T14:40:18Z

I repeat my previous question from May 25 (under the code discussion at the change overview), for which I still am waiting for guidance:

"So, for a Panel I actually can not add columns to the index?
If that's true, how about then I add the docstring like so:

data_columns : only applicable to DataFrames, see examples

and then I add the usage examples with the suggestions you made above?

jreback · 2016-07-01T11:37:45Z

@michaelaye I dont' think can use a shared doc here as this is a doc-string on the class (and not on an object yet). So put a more general doc-string should be enough (e.g. give an example for series, dataframe, panel for which axes you can use).

jorisvandenbossche · 2016-08-19T11:53:17Z

@michaelaye I cherry-picked your commit to combine it with the changes of the other merged PR

jorisvandenbossche · 2016-08-19T11:53:36Z

-> rebased in #14046

jreback added the Docs label May 3, 2016

jorisvandenbossche reviewed May 3, 2016
View reviewed changes

jreback reviewed May 6, 2016
View reviewed changes

jorisvandenbossche added this to the 0.18.2 milestone May 11, 2016

jreback reviewed May 11, 2016
View reviewed changes

DOC: add data_columns to doc string of df.to_hdf() and `HDFStore.ap…

9baf452

…pend()`

michaelaye force-pushed the fix_13061 branch from 8a36c36 to 9baf452 Compare May 26, 2016 04:20

jreback removed this from the 0.18.2 milestone Jun 30, 2016

jorisvandenbossche mentioned this pull request Aug 19, 2016

DOC: add data_columns to doc string of to_hdf and HDFStore.append #14046

Merged

jorisvandenbossche added this to the 0.19.0 milestone Aug 19, 2016

jorisvandenbossche closed this Aug 19, 2016

Uh oh!

add data_columns to doc string #13065

add data_columns to doc string #13065

Uh oh!

Conversation

michaelaye commented May 3, 2016 • edited by shoyer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback commented May 3, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelaye commented May 4, 2016

Uh oh!

jreback commented May 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelaye commented May 4, 2016

Uh oh!

jreback commented May 4, 2016

Uh oh!

michaelaye commented May 4, 2016

Uh oh!

codecov-io commented May 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 84.14%

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelaye commented May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback commented May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorisvandenbossche commented May 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jreback commented May 25, 2016

Uh oh!

michaelaye commented May 26, 2016

Uh oh!

jreback commented Jun 30, 2016

Uh oh!

michaelaye commented Jun 30, 2016

Uh oh!

jreback commented Jul 1, 2016

Uh oh!

jorisvandenbossche commented Aug 19, 2016

Uh oh!

jorisvandenbossche commented Aug 19, 2016

Uh oh!

Uh oh!

michaelaye commented May 3, 2016 •

edited by shoyer

Loading

jreback commented May 4, 2016 •

edited

Loading

codecov-io commented May 4, 2016 •

edited

Loading

michaelaye commented May 6, 2016 •

edited

Loading

jreback commented May 6, 2016 •

edited

Loading