Add context to return from pandas.io.html.read_html

[cancan101]

Currently pandas.io.html.read_html returns a list of DataFrames. This offers no context as to where in the source HTML the table was found. For example, a user might be interested in the title or caption of the table.

For example in an SEC 10-Q filing (see for example: http://apps.shareholder.com/sec/viewerContent.aspx?companyid=GMCR&docid=9277772#A13-6685_110Q_HTM_UNAUDITEDCONSOLIDATEDSTATEMENTSOF_223103), there are many tables. The user might be interested in one or more of them. They are not returned in a "standard" order such that without the title provided in the HTML document, finding the desired table in the return list is difficult.

My suggestion is to offer some way of linking the returned table to the context in which it was found.

[cpcloud]

I think this will probably have to wait for NDFrame metadata to be implemented, so that it (HTML metadata) could be done correctly.

[jreback]

you could return maybe a dict of frames where key is the 'name'?

[cpcloud]

I guess but problems abound when there's no name...I guess then it could be an integer index.

[cpcloud]

or append to a list like {'table_title': df, None: [df1, df2, df3]}

[cancan101]

@jreback I was thinking about using a dict somehow. I am not sure what would be a meaningful "name" in this case or how I would provide it.

@cpcloud Is this what you mean by NDFrame: http://wesmckinney.com/blog/?p=77 ?

[jreback]

I guess you could use a 'name' regex? (and if it matches, use it, otherwise put them in order?)

[cpcloud]

@cancan101 No, sorry I should have been more clear. NDFrame is just the base class of (soon-to-be) Series, DataFrame and Panel

[cpcloud]

table elements can have titles and captions i think...i don't remember what the preferred tag attribute is tho i have to check it out

[cancan101]

I was asking what would be the best way for me to provide a function whose job is to annotate the table that is found? i.e a function that maps the Table element to a name.

[cpcloud]

okie doke turns out it's just <caption> which is the title

[cancan101]

As in would that be a new parameter to pandas.io.html.read_html

[cpcloud]

well first you need to parse that information, currently that element is ignored...

@cancan101 yes that would be a new argument probably called caption='a regex to match strings in the caption'

[cpcloud]

@cancan101 btw you can save yourself a few keystrokes by just using pandas.read_html. no need to refer to the internal name ... this is a top level function 😄

[cancan101]

I think it would need to be a function rather than a regex.

[cpcloud]

why a function?