API: user api for new sql functionality

[jorisvandenbossche]

Starting a new issue to discuss this. See merged PR #5950 and follow-up issue #6292.

Summary of situation:

Current (0.13):

• top-level read_sql, read_frame, write_frame and DataFrame.to_sql
• in sql module: uquery, tquery, has_table (+ read_sql)

New after #5950

• top-level read_sql and DataFrame.to_sql (but not yet used in the docs)
• in sql module: read_sql, read_table, to_sql, has_table, execute
• read_frame, write_frame, uquery, tquery are deprecated

Points to dicsuss

• docs: use top-level functions read_sql and DataFrame.to_sql instead of functions from sql
• sql.read_table vs existing pd.read_table. Is this a problem?
• merge functionality of read_table and read_sql or not (see this comment of @mangecoeur to not do that: ENH: sql support via SQLAlchemy, with legacy fallback #5950 (comment))

[jreback]

I don't recall if we test deprecations in general (e.g. call the function in an assert_produces_warning wrapper)

[jorisvandenbossche]

This is I think easy to agree on, to be consistent with the other io modules:

• DataFrame.to_sql(...) instead of sql.to_sql(df, ...)
• pd.read_sql instead of sql.read_sql

But

• what to do with read_table? (which is, at this moment how it is laid out in the docs, sort of the primary function, http://pandas-docs.github.io/pandas-docs-travis/io.html#io-sql)
  • rename and also put toplevel, eg pd.read_sqltable
  • just keep as is in sql (but this feels odd because, as I said above, this is somewhat the main function of the sql module).
  • rename read_table to toplevel pd.read_sql (the main function), and the current read_sql to something like pd.read_sqlquery or sql.read_query or sql.query(), sql.get_query. But I assume this is not possible due to backwards compatibility.
  • merge read_sql and read_table to one function (but see comments here ENH: sql support via SQLAlchemy, with legacy fallback #5950 (comment))

@hayd @mangecoeur @jreback @danielballan @y-p Opinions?

[jreback]

it seems that you can use read_table if u can reflect the table definition
otherwise u have to use read_sql and then figure out types

well it seems that pandas should then try to use read_table if possible (eg try to reflect the table) then fallback to read_sql?

why is this this a problem or not possible?

it is really confusing to have 2 functions which basically do the same thing depending on an implementation detail

[mangecoeur]

@jreback I addressed some of the reasons here, and some more reasons below
#5950 (comment)

I don't see this as confusing at all, and the functionality is distinct - you have low level SQL querying and higher level "table selection" apis.
You can always reflect a table if you have it's name, so there's no need for read_sql fallback in that sense.

On the other hand, if you issue a read_sql, you will be using an SQL query, that may not return values stored in a db TABLE - e.g. you might do something like
"SELECT COUNT(tbl1.foo), SUM(tbl2.bar) FROM tbl1, tbl2 WHERE tbl1.otherID == tbl2.ID GROUP BY tbl2.groupable_name"
And none of the returned values will be data stored in the DB, instead they are calculated on the fly. You can't reflect type information before hand, so you can't rely on the same type conversion.

If you used read_table in the case you only wanted one table (and we'll see below why that's not straightforward) you would suddenly find you had different type casting rules, because instead of the rules that apply to read_sql (limited type coertion) you get the ones that apply to read_table. You should never create that sort of unexpected behaviour.

In any case, how would you know if you were only selecting one table? You would have to parse the SQL string to find out that "SELECT * FROM table_name" (and it's functionally equivalent alternatives, of which there are a few) meant you want everything from table_name. Which means you need to introduce an SQL parser and to deal with the differences between SQL dialects, which completely defeats the point of using SQLAlchemy.

Next, there are keyword arguments which have meaning for read_table but not for read_sql, and vice-versa. E.g. "flavor" is meaningless for read_table, "columns" is likewise for read_sql. If you tried to merge the functions these options would have a different effect depending on the value of the input arg - that to me is really bad API design, it's as if the accelerator pedal of your car turned into the brake pedal depending on whether you are wearing a hat. It's just nuts!

Having 2 functions which represent different levels of abstraciton from raw SQL completely eliminates these problems. It leaves lots of room to add interesting funcitonality to read_table and to take advantage of the higher level of abstraction offered by SQLAlchemy's expression language.

All we need to do is figure out nice API names. If a top level function is required (though IMHO it's not that important) I think read_sql_table or read_db_table as aliases of sql.read_table would be fine. If it weren't for backward compatibility isses I would prefer even read_table to be called read_sql and the old read_sql to be query_sql or similar.

[jreback]

@mangecoeur ok...I hear ya...so to summarize, read_sql returns a table (that may or may not be defined but pandas will try extra hard to infer dtypes, while the currently namedread_table`` will essentially perform arbitrary sql and not do anything special with the result. ok

The big question here is whether to expose the currently named read_table at the top-level (I think its ok)....but with what name.

Since we are breaking.....why don't we use the names you suggest

I actually think the top-level names ARE important as most users will simply use those by default. In general you don't get too many diving into the modules (nor should they necessarily). If these are both useful, then they can both be top-level.

read_sql_table and read_sql_query; can simply map read_sql to read_sql_table (with a deprc) so things will 'work'.

[hayd]

I had the same thought: sep to read_sql_query and read_sql_table (I think this will be used more).

@jreback Isn't it the other way around? read_sql_table takes a table_name and infers dtypes based on the dtypes of the sql table. read_sql_query takes arbitrary sql and has a go at infering ?

I still think there is a case for read_sql to do read_sql_table(sql) if has_table(sql) else read_sql_query(sql)... I don't think we should shy away from lots of (optional) args, we certainly don't elsewhere!

Definitely the functionality of sql_read_table should be toplevel.

[jreback]

@hayd oh...I see I did put them 'backwards'. Ok if we go with read_sql_query and read_sql_table then map read_sql to read_sql_table (maybe with a deprecate on read_sql directly)....everyone ok with that?

seems minimum pain

[jorisvandenbossche]

@mangecoeur About the top level function: that's the design choice made earlier in pandas, and I think it is important to be consistent on this: the main read_.. function in top level namespace.

Indeed a good summary: there are two cases for reading sql data:

• full table (with currently read_table) -> infer types from table definition via sqlalchemy types
• custom sql query (with currently read_sql) -> try to infer types just from result set

And due to the potential differences between the results of both methods, it is indeed important to clearly identify those two cases in the docs (whether as two cases in one function, or two cases in two functions).

For me it is ok to go with read_sql_query and read_sql_table.
But @jreback, what do you mean with mapping read_sql to read_sql_table? Because isn't that just the problem with the backwards compatibility that read_sql already exists and has a different behaviour? But if this deprecation is OK, then we actually can just use read_sql instead of read_sql_table as the name for the function that reads whole tables (unless we find read_sql_table better as it is more explicit).

[jreback]

I think that I am refering to these top-level functions reversed. so ignore me!

[jorisvandenbossche]

Something else, @mangecoeur, is it also the intention that users could use one of the objects? (in the meaning: has this an added value, so that it would be usefull to mention it in the docs)

Because now, in the docs there is a mention of PandasSQLWIthEngine (just above this header: http://pandas-docs.github.io/pandas-docs-travis/io.html#querying), but I think this is certainly outdated and maybe not your intention of being in the docs? (possibly a leftover from the rebasing complexity, there were some merging conflicts in the docs that I manually, and maybe incorrectly, resolved).

[mangecoeur]

@jorisvandenbossche well spotted, that should be updated to read PandasSQLAlchemy, and maybe also to include PandasSQLTable. But the OO API probably still needs work anyway. It's very useful if you need more fine control for certain use cases (I use it for debugging to check the create table statements before doing an insert for example)