sparkmagic ERROR:Cannot parse object as JSON

[twterry]

%%sql
select a,count(1) from test_table group by a

An internal error was encountered.
Please file an issue at https://github.com/jupyter-incubator/sparkmagic
Error:
Cannot parse object as JSON: '[u'{"a":1,"count(1)":5}', u'{"a":-1,"count(1)":6}', u'']'

[karth295]

I hit this as well with an even simpler repro:

%%sql
select 1

An internal error was encountered.
Please file an issue at https://github.com/jupyter-incubator/sparkmagic
Error:
Cannot parse object as JSON: '[u'{"1":1}', u'']'

Environment:

• Spark 2.2.1, Java 8, Hadoop 2.8.4
• Livy 0.5.0
• pip install sparkmagic (resolved to 0.12.5)
• Had to downgrade pandas due to pandas 0.23.0 causing error #458

[karth295]

Looks like the same error as in #445

[RobertAron]

I'm having this issue as well. It seems to only effect the spark kernel and not the pyspark one.

val someDF = Seq(
  (8, "bat"),
  (64, "mouse"),
  (-27, "horse")
).toDF("number", "word")

%%spark -o someDF

An internal error was encountered.
Please file an issue at https://github.com/jupyter-incubator/sparkmagic
Error:
Cannot parse object as JSON: '['{"number":8,"word":"bat"}', '{"number":64,"word":"mouse"}', '{"number":-27,"word":"horse"}', '']'

Has this issue been addressed yet? I can't seem to find any issues talking about it.

[Tagar]

We have same issue with pyspark not just spark

[dmnapolitano]

Same issue here with pyspark notebooks, sorry to say. For reference:

%%info

Current session configs: {'conf': {'spark.yarn.appMasterEnv.PYTHONUSERBASE': '<dir>', 'spark.yarn.appMasterEnv.PYSPARK_PYTHON': '/usr/lib/anaconda3/bin/python3'}, 'kind': 'pyspark'}

%%sql -o df
select 1, 2, 3

An internal error was encountered.
Please file an issue at https://github.com/jupyter-incubator/sparkmagic
Error:
Cannot parse object as JSON: '['b\'{"1":1,"2":2,"3":3}\'']'

Python 3.6.5, Java 8 (OpenJDK) on CentOS 7.4. I'd be happy to help out however I can. Thanks 😄

[te14017]

Same issue as me.
I looked into the source code and try to execute the same constructed command as in _scala_command function in https://github.com/jupyter-incubator/sparkmagic/blob/634ee0d356b8e9685fe006739b7543149cfef374/sparkmagic/sparkmagic/livyclientlib/sqlquery.py:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val clientSp = sqlContext.sql("SELECT * FROM clientSpTable")

clientSp.toJSON.collect.foreach(println)

It prints result correctly without format error. Now I am searching for the source or convertion error.

[mhuang74]

Filtering out empty string elements from array after split in utils.records_to_dataframe seems to have fixed the problem for me:

def records_to_dataframe(records_text, kind, coerce=None):
    if records_text in ['', '[]']:
        strings = []
    else:
        strings = records_text.split('\n')
        # fix https://github.com/jupyter-incubator/sparkmagic/issues/471 by removing empty string elements from array
        strings = filter(None, strings)
    try:
        data_array = [json.JSONDecoder(object_pairs_hook=OrderedDict).decode(s) for s in strings]

[alexmnyc]

Same here

[itamarst]

The code now has a .strip() which should strip leading/ending newlines, which probably explains why I can't reproduce this. So going to say it's fixed. Please re-open if you're still having issues.