Sphinx doesn't render autodoc_docstring_signature correctly w/ Python 2

[mengxr]

Subject: Some pyspark.ml docs were generated without rendering the docstring signature when Python 2 is used.

Problem

See https://issues.apache.org/jira/browse/SPARK-24530. We use autodoc_docstring_signature in PySpark to render some ML class docs correctly. A correct version looks like the following:

And when it doesn't render correctly, it shows

We tried different Python/Sphinx versions and it seems it only appears with Python 2.

Procedure to reproduce the problem

# use python 2
git checkout git@github.com:apache/spark.git
cd spark/python/docs
make html
open _build/html/pyspark.ml.html

Check the generated doc of LogisticRegression

Error logs / results

Didn't notice any suspicious warning messages (-v -w warn.txt).

Reproducible project / your project

apache/spark

Environment info

• OS: macOS
• Python version: 2
• Sphinx version: v1.6.3

[tk0miya]

Okay, I'll take a look later.

[tk0miya]

inspect.getargspec() can't recognize its argument well. It seems some decorator hides its arguments.

$ python
Python 2.7.15 (default, May  1 2018, 22:52:16)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark.ml.classification import LogisticRegression
>>> import inspect
>>> inspect.getargspec(LogisticRegression.__init__)
ArgSpec(args=['self'], varargs='args', keywords='kwargs', defaults=None)

AFAIK, there are no way to obtain original signature of the methods in python 2.x.
So I feel it is hard to fix this.

[HyukjinKwon]

Hi, @tk0miya. Thanks for checking this out. I was hesitant to leave a comment and I am glad that we had similar investigations. Yup, the cause looked indeed functools.wraps fails to copy its signature in Python 2 whereas Python 3 succeeds, and I agree there could be no way to find out the original signature in this case.

One thing I am wondering is though, I thought we could override the signature by autodoc feature when the feature is enabled and the first line of docstring specifies its signature.

[tk0miya]

It seems the first line of the docstring is blank. Unfortunately, the signature is defined at next line. So autodoc_docstring_signature does not work.

$ python
Python 3.6.5 (default, Mar 31 2018, 01:44:52)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark.ml.classification import LogisticRegression
>>> LogisticRegression.__init__.__doc__.splitlines()[0]
''
>>> LogisticRegression.__init__.__doc__.splitlines()[1]
'        __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction",                  maxIter=100, regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True,                  threshold=0.5, thresholds=None, probabilityCol="probability",                  rawPredictionCol="rawPrediction", standardization=True, weightCol=None,                  aggregationDepth=2, family="auto",                  lowerBoundsOnCoefficients=None, upperBoundsOnCoefficients=None,                  lowerBoundsOnIntercepts=None, upperBoundsOnIntercepts=None):'

Could you check your docstring please?

[HyukjinKwon]

Ah, thank you so much @tk0miya. Will give some updates here after checking and fixing it within Spark side

[tk0miya]

PEP-257 expects multiline-docstring starts just after triple-quotes like this.

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

So the rule of autodoc_docstring_signature is not strict to me.

[HyukjinKwon]

Hmmm .. but

def complex(real=0.0, imag=0.0):
    """
    Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

this also complies PEP 257:

The summary line may be on the same line as the opening quotes or on the next line.