Document derivative operator and notations

[sagetrac-schymans]

Taking the derivative of a symbolic function returns the D-notation:

sage: var('x y z')
sage: f(x) = function('f',x,y,z);
sage: f(x).diff(x,y)
D[0, 1](f)(x, y, z)

Unfortunately, the meaning of this notation is not documented anywhere, neither in diff(), nor in derivative() nor in function(). There is a ton of tickets about improving ambiguities and malfunctions related to this notation, but it would be very helpful to at least document how it is supposed to work and what it means if a user sees output as above.

See here for related tickets:

• Typesetting partial derivatives in new symbolics #6344 - allow typesetting in "diff" format (possibly only as non-default option)
• Implement diff format symbolic derivative in new symbolics #6756 - add input to Sage in "diff" format for derivatives (the most controversial)
• Failure with diff(f(x),x).subs(f(x) == g(x)) #6480 - clarify or fix substituting functions inside of symbolic derivatives
• Derivative at a point is not translated into Maxima #7401 - bug in our interaction with Maxima with evaluating derivative at a point (needs work due to multivariate derivatives not being there)
• Allow more general evaluation of FDerivativeOperator #12796 - allow evaluation at points

and these discussions:

• sage-devel, 2012-04, D notation input for ODEs
• sage-devel, 2009-07, Is new symbolic derivative really worth the efforts?

CC: @kcrisman @eviatarbach @slel

Component: symbolics

Issue created by migration from https://trac.sagemath.org/ticket/17445

[nbruin]

comment:1

OK, I do not know where it should go in the documentation for best visibility (I'd think in "diff" somewhere), but the explanation should be along the lines of:

Partial derivatives are represented in sage using differential operators, referencing the position of the variable with respect to which the partial derivative is taken. This means that for a function f in $r+1$ variables we have

[Di_1,\ldots,i_n(x_0,\ldots,x_r) = \left.\frac{\partial f(t_0,\ldots,t_r)}{\partial t_{i_1}\cdots \partial t_{i_n}} \right|_{t_0=x_0,\ldots,t_r=x_r}]

An advantage of this notation is that it is clear which derivative is taken, regardless of the names of the variables. For instance, if we have

sage: var("x,y,t")
sage: f(x,y)=function('f',x,y)
sage: g=f(x,y).diff(x,y); g
D[0, 1](f)(x, y)
sage: g.subs(x=1,y=1)
D[0, 1](f)(1, 1)
sage: g.subs(x=t,y=t+1)
D[0, 1](f)(t, t+1)

Note that in the last two lines are completely unambiguous using operator notation, whereas Leibnitz notation would require the use of some arbitrary explicit choice of auxiliary variable names.

[kcrisman]

Description changed:

--- 
+++ 
@@ -7,10 +7,10 @@
 Unfortunately, the meaning of this notation is not documented anywhere, neither in diff(), nor in derivative() nor in function(). There is a ton of tickets about improving ambiguities and malfunctions related to this notation, but it would be very helpful to at least document how it is supposed to work and what it means if a user sees output as above.
 
 See here for related tickets:
-https://github.com/sagemath/sage-prod/issues/6344
-https://github.com/sagemath/sage-prod/issues/6480
-https://github.com/sagemath/sage-prod/issues/6756
-https://github.com/sagemath/sage-prod/issues/12796
+* #6344
+* #6480
+* #6756
+* #12796
 
 and this discussion:
 https://groups.google.com/forum/#!topic/sage-devel/_xD5lymnTuo

[kcrisman]

comment:2

Are there times where this notation is ambiguous, though? I seem to recall that being the case.

Also, I think something about this should show up at the top of the "sage/calculus/calculus.py" file as well. It's a constant source of questions I don't really know the answer to.

[nbruin]

comment:3

Replying to @kcrisman:

Are there times where this notation is ambiguous, though? I seem to recall that being the case.

I am not aware of any such cases. It's reflecting all information that sage has stored on the object, so an ambiguity would imply that sage is working with an ill-defined object.

If you rewrite it in Leibnitz notation using a set of auxiliary variables, you see that any expression has a clear interpretation.

Go ahead and put it in the docs!

[kcrisman]

comment:4

Can you comment on some of the chain-rule type issues in #6480, then? I have to say that in particular the stuff at http://ask.sagemath.org/question/9932/how-to-substitute-a-function-within-derivatives/ and #6480 is massively confusing. Heck, let's add #7401 while we're at it.

I don't even know whether any of those things are "really" right or wrong at this point. I suppose you shouldn't be allowed to substitute in a function that "isn't there" in 6 and 7, but then why does 8 "work"? In any case, shouldn't there be an error raised if one attempts something like this when it's not "legitimate"?

# 6. Fails.
x = var('x')
f = function('f', x)
g = function('g', x)
p = f.diff()
print p.substitute_function(f, g) # Outputs "D[0](f)(x)"

# 7. Fails.
x = var('x')
f = function('f', x)
g = function('g', x)
p = f.diff()
print p.substitute_function(f(x), g(x)) # Outputs "D[0](f)(x)"

# 8. Works.
x = var('x')
f = function('f')
g = function('g')
p = f(x).diff()
print p.substitute_function(f, g) # Outputs "D[0](g)(x)"

These are very subtle differences to anyone who is not in symbolic algebra/expressions, and part of the issue is the difference between expressions and functions, no doubt. So comment:1 is a good start, but definitely only a start.

[kcrisman]

Description changed:

--- 
+++ 
@@ -11,6 +11,7 @@
 * #6480
 * #6756
 * #12796
+* #7401 (similar topic, not same issue)
 
 and this discussion:
 https://groups.google.com/forum/#!topic/sage-devel/_xD5lymnTuo

[kcrisman]

comment:5

Okay, at least now I actually understand what the different tickets are about. Phew.

[kcrisman]

Description changed:

--- 
+++ 
@@ -7,11 +7,11 @@
 Unfortunately, the meaning of this notation is not documented anywhere, neither in diff(), nor in derivative() nor in function(). There is a ton of tickets about improving ambiguities and malfunctions related to this notation, but it would be very helpful to at least document how it is supposed to work and what it means if a user sees output as above.
 
 See here for related tickets:
-* #6344
-* #6480
-* #6756
-* #12796
-* #7401 (similar topic, not same issue)
+* #6344 - allow typesetting in "diff" format (possibly only as non-default option)
+* #6756 - add *input* to Sage in "diff" format for derivatives (the most controversial)
+* #6480 - clarify or fix substituting functions inside of symbolic derivatives
+* #7401 - bug in our interaction with Maxima with evaluating derivative at a point (needs work due to multivariate derivatives not being there)
+* #12796 - allow evaluation at points
 
 and this discussion:
 https://groups.google.com/forum/#!topic/sage-devel/_xD5lymnTuo

[nbruin]

comment:6

Replying to @kcrisman:

I don't even know whether any of those things are "really" right or wrong at this point.

There is an internal logic that explains the behaviour. I don't think this stuff can ever be made "intuitive" to the average calculus afficionado because the tradition in analysis notation is just to irredeemably confuse "function" and "function evaluated at...". Humans can handle this confusion to some extent, but I think it's just too incompatible with how computers represent math.

# 6. Fails.
x = var('x')
f = function('f', x)
g = function('g', x)
p = f.diff()
print p.substitute_function(f, g) # Outputs "D[0](f)(x)"

the problem: f is not a function, but a function evaluated at x, whatever x is. I'm not going to defend that function('f', x) doesn't return a function. THAT is the real source why this example seems confusing.

sage: f
f(x)

The fact that f.diff() doesn't fail is apparently a heuristic, because there's only one variable discernible in the expression. But then you see:

sage: p
D[0](f)(x)

as you can see, there's no f(x) appearing in that expression, so of course substituting f(x) for something else has no effect. Really,

p.substitute_function(f, g)

should give a TypeError, because the types of the arguments don't match. Indeed:

sage: p.substitute_function(f.operator(), g)
DeprecationWarning: Substitution using function-call syntax ...

Apparently sage does try to convert the argument g(x) (that is bound to g) to a function.

# 7. Fails.
x = var('x')
f = function('f', x)
g = function('g', x)
p = f.diff()
print p.substitute_function(f(x), g(x)) # Outputs "D[0](f)(x)"

same problem, but even worse: f(x)(x) should have been deprecated already.

# 8. Works.
x = var('x')
f = function('f')
g = function('g')
p = f(x).diff()
print p.substitute_function(f, g) # Outputs "D[0](g)(x)"

The fact that function('f') and function('f',x) return different types of arguments is really bad. It prevents any convincing explanation of the distinction that is essential here, because the interface itself confuses the two different issues. On top of that, we have a lot of documentation that was written by people who were equally confused, so the calculus doc makes a good effort to confuse any new user too.

These are very subtle differences to anyone who is not in symbolic algebra/expressions, and part of the issue is the difference between expressions and functions, no doubt. So comment:1 is a good start, but definitely only a start.

It's an entirely different issue, though. Apparently the documentation never explains what D[0](f)(x) means. We can do that independent of whether we go out of our way to confuse users about the distinction between functions and functions evaluated at ...

[eviatarbach]

comment:7

It would probably be good to add a note that D[0, 1](f)(x, y) does not in general equal D[1, 0](f)(x, y) (https://en.wikipedia.org/wiki/Symmetry_of_second_derivatives).

[sagetrac-schymans]

comment:8

Replying to @nbruin:

The fact that function('f') and function('f',x) return different types of arguments is really bad. It prevents any convincing explanation of the distinction that is essential here, because the interface itself confuses the two different issues. On top of that, we have a lot of documentation that was written by people who were equally confused, so the calculus doc makes a good effort to confuse any new user too.

Wow, the reason while I hesitated about adding documentation of the D-notation was that I thought it should go into the documentation of function() and then I found out that the documentation of function() itself is already incomplete and confusing. Should we open a ticket for that, too?
For example, none of the methods described in http://www.sagemath.org/doc/reference/calculus/sage/symbolic/function_factory.html show up when I type:

function?

The distinction between

f = function('f')

and

f = function('f', x)

is also not documented.
Only now I realised that function('f') returns a function, while function('f',x) returns an expression. Furthermore, differentiation of a function is not supported, it needs to be converted to an expression first:

f = function('f')  
fx = function('f', x) 
print type(f)
print type(fx)
print type(f(x))
print fx.diff()
print f(x).diff()
print f.diff()
///
<class 'sage.symbolic.function_factory.NewSymbolicFunction'>
<type 'sage.symbolic.expression.Expression'>
<type 'sage.symbolic.expression.Expression'>
D[0](f)(x)
D[0](f)(x)
Traceback (most recent call last):    print f(x).diff()
  File "", line 1, in <module>
    
  File "/tmp/tmpMe44Jp/___code___.py", line 9, in <module>
    exec compile(u'print f.diff()' + '\n', '', 'single')
  File "", line 1, in <module>
    
AttributeError: 'NewSymbolicFunction' object has no attribute 'diff'

It's an entirely different issue, though. Apparently the documentation never explains what D[0](f)(x) means. We can do that independent of whether we go out of our way to confuse users about the distinction between functions and functions evaluated at ...

If we modify the documentation of diff(), though, we should explain why f(x).diff() works but f.diff() does not.