Tests for @non_differentiable

[willtebbutt]

It's ocassionally not possible to test @non_differentiable using test_rrule because finite differencing gives the wrong / unreliable results.

For example, this is the case in the Vector{Floa64}(undef, 5) constructor implemented in JuliaDiff/ChainRules.jl#491 -- in this case, the values of the array produced are unreliable, so finite difference estimates of the gradient aren't consistent between different calls to the function.

One option would be to define a test_non_differentiable suite that looks something like

test_non_differentiable(f, args...; use_finite_differencing=true)

in which we prescribe a set of standard tests. In the case that the user sets use_finite_differencing=false, we could just ensure that f does indeed return NoTangents, in addition to standard things like ensuring that the primal agrees with the first output of the rrule being tested.

[oxinabox]

We mostly just don't test them at all right now, because the implementation is trivial, and if it is wrong it is probably a bug in ChainRulesCore.@non_differentiable.
Sometimes we test them if we want to be sure one is defined, or that it has the right sig.

To test that we never want to do finite differencing, so I think a test_non_differentiable would never do finite differencing.
If that is wanted test_rrule and test_frule exist, and will correctly test someting non-differentiable, if it is indeed finite-difference-able

[willtebbutt]

Sometimes we test them if we want to be sure one is defined, or that it has the right sig.

Agreed -- I think this + avoiding regressions is the value in such a test.

To test that we never want to do finite differencing, so I think a test_non_differentiable would never do finite differencing.
If that is wanted test_rrule and test_frule exist, and will correctly test someting non-differentiable, if it is indeed finite-difference-able

Fair enough. Then I guess the functionality that we would want would always just be to test agreement with f and that NoTangents are returned?