Forcing all LHS variables of tilde to be part of model arguments

[yebai]

Currently, LHS variables of tilde statements can be optionally included in the model arguments. If they do appear, and non-nothing values are passed, these variables are viewed as observed.

Turing.@model function demo(y)
   x ~ Normal()
   y ~ Normal(x, 1)
end 

demo(1.) # y is data, x is parameter
demo(nothing) # both x and y are parameters
demo() # this will error

This was conceived as intuitive when I initially introduced it. However, it always felt a bit "magical" and not satisfactory to me, since the variable x is treated differently from y and involves complex code in DynamicPPL to handle it reasonably well. Even so, there is the well-known issue explained in #519. In addition, if x is an Array, it will get repeatedly re-allocated for every model evaluation, which is wasteful. These considerations motivate the following proposal:

Turing.@model function demo((;x, y)::NamedTuple, s)
   x ~ Normal()
   y ~ Normal(x, s)
end 

demo((;), 1.0) # option 1, both x and y are parameters
demo((;x=1.), 1.0) # option 2, x is observed, y is parameter
demo((;y=1.), 1.0) # option 3, y is observed, x is parameter
demo((;x=1., y=1.), 1.0) # option 4, both x and y are data
demo() # this will error
demo(1.0) # this will error since the first argument is not a namedtuple

Here, we explicitly require all LHS variables of tilde statements to be part of the model arguments. However, during the model construction step, the first argument could be a namedtuple containing a subset of LHS variables. These variables will be treated as data, similar to the current Turing syntax. If a variable is missing in the namedtuple, it can either be conditioned via the condition(...) method or left as a parameter.

[Red-Portal]

So is the first argument always reserved to be a NamedTuple?

[yebai]

It could be any NamedTuple or struct type that contains fieldnames referenced by the destructuring syntax (;x, y).

[mhauru]

In the example from the other issue, slightly modified,

@model function demo((;m, v, y), sigma)
    m ~ Normal(0, sigma)
    v = 2*y
    v ~ Normal(m, sigma)
end

Could I initialise this both as demo((; v=10.0), 1.0) and as demo((; y=5.0), 1.0)? What would happen if I did demo((; v=1.0, y=-1.0), 1.0)?

Oh also, writing that made me think of this:

@model function demo((;m, v, y), sigma)
    m ~ Normal(0, sigma)
    v = 2*y
    v ~ Normal(m, sigma)
    y ~ Normal()
end

demo((; v=10.0), 1.0)

Would this treat v as data and y as a parameter, severing the v = 2*y connection?

Also, more generally than just for this proposal, what do we want this to do?

@model function f((; x, y))
    x ~ Normal()
    y = 2*x
    y ~ Exponential()
end

[yebai]

@mhauru The variables, such as (;m, v, y), will be assumed to be immutable, i.e., they can only be assigned outside model log density functions.

There are variables with special values, such as nothing, and those that are lazily instantiated (e.g., variables created by particle Gibbs in varying dimensional models) will be treated specially.

As a result, statements like v = 2*y will trigger an error to be thrown, because v is part of (;m, v, y) and we assume they are immutable during model log density calls.

[mhauru]

In that case, what would be the right way to write a model like

    m ~ Normal(0, sigma)  # assume
    v = 2*y
    v ~ Normal(m, sigma)  # observe

where y is data we might condition on and we want to transform it somehow to make v, before calling the observe statement on v?

[yebai]

 m ~ Normal(0, sigma)  # assume
v = 2*y
v ~ Normal(m, sigma)  # observe

In this case, v should be computed outside the model and passed as a constant to the model. This would also help performance, as transformed data remains constant and should ideally be moved out of model definitions.

As I remember correctly, this design pattern is very similar to those in functional programming languages and the BUGS language; i.e., each variable is assigned only once.

Does that make sense?

[mhauru]

How would we forbid this sort of reassignment? We still have to allow for things like

v = Vector{T}(undef, 5)
v ~ MvNormal(...) # or v .~ Normal(), or v[i] ~ Normal()

, right? I'm unsure how to distinguish that from the sort of case we'd like to forbid.

[mhauru]

Throwing more ideas in the mix: Could there be merit to turning this on its head, and saying that no parameter values/data can ever be provided in the model arguments?

The rule would be that the set of parameter names is the set of all symbols that appear as the symbols of VarNames on the LHS of ~ statements. So in a model that has

v = Vector{T}(undef, 5)
for i in eachindex(v)
    v[i] ~ Normal()
end
y ~ Normal()
x.a ~ MvNormal()

the parameter names would be v, y, and x. This set can be created at macro time. Having any of those parameter names appear as an argument to the model function would be an error. The idea is to force everyone to use condition/decondition, which resolves the usual case of #519. [*]

I may not have thought this through, but I wonder if this could be useful. It would solve the issue raised by someone somewhere (might have been on Slack) that function signatures would get really big if you have a lot of variables. It would also allow the simplicity of writing models that we currently have, where you can just do @model function demo() x ~ Normal() end. But it would still resolve some of the confusion of #519, and also allow us to treat all conditioned variables in a unified way.

For users to provide more information about the structure of the VarInfo object, or to preallocate variable arrays, like with @parameters/of, we could then add a syntax distinct from model arguments. Maybe something like a_function_that _i_dont_know_how_to_name!(my_model_constructor, MyParameterType) would specify that models instantiated with my_model_constructor() should use MyParameterType, which would have been constructed with something like of. This would make it clear that this is only about providing structure for the parameters, not about providing values. All values would have to go through condition.

[*] I say it resolves "the usual case" of #519 because I still don't quite know what to do with this:

    x ~ Normal()
    y = 2*x
    y ~ Exponential()

[yebai]

I expect that the above example will need to be written as

@model function demo((;m, v), sigma)
    m ~ Normal(0, sigma)
    v ~ MvNormal(m, ...)
end

demo((; m=missing, v=Vector{Missing}(undef, 5)), 1.0)

LHS variables of title statements can no longer be local variables. This has numerous advantages, such as avoiding naming clashes in the model (i.e., VarInfo) and also avoiding the repeated allocation of v for every model evaluation.

It is worth noting that the above example passes all LHS variables to the model constructor demo((; m=missing, v=Vector{Missing}(undef, 5)), 1.0). I expect this can be simplified in certain cases.

(I saw Markus posted a new message while I type this response)

[yebai]

Throwing more ideas in the mix: Could there be merit to turning this on its head, and saying that no parameter values/data can ever be provided in the model arguments?

I've also thought about this idea.

Note that my proposal here is not really about whether we want to use condition to replace model arguments. Instead, the point is to

(1) assume all LHS variables passed as model arguments, implying they are "global" inside the model body
(2) assume the shape information of LHS variables is passed to the model, or inferred automatically, where it is proven safe.

These properties improve the clarity of the model semantics, i.e. a mapping from input arguments to a log density function (other model evaluations are omitted without loss of generality).

[mhauru]

Might there ever be a situation where a user would want to make observe statements, where the observable data has dependency on other parameters? Maybe something like this (using our current syntax)

@model function f(data)
    x ~ Normal()
    modified_data = somefunction(data, x)
    modified_data ~ Normal()  # This is an observe statement.
end

I know we don't currently support this (except using @addlogprob!), but even though I don't have a use in mind, this seems like something that might naturally emerge.

[yebai]

Might there ever be a situation where a user would want to make observe statements, where the observable data has dependency on other parameters? Maybe something like this (using our current syntax)

That would not lead to an invalid statistical model in the general case.

[mhauru]

Sorry @yebai, do you mean would not lead to a valid model? I'm unsure if I'm reading the double negation correctly. Is there a concise explanation for why it would be invalid, or some keyword I can search for to understand this better?

[yebai]

oops, sorry for the confusion. Your interpretation is correct; it would lead to invalid models. A simple example suffice to demonstrate the pathological behaviour

skipfirst(data, x) = x

@model function f(data)
    x ~ Normal()
    modified_data = skipfirst(data, x)
    modified_data ~ Normal()  # This is an observe statement.
end

You can see variable x appears twice on the LHS of tilde statements, once as a parameter, and the other as data. Although these models might be useful in specific cases, they are relatively uncommon. So it is an advantage that the new syntax detects and forbids it, and only allows experienced users to implement them using addlogprob!.