MLP Regression: make testing easier

[weefuzzy]

As it stands, we don't make it easy to evaluate the performance of regression against test data. This is a shame, because it makes it harder to make judgements about how well a network is going to perform in practice.

I propose a two-part remedy:

1. A way to split a dataset into (shuffled) training and testing sets. sk-learn does this with a function. We could either add a message to dataset, or a dedicated object.
2. A way to retrieve the MSE for a supervised prediction. This is a pain to do in the CCE. Suggest adding methods to the regressor test <dataset inputs_in> <dataset outputs_in> <dataset loss_out> and testPoint <buffer input_in> <buffer output_in> -> double. These take two inputs (i.e. training pairs) and report the loss. (Although, should the batch version just return a double as well, viz. the mean loss across the whole set? Maybe that makes more sense)

In this way we should be able to enable a more principled workflow for trickier examples where we can monitor the test loss alongside the training loss.

paging @g-roma @jamesb93 @tedmoore @tremblap

[jamesb93]

1. I was certain this should be a message to fluid.dataset~ and I was wrote out my justification I reneged internally. If the test train split is configurable does that mean new attributes to fluid.dataset~ too?

Something like split test train which makes two new datasets~ with our data split into test/train. Kinda sucks if you also now need to jiggle attributes on the dataset~ (potentially clearing it while patching).

Alternatively some object like fluid.testtrain~ where you set the params point it an input and an output like so:

split <input> <test> <train>

Not sure if it should conform to fit/transform terminology although thats not what it is is doing is it.

2. I think returning the loss across the whole set and scrapping the buffer output makes sense to me... although I am uninitiated. It should feel similar to calling fit imo, where we get the number on the other side and and work from there.

[g-roma]

I think it would be good for the dataset to have a split method, and possibly also a shuffle method. There is code in SGD for suffling. A separate object could make it easier to have K-fold splits, e.g. split in 10 and return one of the splits.
For evaluation a separate object would be better because the same measures would serve for different regressor algorithms. A test for classifiers would be good as well, I am not sure if they would belong together, one option is that it could guess the type of test from the types (DataSet vs LabelSet) but I am not sure this is possible in the client.

[weefuzzy]

If we had a separate object are you envisaging we pass the regressor to it as well, or that we train (a bit), call predict on our test set and pass the result of this along with the desired truth to the evaluator to calculate the loss?

Being able to dispatch at runtime on the type of container is tricky. The obvious way to do it would be to handle std::variant arguments, then the client dispatches through a visitor. Something for the very-future, I feel.

[g-roma]

It would be more flexible to do it from prediction and ground truth, you could obtain predictions in different ways, combine models or whatever. I would definitely not tie it to the model.

[weefuzzy]

Architecturally, I'm with you, it's definitely more flexible. Pedagogically, I have misgivings if we insist on extra patching / coding to perform an important step.

Is there not a case for following sklearn's example again, and having both? The sklearn MLPRegressor has a score method (which returns the R² not the MSE, but whatevs), and then a bunch of free functions in sklearn.metrics for calculating a range of different things.

[tedmoore]

For splitting into test and train, I think @jamesb93 suggestion of fluid.testtrainsplit~ is best. It feels like the other objects in that one passes in a dataset and it is "transformed" in a way and written to a different dataset (two in this case). Of course one will have to make sure that the MLP's inputs and outputs are shuffled and split the same, which is why I think it needs its own object rather than a method call.

~x_data = FluidDataSet(s); // has some stuff in it
~y_data = FluidDataSet(s); // has some stuff in it
~x_train = FluidDataSet(s);
~y_train = FluidDataSet(s);
~x_test = FluidDataSet(s);
~y_test = FluidDataSet(s);
~splitter = FluidTrainTestSplit(s,0.8); // split pct

~splitter.transform(~x_data,~y_data,~x_train,~y_train,~x_test,~y_test,{
"done splitting".postln;
});

This feels very clear to me.

For testing, I agree with @weefuzzy misgivings on adding too much extra for this. I like the sklearn's score method.

[tremblap]

For point 1 of the OP, I have a question about how to make a representative sample. Just shuffle and take 20% seems blunt, no? We mostly work in 'small data' world where it is easy to get the wrong sample, no?

[tremblap]

For point 2 of the OP, I'm not sure if we couldn't do an object to do that. IIUC, we are trying to get the individual loss for each item of a dataset, then to get a score. These are 2 steps and we could probably add a MLP message to get the error for each input output loss_out (which is great) then people can send loss_out to a scoring object or dump or look or whatever, that feels more flexible. Who knows what one can do with a matrix of error? I could decide to take the item with the worse prediction for instance...

[tedmoore]

Not sure how useful it is to drop here, but I thought I would add to the conversation that since this discussion, this has been added to the learn web site:

https://learn.flucoma.org/learn/training-testing-split/