Avoid keeping test set in memory unnecessarily

[evancofer]

A test set is created whenever "train" is included as an op and test_holdout is specified in the config file. A test set should only be used and kept in memory when "evaluate" is listed as an op in the config file, as the test set will only be evaluated by a call to evaluate in TrainModel (which subsequently calls _evaluate_on_data(self._test_data)). Unfortunately, this test set is kept in memory throughout training, regardless of whether the test set is actually used (i.e. irrespective of whether "evaluate" is included as an op in the config file). I believe that it would be optimal to avoid keeping the test set in memory unnecessarily (i.e. delete it if evaluate() is never going to be called). To test the possible improvements that such a change would make, I ran case 2 from the manuscript examples with "train" as the only op specified in the config file. When I set self._test_data = None in TrainModel, it decreased memory usage by about 10 GB.

[evancofer]

There are two simple fixes for this. In either case, we must pass a parameter to the TrainModel indicating if a test set is to be used for evaluation after training.

The first solution would be to add a function that allows us to explicitly call for the destruction of the data associated with testing. It is more complicated than it sounds. I considered only deleting the test data held by TrainModel, but this does little to reduce the memory overhead. As such, it is necessary to delete the sampler's information about the test data as well. To do this, we need to implement many sampler-specific methods for deleting loaded test data.

The second approach is to never create the test set, and not allow saving of a test set without testing the model. This is by far the simplest approach, and doesn't require any modifications of the sampler classes.

[kathyxchen]

I'll make this modification in my next PR - TrainModel will not create self._test_data until the evaluate method is called.