-
-
Notifications
You must be signed in to change notification settings - Fork 95
Description
Some mlr3 Measures don't require a prediction object to operate on, such as msr("selected_features")
or msr("internal_valid_score")
.
Let's say I want to resample xgboost with early stopping and I only care about the internal validation scores (e.g. in the context of tuning). I could then avoid making predictions on the test set alltogether. What I want to do is therefore:
l = lrn("classif.xgboost", nrounds = 100,
early_stopping_rounds = 10, validate = "test", predict_sets = NULL)
rr = resample(tsk("iris"), l, rsmp("holdout")
rr$aggregate(msr("internal_valid_score"))
The reason why the code above does not run is because each measure must have at least one required predict sets because of this check:
Line 147 in 3f159d9
self$predict_sets = assert_subset(predict_sets, mlr_reflections$predict_sets, empty.ok = FALSE) |
I.e., for MeasureInternalValidScore
I need to pick "test"
, "train"
or "internal_valid"
, even though none are required.
The last call to $aggregate()
will therefore err, no matter how I define the MeasureInternalValidScore
.