Skip to content

Feature Request: Allow no required predict sets for a Measure #1094

@sebffischer

Description

@sebffischer

Some mlr3 Measures don't require a prediction object to operate on, such as msr("selected_features") or msr("internal_valid_score").

Let's say I want to resample xgboost with early stopping and I only care about the internal validation scores (e.g. in the context of tuning). I could then avoid making predictions on the test set alltogether. What I want to do is therefore:

l = lrn("classif.xgboost", nrounds = 100,
  early_stopping_rounds = 10, validate = "test", predict_sets = NULL)
  
rr = resample(tsk("iris"), l, rsmp("holdout")

rr$aggregate(msr("internal_valid_score"))

The reason why the code above does not run is because each measure must have at least one required predict sets because of this check:

self$predict_sets = assert_subset(predict_sets, mlr_reflections$predict_sets, empty.ok = FALSE)

I.e., for MeasureInternalValidScore I need to pick "test", "train" or "internal_valid", even though none are required.
The last call to $aggregate() will therefore err, no matter how I define the MeasureInternalValidScore.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions