Skip to content

Append, not overwrite learning_stats.csv when continuing training from checkpoint [minor request] #3176

@guptadivyansh

Description

@guptadivyansh

Is your feature request related to a problem? Please describe.
By looking at the (very useful) test metrics in learning_stats.csv, I decided to continue training my network for some more epochs. I wanted to plot the validation RMSE curves, but as soon as I restarted training, the csv only had the new epochs and the old metrics were gone.

Describe the solution you'd like
The CSV logger should just append more epochs as rows when training is continued, rather than creating (and thus overwriting) a 'new' file.

Describe alternatives you've considered
Parsing the lost metrics from the raw logs in 'train.txt'. Fortunately, 'train.txt' already opens the file in append mode, so the logging continues with retraining. The learning_stats.csv is obviously much nicer to parse though.

Additional context
dlc version 3.0.0rc13 using pytorch models.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions