Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Conversation

peterdudfield
Copy link
Contributor

@peterdudfield peterdudfield commented Oct 11, 2021

Pull Request

Description

  1. Changed save to netcdf (batch) to save .nc files for each data source
  2. Adapt method to load from .nc files
  3. Add data source specifc method to save to .nc

Fixes #202

How Has This Been Tested?

Save netcdf files for batch in seperate folders

  • No
  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

Copy link
Contributor

@JackKelly JackKelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!


# make folder
folder = os.path.join(path, name)
make_folder(path=folder)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much time does make_folder take to run if folder already exists? (I ask because save_netcdf() will be called tens of thousands of times in one run of prepare_ml_data.py!)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps this should just happen on the first batch, i.e 'batch_idx ==0 '. Althought I dont think this will take very long

@peterdudfield peterdudfield merged commit 2d7d977 into main Oct 12, 2021
@peterdudfield peterdudfield deleted the issue/202-modality-files branch October 20, 2021 10:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use independent processes for each "modality"

2 participants