-
Notifications
You must be signed in to change notification settings - Fork 7.1k
fix eurosat prototype mock data setup #5549
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
a5b4f2d
ac5a25f
6847414
3b9b6e9
5d427fe
3062c71
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -1329,20 +1329,20 @@ def cub200(info, root, config): | |||||||||||||||||
|
||||||||||||||||||
@register_mock | ||||||||||||||||||
def eurosat(info, root, config): | ||||||||||||||||||
data_folder = pathlib.Path(root, "eurosat", "2750") | ||||||||||||||||||
data_folder = root / "2750" | ||||||||||||||||||
data_folder.mkdir(parents=True) | ||||||||||||||||||
|
||||||||||||||||||
num_examples_per_class = 3 | ||||||||||||||||||
classes = ("AnnualCrop", "Forest") | ||||||||||||||||||
for cls in classes: | ||||||||||||||||||
categories = ["AnnualCrop", "Forest"] | ||||||||||||||||||
for category in categories: | ||||||||||||||||||
create_image_folder( | ||||||||||||||||||
root=data_folder, | ||||||||||||||||||
name=cls, | ||||||||||||||||||
file_name_fn=lambda idx: f"{cls}_{idx}.jpg", | ||||||||||||||||||
name=category, | ||||||||||||||||||
file_name_fn=lambda idx: f"{category}_{idx + 1}.jpg", | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you explain how the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It doesn't, but it aligns the mock data more with real data. I just found this while looking into the real data to debug this issue and didn't want to have an extra PR for it. |
||||||||||||||||||
num_examples=num_examples_per_class, | ||||||||||||||||||
) | ||||||||||||||||||
make_zip(root, "EuroSAT.zip", data_folder) | ||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. By using Although not correct, that wouldn't be an issue if weren't for a quirk on Windows and macOS: vision/torchvision/prototype/datasets/utils/_resource.py Lines 92 to 99 in e836b3d
is picking up the empty There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Is this a bug, or a documented "feature" ? Are we likely to hit more bugs in the future because of this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not a bug nor documented, since this is part of system. Neither macOS nor Windows support case sensitive files. IMO the only places where we might hit this is in the test suite if the mock data is setup wrong. A scenario like above cannot happen during normal operation unless a user manually manipulates the data folders. |
||||||||||||||||||
return len(classes) * num_examples_per_class | ||||||||||||||||||
return len(categories) * num_examples_per_class | ||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
@register_mock | ||||||||||||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -53,6 +53,8 @@ def test_sample(self, test_home, dataset_mock, config): | |
|
||
try: | ||
sample = next(iter(dataset)) | ||
except StopIteration: | ||
raise AssertionError("Unable to draw any sample.") from None | ||
Comment on lines
+56
to
+57
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you explain how this relates to the fix? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It makes this case more expressive. Before you got an error message like |
||
except Exception as error: | ||
raise AssertionError("Drawing a sample raised the error above.") from error | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come the linux tests were passing before with the extra
eurosat
folder?I would assume that if we change the structure in the , this would have to be reflected somehow in the dataset code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the prototype datasets never look at the absolute path, but only relative starting from the right
vision/torchvision/prototype/datasets/_builtin/eurosat.py
Line 38 in 27745e5
adding more folders on the left side makes no difference.