Triplet Image Dataset

[dakshjotwani]

Hi,

I am currently working with semi-supervised learning to implement face recognition. I required a dataset that could load triplets (three images, where the first two belong to the same class, and the last image belongs to a different class. The facenet paper has more details on this.) for training. Currently, I have implemented this dataset for my own use. I was wondering if this is a feature that would be nice to have as part of the library itself.

Thanks,

Daksh

[fmassa]

Hi,

I believe triplet (or n-uple) datasets could potentially be handled generically, with the pairwise/triplet/etc combinations generated on the fly, and making the dataset n ** 3 in size (with some potential pruning on the undesired cases).

Thoughts?

[dakshjotwani]

That sounds like a good idea!

We will, however, need to come up with some sort of strategy to select valid k-tuples. The triplet I described has a specific format: the first two entries belong to the same class, while the last entry belongs to a different class. One of the ways we can solve this problem is by allowing the user to pass in a filter function as an argument, which we can then use to select valid k-tuples. How does that sound?

The other problem is that we will have to go through n ** k potential tuples from the dataset to construct the k-tuple dataset, which is quite large and could take forever to run. To avoid this, my current approach to generate triplets is to pick them randomly from the dataset:

classes = ['class_1', 'class_2', ..., 'class_m']
class_imgs = {'c_1': ['/path/1', '/path/2', ...], ...}
num_triplets = 10000

samples = []
for _ in range(num_triplets):
    c_1, c_2 = np.random.choice(classes, size=2, replace=False)
    img_1, img_2 = np.random.choice(class_imgs[c_1], size=2, replace=False)
    img_3 = np.random.choice(class_imgs[c_2])
    triplet = (img_1, img_2, img_3)
    samples.append(triplet)

This approach solved my problem, but may not be suitable for the generic case that you proposed. What do you think?

[fmassa]

@dakshjotwani I'd probably want to have the dataset triplets to be deterministic, at least for torchvision.

So randomly sampling the triplets on a given idx is not something I was expecting to do, because it breaks the property that dataset[idx] gives you always the same thing

[fmassa]

Another option would be to make it an IterableDataset, instead of a standard dataset.
In this case, I would be ok having the __getitem__ implement what you proposed.

[dakshjotwani]

@fmassa In my current implementation, dataset[idx] == samples[idx], so it will be deterministic for each object instance. However, two instances of the triplet dataset acting on the same directory will most likely not be equal at the same index (dataset1[idx] != dataset2[idx]). Would this be fine for a VisionDataset or would this still be an IterableDataset?

I'm not sure if it's practical to go through all possible k-tuples, since that would require going through n ** k potential tuples. Maybe I do not see what you mean by 'generated on the fly'?

[fmassa]

@dakshjotwani that's what I mentioned wrt it not being deterministic: two different runs will have different values, and are indeed different datasets.

About the second point:
If we pass a groups list to the constructor of TripletDataset(dataset, groups), which contains either False or True, and each element is sampled such that there is True, True, False, then all we need to do is to go over the list during the constructor. This is still n ** k, but depending on the structure of groups, it might be possible to have it done with an equation instead of having to generate the triplet list in memory.

But this might just be a better fit for the IterableDataset as I mentioned, and it would definitely simplify a few things in the implementation

[dakshjotwani]

Alright, I agree that this dataset should be an IterableDataset, and that it would make implementing it much easier as well. I have some final questions regarding implementation before I get to implementing it:

1. Should I start with TripletDataset or implement KTupleDataset?
2. If KTupleDataset, what kind of args should I ask for to specify the format? Currently I was thinking of something like KTupleDataset(path, format) where format is a dictionary which looks like this:

format = {
    'class_0': {
        'num_samples': 3,
        'replacement': False,
    },
    'class_1': {
        'num_samples': 2,
        'replacement': True,
    },
    ...
}

The __iter__ method of the KTupleDataset above will return tuples of the form (c0, c0, c0, c1, c1, ...).
3. Since it will be an IterableDataset, I'm assuming that num_samples will not have to be specified in the constructor? The __iter__ method would just return the next randomly generated triplet from the dataset directory.
4. Currently there is no pytorch equivalent of np.random.choice. Would it be okay to use this method in my implementation?

[fmassa]

@dakshjotwani I think that it might make sense to start with TripletDataset. I'm not clear on what the semantics of a more general dataset should be, while the TripletDataset is more standardized I'd say

3 - I think it might be good to specify a num_samples in the constructor, so that we know when to raise a StopIteration in __iter__.
4 - There is a PR in pytorch/pytorch#18624 that adds support for choice. But I'd say that you should just start with using torch.multinomial for now, and do the indexing yourself.

Also, ccing @ssnl , as he was the one who originally designed IterableDataset, and might have some more insights on what would be good to have.

[dakshjotwani]

Sounds good. I look forward to what @ssnl has to say about our TripletDataset outline.

[ssnl]

This is a bit of an interesting use of IterableDataset. This would make the dataset unusable with any sampler, but it probably won't be too big of an issue. There is a lot of subtlety with using an IterableDataset with num_workers > 0, but if you only want to sample with replacement, then only taking care of the num_samples should be enough. That said, if this is going to be put in a general purpose library, e.g., torchvision, please be careful to make it work correctly. Feel free to tag me on the PR for review.

[rwightman]

Use cases for generic tuple datasets likely exist, but re motivation for the original task, I've never used an n-tuple dataset (n**k is not pleasant) with siamese or triplet loss fn. I've always used hard or semi-hard mining and more typical dataset that returns one element at a time but has a custom sampler to make sure each batch has the desired examples per identity.

I think most methods for identity recognition, face or otherwise since the FaceNet era have moved to online hard or semi-hard example mining for the triples and pairs. Or don't use them at all if they're using the various softmax + geometric loss like arcface, cosface, ring loss, etc etc.

Edit: This is a great blog post on the topic https://omoindrot.github.io/triplet-loss ... also, forgot my paper history, but FaceNet was actually the paper that introduced online selection and recommended not building a dataset that did offline triplet selection

[fmassa]

@rwightman yes, I agree that hard-negative mining is something that we should add support for.

But performing hard negative mining on a large dataset on-the-fly is very expensive computation-wise.
So what is generally done is that the features are computed once every few epochs, and the hard-mining is done on those features.

This means that we can re-create the dataset once every few epochs or so, by updating the similarity matrix that is provided to it.

This would still kind of match with the approach in #1061 (but some changes would need to be done).