Add augmentation for segmentation/annotation

[liavassif]

Segmentation labels (annotation) are usually provided as a matrix of labels (integers).
For this task the same augmentation as the images should be performed, except interpolation should be 'nearest neighbours' rather than bilinear or similar one.
Some form of retrieving the sampled parameters (for example: rotation angle was 32 degrees) would be usefull so we could re-create the same augmentation, but with different interpolation.
Deterministic augmentation doesn't help in this case.
In the meantime, as a workaround we can use:

from imgaug import augmenters as iaa
rotate_max=30
seq = iaa.Sequential([
    iaa.Fliplr(0.5), # horizontally flip 50% of the images
],random_state=0)
rotate=iaa.Affine(rotate=(0, rotate_max),mode='edge')
rotate_nn_interpolation=iaa.Affine(rotate=(0, rotate_max),order=[0],mode='edge')
seq_nn_interpolation=seq.deepcopy()
seq.append(rotate)
seq_nn_interpolation.append(rotate_nn_interpolation)

aug_seed=0
for batch in...:
    batch_images = images[batch, :, :, :]
    batch_labels = labels[batch, :,:]
    seq.reseed(aug_seed)
    seq_nn_interpolation.reseed(aug_seed)
    batch_images=seq.augment_images(batch_images)
    batch_labels=seq_nn_interpolation.augment_images(batch_labels)
    aug_seed+=1

[aleju]

Something like a history might be added in the future, not sure yet.

Without having tested it, you might also do something like the following, which I think should work (though just reseeding might be faster):

from imgaug import augmenters as iaa
from imgaug import parameters as iap

rotate_max = 30
seq = iaa.Sequential([iaa.Fliplr(0.5), iaa.Affine(rotate=(0, rotate_max), order=[0, 1], mode="edge" name="Affine")])

for batch in batches:
    seq_det = seq.to_deterministic()
    seq_det_cp = seq_det.deepcopy()
    seq_det_cp.find_augmenters_by_name("Affine")[0].order = iap.Deterministic(0)
    images_aug = seq_det.augment_images(batch.images)
    segmentations_aug = seq_det_cp.augment_images(batch.segmentations)
    ...

Note that rotate=(0, rotate_max) will rotate images by 0° to rotate_max°, not -rotate_max° to rotate_max°.

[liavassif]

Thanks about the rotation range info.

I've tried something along the deterministic approach but regardless of using augmentation for segmentation or not, I would like every batch augmentation to possibly be different (between batches). I guess some of the other helper functions ('Sometimes' etc.) could be usefull but a simple reseed() seems OK for now.

However if the number of augmentors between the segmentations and images are different (for example if there is an augmentor which is disabled for the segmentations but enabled for the images) I guess the non deteministic approach breaks (due to sampling more numbers from the random number generator when augmenting images). In that case does the deterministic approach help?

[aleju]

Each augmenter can have its own random state, which is independent from all other random states. By default that one points to the global random state, but if you call reseed(), then that should create a unique random state for each augmenter.
Though, if you have different numbers of augmenters in your Sequential, then reseed will likely create different random states, as it samples automatically a new seed per augmenter (based on its input parameter seed). If your augmenter numbers are the same and you seed them with the same value, your results should always be the same (for images of the same height, width and number of channels).

[liavassif]

Thanks. So for the case of different number of augmenters I guess some sort of history/sampled values/used seeds could be usefull.

[EpochalEngineer]

@aleju another common case is the desire to do appearance augmentations, of which should only occur on the pixel data, not the label image. What is the correct way to handle this case, or is it currently not possible?

If one defines an augmentation for the images with appearance+geometric augmentation sequences, and one for the labels of just appearance, reseeding would yield different random states, as you mention.

Deepcopying augmentation sequences would cause the appearance based augmentations to modify the per-pixel label image. Would it work to not only modify the order, but also all the appearance parameters to effectively be no-op? E.g. for GaussianBlur, to set the sigma 0? Would it also be necessary to stay within the same parameter type? E.g. sigma=Custom StochasticParameter -> sigma=iap.Uniform(0,0) and not sigma=0?

EDIT: I noticed #52 does this by defining different sequences not based on images and labels, but rather on ones to apply to both and only images. Then the ones applied to both could be modified as above

[aleju]

Do you mean that you want to define your augmentation sequence once, apply all augmenters to one input, but only some augmenters to another? In that case you can use hooks to dynamically deactivate some augmenters during execution. Something like:

seq = iaa.Sequential([
    iaa.GaussianBlur((0.1, 2.0), name="GaussianBlur"),
    iaa.Dropout(0.02, name="Dropout"),
    iaa.Affine(rotate=(-35, 35), name="Affine")
])

# change the activated augmenters for heatmaps,
# we only want to execute horizontal flip, affine transformation and one of
# the gaussian noises
def activator_heatmaps(images, augmenter, parents, default):
    if augmenter.name in ["GaussianBlur", "Dropout"]:
        return False
    else:
        # default value for all other augmenters
        return default
hooks_heatmaps = ia.HooksImages(activator=activator_heatmaps)

seq_det = seq.to_deterministic() # call this for each batch again, NOT only once at the start
images_aug = seq_det.augment_images(images)
heatmaps_aug = seq_det.augment_images(heatmaps, hooks=hooks_heatmaps)