Can I use features extracted from I3D?

[htowa]

Hi.

Thank you for providing this useful feature!

My question is, is it possible to apply the feature map (.npy) extracted from I3D and check the accuracy?
The geometry extracted from I3D is as follows.
(n/16, 1024)
n/16 converts to 32 where n is the total number of frames in the video and 16 frames are one segment.
1024 is the feature size.

[ekosman]

Should be possible, but not implemented yet. I would welcome a pull request if you want to contribute, but probably don't have time to do it myself.

[htowa]

Should be possible, but not implemented yet. I would welcome a pull request if you want to contribute, but probably don't have time to do it myself.

https://github.com/v-iashin/video_features

I borrowed the I3D code from this URL and implemented it.
I haven't trained MIL yet, but the features were extracted and saved to a text file.
When normalizing, I3D set mean to [1, 1, 1] and std to [0, 0, 0], but is this correct?
I'm waiting to hear the reason for these values!

Also, when I actually extracted features using I3D and output them to a text file, they were saved at about 700KB in size regardless of the video playback time.
The length of the feature vector should differ depending on the playback time, so why are they the same size?

[ekosman]

For live streaming, that miht be true. But the original paper states a different training methodology. It slices the video into 32 segments no matter how long the video is. The features are averaged over each segment.

Can you contribute the feature extraction using I3D?

[ekosman]

Regarding mean and std, they might be wrong. Notice that the model is not implemented yet in the feature extraction script, so I assume the feature is not used and the values don't matter. Feel free to contribute and fix it.

[htowa]

ライブストリーミングの場合、それは真実かもしれません。しかし、元の論文では、別のトレーニング方法が述べられています。ビデオの長さに関係なく、ビデオを32のセグメントにスライスします。フィーチャは、各セグメントで平均化されます。

I3Dを使用した特徴抽出に貢献できますか?

For live streaming, that miht be true. But the original paper states a different training methodology. It slices the video into 32 segments no matter how long the video is. The features are averaged over each segment.

Can you contribute the feature extraction using I3D?

I understand that your code averages the features to 32 segments after feature extraction, so the fal size becomes uniform.

When I extracted features using I3D pre-trained on the Kinetics400 dataset, the accuracy was AUC:0.79.
It seems that the normalization method is such that mean and std are in the range of -1 to 1.

The following URL has been pointed out for I3D, which may be related to the accuracy.
URL: hassony2/kinetics_i3d_pytorch#32

Also, why do you apply L1 and L2 regularization to the loss function?

Also, when I run feature_extractor.py, I get the following warning. Does this warning affect accuracy?
UserWarning: The pts_unit 'pts' gives wrong results. Please use pts_unit 'sec'.
warnings.warn("The pts_unit 'pts' gives wrong results. Please use pts_unit 'sec'.")

[ekosman]

Also, why do you apply L1 and L2 regularization to the loss function?

The original code does it as well: https://github.com/WaqasSultani/AnomalyDetectionCVPR2018/blob/master/TrainingAnomalyDetector_public.py#L31

[ekosman]

Also, when I run feature_extractor.py, I get the following warning. Does this warning affect accuracy? UserWarning: The pts_unit 'pts' gives wrong results. Please use pts_unit 'sec'. warnings.warn("The pts_unit 'pts' gives wrong results. Please use pts_unit 'sec'.")

Should be fine as per: pytorch/vision#1931

[htowa]

また、feature_extractor.pyを実行すると、次の警告が表示されます。この警告は精度に影響しますか?UserWarning: pts_unit 'pts' は間違った結果をもたらします。「sec」pts_unitを使用してください。warnings.warn("pts_unit 'pts' は間違った結果をもたらします。pts_unit'sec'を使用してください。

pytorch/vision#1931に従って問題ないはずです

Nice

By the way, I think this method removes the frame, height, and width tensors after feature extraction and trains using only the segments and channels.

(32, 1024) ← After feature extraction

If you use a feature map that stores frame, height, and width information such as (1, 256, 8, 112, 112), can you ignore the operation to make it 32 segments?

[ekosman]

また、feature_extractor.pyを実行すると、次の警告が表示されます。この警告は精度に影響しますか?UserWarning: pts_unit 'pts' は間違った結果をもたらします。「sec」pts_unitを使用してください。warnings.warn("pts_unit 'pts' は間違った結果をもたらします。pts_unit'sec'を使用してください。

pytorch/vision#1931に従って問題ないはずです

Nice

By the way, I think this method removes the frame, height, and width tensors after feature extraction and trains using only the segments and channels.

(32, 1024) ← After feature extraction

If you use a feature map that stores frame, height, and width information such as (1, 256, 8, 112, 112), can you ignore the operation to make it 32 segments?

Sure, but this is not the method described in the original paper. This repo aims to mimic the method as precisely as possible.