| Title | : | Efficient Video Classification Using Fewer Frames |
| Speaker | : | Shweta Bhardwaj (IITM) |
| Details | : | Mon, 1 Jul, 2019 10:30 AM @ A M Turing Hall |
| Abstract: | : | Recently, there has been a lot of interest in building compact models for video classification which have a small memory footprint. While these models are compact, they typically operate by repeated application of a small weight matrix to all the frames in a video. For example, recurrent neural network based methods compute a hidden state for every frame of the video using a recurrent weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD have a learnable clustering matrix which is used to assign soft-clusters to every frame in the video. Since these models look at every frame in the video, the number of floating point operations (FLOPs) is still large even though the memory footprint is small. In this work, we focus on building compute-efficient video classification models which process fewer frames and hence have less number of FLOPs. Similar to memory efficient models, we use the idea of distillation albeit in a different setting. Specifically, in our case, a compute-heavy teacher which looks at all the frames in the video is used to train a compute-efficient student which looks at only a small fraction of frames i.e., uniformly spaced k frames in the video. This is in contrast to a typical memory efficient Teacher-Student setting, wherein both the teacher and the student look at all the frames in the video but the student has fewer parameters. Our work thus complements the research on memory efficient video classification. We do an extensive evaluation with three types of models for video classification, viz., (i) recurrent models (ii) cluster-and-aggregate models and (iii) memory-efficient cluster-and-aggregate models and show that in each of these cases, a see-it-all teacher can be used to train a compute efficient see-very-little student. Overall, we show that the proposed student network can reduce the number of FLOPs by a significant margin with a negligible drop in the performance. As an additional analysis, we tried to explore the dynamic selection of frames based on the input video, which are different from uniformly spaced and relevant to the classification task. We empirically establish that the agent learns to select the video frames which lie within a small neighborhood of uniformly spaced frames. Hence, to do so under a strict budget of k frames, picking uniformly spaced frames seems to be an easier and efficient strategy to identify multiple video categories. |
