1. Introduction

Video action recognition is an important function in the realm of computer vision with its applications including automated surveillance, self-driving vehi-cles and drone navigation. Convolutional neural net-works (CNNs) have become standard of image classi-fication[1,2]. Two-stream architecture CNN [3-5] has been extremely popular for action recognition ex-ploiting RGB frames and optical flow as input stream then combining its feature to produce a final result. There are many models based on two-stream network such as optical flow guided feature(OFF) [13], hidden two-stream convolutional networks (H-TSCN) [14], and Actionflownet[15].

The optical flow computation for the action infor-mation in video is realized by calculating the dis-placement of the objects between each pair of ad-jacent frames. Each action in video frames lasts from 5 to 30 frames or longer in most video dataset. As the frame increase in size the computational power of optical flow to process frame iterations increases. Subsequently, conventional optical flow process be-comes a time consuming process with increase in la-tency and diminished function in real-time applica-tion.

The purpose of the study is to design and evaluate a novel CNN for action recognition based on two-stream network of teacher-student architecture [16]. There are two sub-networks in our neural networks, the optical flow sub-network as a teacher and the RGB frames sub-network as a student. In training stage, we trained the feature of the optical flow to teacher sub-network, and then transmitted the feature to student sub-network as a baseline to train the student sub-network. In the test stage, we only used the student sub-network to reduce latency eliminating computing optical flow.

2. Related Work

There are some significant progress with CNNs in computer vision tasks like object classification[2,17] and object detection [18]. When a large video datasets are published such as UCF101, HMDB51 [11,12], the CNNs are applied in action recognition [3,19,20].

2.1 Two-Stream Network

Traditional two-stream network [3] proposed by Si-monyan et al. is a 2D CNN model with RGB frames and optical flow sub-networks. It uses video clips as the input stream, and decompose the clip into RGB frames. Then a stream of RGB frames functions as a spatial component while stream of optical flow calcu-lates adjacent RGB frame as its temporal component. The spatial part carries appearance information about the objects, while the temporal part carries the move-ment information. Each stream is implemented by a sub-network and the result is combined by late fusion. In a research of two-stream network, Feichtenhofer et al. improved fusion of the two streams [4]. They ini-tially focused on 2D CNNs, but transitioned to 3D CNNs for improved spatiotemporal features. Simi-larly, Diba et al. used C3D to learn motion from opti-cal flow for end-to-end applications [5].

2.2 Teacher-Student Architecture

There are various designs of teacher-student net-works [6-9]. The teacher-student architecture consists of two parallel CNNs, a large and a small models [6]. Traditionally, the large model has many nodes and parameters than the small one and often results in better outcome. The small model can process small dataset with fast result. In real-world applications, the large model spends costly resources and time, while the small network needs less resources, it only fits for little data for fast result. The distillation can transfer the knowledge learned by large model to small one, and the small model can use the distillation to access the problem and gain fast and accurate results. In process of transferring the distillation, the large mod-el acts as a teacher and the small model as a student. In recent research, Kong et al. proposed the single learning student network to tackle the challenges of learning student networks with few data [7], and Bashivan et al. designed teacher guided architecture to gain more computational efficiency [8].

3. Proposed Method

Fig. 1.

Architecture of Action Recognition

Fig. 2.

M1: The Module of Optical Flow Extraction

Our model is based on two-stream network and teacher-student architecture [16] as shown in Fig. 1. We used video clips as input stream and extracted the RGB frames to feed the teacher-student network. There are two sub-networks in our architecture, teacher sub-network for optical flow branch and the student sub-network for RGB frames branch. We divided the pro-cessing of network into two stages. In a training stage, we calculated the optical flow stream in teacher sub-network to possess important motion information. Then information is used to train the network for action re-cognition and then freeze on its finalized weights. We then used the knowledge of optical flow to train the student sub-network. In the test stage, we only used stu-dent sub-network with RGB frames stream to avoid optical flow computation to save resources and time.

3.1 Teacher Sub-network

There are two modules in the teacher sub-network, the optical flow extraction module is named as M1 and action recognition module as M2. The module M1 cal-culates optical flow from the sequential RGB frames. The module M2 extracts the features of optical flow.

In the module M1, we don't employ complex net-works [21,22] to compute optical flow, its extraction is based on the brightness consistency assumption be-tween the sequential images I1 and I2, with small change in moment of the object. The approximating optical flow is formulated as shown in Equation (1).

[TeX:] $$I_{1}(x, y)$$ denotes the object at the location [TeX:] $$(x, y)$$ of the image at time [TeX:] $$t$$. [TeX:] $$I_{2}(x, y)$$ is the object at the loca-tion in the image after time [TeX:] $$\Delta t$$, and [TeX:] $$(\Delta x, \Delta y)$$ is the object spatial pixel displacements in [TeX:] $$x$$ and [TeX:] $$y$$ axes, respectively. It can be approximated with a Taylor series as shown Equation (2).

Zach et al. proposed [TeX:] $$\text { TV-L }{ }^{1}$$ method [10,11] to cal-culate the optical flow approximately. The total var-iational method estimates the optical flow by an iter-ative optimization method. The tensor [TeX:] $$u \in R^{2 \times W \times H}$$ is the [TeX:] $$x$$ and [TeX:] $$y$$ directional optical flow for each location of the object in the image. The method first computes the gradient in both [TeX:] $$x$$ and [TeX:] $$y$$ directions: [TeX:] $$\nabla I_{2}$$. The ini-tial optical flow is set to [TeX:] $$u=0$$. The [TeX:] $$\rho$$ denoting the image residual between [TeX:] $$I_{1}$$ and [TeX:] $$I_{2}$$, the iterative opti-mization is performed with updating [TeX:] $$u, v, p$$ as shown in Equation (3)-(6).

where hyper-parameter [TeX:] $$\theta$$ is the weight of the [TeX:] $$\text { TV-L }{ }^{1}$$ regularization term, [TeX:] $$\lambda$$ is weight of the data term, [TeX:] $$\tau$$ is the time-step and [TeX:] $$\tau \leq \frac{1}{8}$$. The dual vector field [TeX:] $$p$$ is used to minimize the energy.

We computed the optical flow using formula mentioned above via residual networks as shown in Fig. 2.

The module M2 extracts the features of optical flow denoted as [TeX:] $$F_{\text {of }}$$ in Fig. 3. Subsequently, we trained the teacher sub-network to classify actions using optical flow stream with a cross entropy loss function be-tween the predicted class labels [TeX:] $$P_{\text {of }}$$ and the true class labels T. Finally, the [TeX:] $$F_{\text {of }}$$ is transmitted to the student sub-network to train its network by back pro-pagation.

Fig. 3.

M2: The Module of Optical Flow Feature Extraction

Fig. 4.

The Module of RGB Frames Action Recognition

3.2 Student Sub-network

The module M3 in student sub-network extracts the feature of RGB frames as [TeX:] $$F_{\text {rgb }}$$, and receives the fea-ture of optical flow from the teacher sub-network as [TeX:] $$F_{\text {of }}$$ shown in Fig. 4.

We used the loss function of Mean Squared Error (MSE) on both [TeX:] $$F_{\text {rgb }}$$ and [TeX:] $$F_{\text {of }}$$ to back propagate its net-work. Thereof, the RGB stream feature can simulate the features of optical flow stream to train the early part of student sub-network. We then used a loss func-tion of cross-entropy between the predictive class de-noted as [TeX:] $$P_{\text {rgb }}$$ and the true class T with MSE to train the student sub-network entirely as shown in Equation (7).

where [TeX:] $$\lambda$$ is the scalar weight as the influence of motive feature.

4. Experiment

We focused on the popular dataset for action re-cognition: HMDB51 [11]. It consists of 51 action classes with more than 6800 videos and 3 splits for training and test. There are 3570 clips in training set and 1530 clips in test set. In our experiment, we extracted 25 RGB frames from each video clip randomly as an in-put stream and a sample of 224 [TeX:] $$\times$$ 224 cropped image.

In the action recognition modules, we used the SGD optimization method with a weight decay of 0.0005, momentum of 0.9, and an initial learning rate of 0.1. The accuracy of our experiment on HMDB51 was 54.5%. Table 1 includes R3D-18 with single RGB, [TeX:] $$\text { TV-L }{ }^{1}$$ and two-stream network modules and its accuracy.

The optical flow frames are shown in Fig. 5. The three columns on the left are RGB frames extracted from video clips, and the two images on the right are two optical flow frames [TeX:] $$u$$ extracted from the adja-cent two RGB frames.

In the first row, the action class is ‘pour’. There is a significant movement in frames with obvious change in optical flow trajectory in frame (d) and (e). In the second row, the action class is ‘cartwheel’, with the objects including a girl and a cartwheel. We can see the change of movement information from (d) and (e). In the third row, the class is ‘shake_hands’. The main motion is of two people shaking hand. There is no obvious movement change of object between frames (b) and (c) from the optical flow image (e). In the forth row, the main motion is hitting a golf ball by the man. We can see obvious movement changes from (d) and (e). In the fifth row, the class is ‘hand_ stand’. The movements and posture of the active man have changed significantly. In the sixth row, the class is ‘shoot_ball’. The action changes of the boy's shooting and the coach's posture can be seen from (d) and (e).

Fig. 5.

Examples of Optical Flow: For Every Row, The Image (d) is The Optical Flow Between Frame (a) and (b), While Image (e) is The Optical Flow Between (b) and (c).

5. Conclusion

We introduced an architecture for action recog-nition based on teacher-student neural network. The student model only took video clips as an input stream but was able to extract both the appearance and the motion information. The model worked by training a sub-network to minimize the loss between its features and the features of optical flow stream and combining cross entropy loss for action recogni-tion. Our architecture resulted in higher accuracy in HMDB51 benchmark compared to other popular met-hods. The proposed method is sensitive and affected by lighting changes and camera motion between frames. In future studies, we will improve these defects and expand to other video dataset such as UCF101 [12], and improve our network in architecture to get better performance.