Stacked collaborative transformer network with contrastive learning for video moment localization

Article type: Research Article

Authors: Megala, G. | Swarnalatha, P.^*

Affiliations: School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India

Correspondence: [*] Corresponding author: P. Swarnalatha, School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India. E-mail: [email protected].

Abstract: Video grounding intends to perform temporal localization in multimedia information retrieval. The temporal bounds of the target video span are determined for the given input query. A novel interactive multi-head self-attention (IMSA) transformer is proposed to localize an unseen moment in the untrimmed video for the given image. A new semantic-trained self-supervised approach is considered in this paper to perform cross-domain learning to match the image query – video segment. It normalizes the convolution function enabling efficient correlation and collecting of semantically related video segments across time based on the image query. A double hostile Contrastive learning with Gaussian distribution parameters method is advanced to learn the representations of video. The proposed approach performs dynamically on various video components to achieve exact semantic synchronization and localization among queries and video. In the proposed approach, the IMSA model localizes frames greatly compared to other approaches. Experiments on benchmark datasets show that the proposed model can significantly increase temporal grounding accuracy. The moment occurrence is identified in the video with a start and end boundary ascertains an average recall of 86.45% and a mAP of 59.3%.

Keywords: Contrastive learning, gaussian parameter, self-attention transformer, temporal localization, video grounding

DOI: 10.3233/IDA-240138

Journal: Intelligent Data Analysis, vol. Pre-press, no. Pre-press, pp. 1-18, 2024