Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences

∗ Zhu Zhang1, Zhou Zhao1, Yang Zhao1, Qi Wang2, Huasheng Liu2, Lianli Gao3 1College of Computer Science, Zhejiang University 2Alibaba Group, 3University of Electronic Science and Technology of China {zhangzhu, zhaozhou, zhaoyang}@zju.edu.cn, {wq140362, fangkong.lhs}@alibaba-inc.com

Abstract

In this paper, we consider a novel task, Spatio-Temporal 25.12s 27.53s Video Grounding for Multi-Form Sentences (STVG). Given Declarative Sentence: A little boy with a Christmas hat is catching a yellow toy. an untrimmed video and a declarative/interrogative sen- Interrogative Sentence: What is caught by the squatting boy on the floor? tence depicting an object, STVG aims to localize the spatio- Figure 1. An example of STVG for multi-form sentences. temporal tube of the queried object. STVG has two chal- lenging settings: (1) We need to localize spatio-temporal object tubes from untrimmed videos, where the object may age, which has attracted much attention and made great only exist in a very small segment of the video; (2) We progress [14, 25, 6, 42, 39]. Recently, researchers begin deal with multi-form sentences, including the declarative to explore video grounding, including temporal grounding sentences with explicit objects and interrogative sentences and spatio-temporal grounding. Temporal sentence ground- with unknown objects. Existing methods cannot tackle the ing [8, 11, 48, 37] determines the temporal boundaries of STVG task due to the ineffective tube pre-generation and events corresponding to the given sentence, but does not the lack of object relationship modeling. Thus, we then localize the spatio-temporal tube (i.e., a sequence of bound- propose a novel Spatio-Temporal Graph Reasoning Net- ing boxes) of the described object. Further, spatio-temporal work (STGRN) for this task. First, we build a spatio- grounding is to retrieve the object tubes according to textual temporal region graph to capture the region relationships descriptions, but existing strategies [50, 1, 38, 4] can only with temporal object dynamics, which involves the implicit be applied to restricted scenarios, e.g. grounding in a frame and explicit spatial subgraphs in each frame and the tem- of the video [50, 1] or grounding in trimmed videos [38, 4]. poral dynamic subgraph across frames. We then incorpo- Moreover, due to the lack of bounding box annotations, re- rate textual clues into the graph and develop the multi-step searchers [50, 4] can only adopt a weakly-supervised set- cross-modal graph reasoning. Next, we introduce a spatio- ting, leading to suboptimal performance. temporal localizer with a dynamic selection method to di- To break through above restrictions, we propose a novel rectly retrieve the spatio-temporal tubes without tube pre- task, Spatio-Temporal Video Grounding for Multi-Form generation. Moreover, we contribute a large-scale video Sentences (STVG). Concretely, as illustrated in Figure 1, arXiv:2001.06891v3 [cs.CV] 24 Mar 2020 grounding dataset VidSTG based on video relation dataset given an untrimmed video and a declarative/interrogative VidOR. The extensive experiments demonstrate the effec- sentence depicting an object, STVG aims to localize the tiveness of our method. The VidSTG dataset is available spatio-temporal tube of the queried object. at https://github.com/Guaranteer/VidSTG-Dataset. Compared with previous video grounding [50, 1, 38, 4], STVG has two novel and challenging points. First, we lo- calize spatio-temporal object tubes from untrimmed videos. 1. Introduction The objects may exist in a very small segment of the video and be hard to distinguish. And the sentences may only Grounding natural language in visual contents is a fun- describe a short-term state of the queried object, e.g. the damental and vital task in the visual-language understand- action ”catching a toy” of the boy in Figure 1. So it is cru- ing field. Visual grounding aims to localize the object cial to determine the temporal boundaries of object tubes described by the given referring expression in an im- by sufficient cross-modal understanding. Secondly, STVG ∗Zhou Zhao is the corresponding author. deals with multi-form sentences, that it, not only grounds

1 the conventional declarative sentences with explicit objects, determine the temporal boundaries of the tube, and then ap- but also localizes the interrogative sentences with unknown ply a spatial localizer with a dynamic selection method to objects, for example, the sentence ”What is caught by the ground the object in each frame and generate a smooth tube. squatting boy on the floor?” in Figure 1. Due to the lack of To facilitate this STVG task, we contribute a large-scale the explicit characteristics of objects (e.g. the class ”toy” video grounding dataset VidSTG by adding the multi-form and visual appearance ”yellow”), grounding for interroga- sentence annotations into video relation dataset VidOR. tive sentences can only depend on relationships between the Our main contributions can be summarized as follows: unknown object and other objects (e.g. the action relation ”caught by the squatting boy” and spatial relation ”on the • We propose a novel task STVG to explore the spatio- floor”). Thus, sufficient relationship construction and cross- temporal video grounding for multi-form sentences. modal relation reasoning are crucial for the STVG task. • We develop a novel STGRN to tackle this STVG task, Existing video grounding methods [38, 4] often extract a which builds a spatio-temporal graph to capture the re- set of spatio-temporal tubes from trimmed videos and then gion relationships with temporal object dynamics, and identify the target tube that matches the sentence. However, employs a spatio-temporal localizer to directly retrieve this framework may be unsuitable for STVG. On the one the spatio-temporal tubes without tube pre-generation. hand, the performance of this framework is heavily depen- dent on the quality of tube candidates. But it is difficult to • We contribute a large-scale video grounding dataset pre-generate high-quality tubes without textual clues, since VidSTG as the benchmark of the STVG task. the sentences may describe a short-term state of objects in • The extensive experiments show the effectiveness of a very small segment, but the existing tube pre-generation our proposed STGRN method. framework [38, 4] can only produce the complete object tubes from trimmed videos. On the other hand, these meth- ods only consider single tube modeling and ignore the rela- 2. Related Work tionships between objects. However, object relations are vi- tal clues for the STVG task, especially for interrogative sen- 2.1. Temporal Localization via Natural Language tences that may only offer the interactions of the unknown Temporal natural language localization is to detect objects with other objects. Thus, these approaches cannot the video clip depicting the given sentence. Early ap- deal with the complicated grounding of STVG. proaches [11, 8, 12, 23, 24] are mainly based on the slid- To tackle above problems, we propose a novel Spatio- ing window framework, which first samples abundant can- Temporal Graph Reasoning Network (STGRN) to capture didate clips and then ranks them. Recently, people be- region relationships with temporal object dynamics and di- gin to develop holistic and cross-modal methods [46, 2, 3, rectly localize the spatio-temporal tubes without tube pre- 35, 48, 37] to solve this problem. Chen, Zhang and Xu generation. Specifically, we first parse the video into a et al. [2, 3, 48, 22, 37] build frame-by-word interactions spatio-temporal region graph. Existing visual graph mod- from visual and textual contents to aggregate the matching eling approaches [41, 19] often build the spatial graph in clues. Zhang et al. [46] deal with the structural and semantic an image, which cannot utilize the temporal dynamics in- misalignment challenges by an explicitly structured graph. formation in videos to distinguish the subtle differences of Wang et al. [35] propose a reinforcement learning frame- object actions, e.g. distinguish ”open the door” and ”close work to adaptively observe frame sequences and associate the door”. Different from them, our spatio-temporal region video contents with sentences. Further, Mithun and Lin et graph not only involves the implicit and explicit spatial sub- al. [26, 21] devise the weakly-supervised temporal local- graphs in each frame, but also includes a temporal dynamic ization methods requiring only coarse video-level annota- subgraph across frames. The spatial subgraphs can capture tions for training. And besides the natural language query, the region-level relationships by implicit or explicit seman- Zhang et al. [49] attempt to localize the unseen temporal tic interactions, and the temporal subgraph can incorporate clip according to an image query. Although these methods the dynamics and transformation of objects across frames to have achieved promising performance, they still remain in further improve the relation understanding. Next, we fuse the temporal grounding. We further explore spatio-temporal the textual clues into this spatio-temporal graph as the guid- grounding in this paper. ance, and develop the multi-step cross-modal graph reason- 2.2. Object Grounding in Images/Videos ing. The multi-step process can support the multi-order re- lation modeling like ”a man hug a baby wearing a red hat”. Visual grounding [14, 25, 36, 43, 27, 44, 6, 51, 13, After it, we introduce a spatio-temporal localizer to directly 42, 47, 39, 40] aims to localize the visual object de- retrieve the spatio-temporal object tubes from the region scribed by the given referring expression. Early meth- level. Concretely, we first employ a temporal localizer to ods [14, 25, 36, 43, 27, 44] often extract object features . , ] i t i (1) , h t i … , w , … t i ) i , y γ is the entity- t i ) i a x is the concate- γ s , i.e., the visual i exp( s = [ =1 frame N t th =1 exp( t i } - L i t b i+2 f P { , where = s i , d ] e γ frame R a words, we first input the word , , which represents the queried s th Temporal - - ) ; ∈ L i Spatial e i+1 Localizer Localizer Localizer =1 Graph Temporal s s L i for ”boy” in Figure 2. Note that s 2 } Spatio =1 i 3 L i s with = [ s } W is the query representation. i { ( are projection matrices, s q Dynamic s q · s are the normalized width and height. Be- s 2 s { frame , > ) i t i ) th - W i s are the normalized coordinates of the center e from i s , h Temporal ) e e γ s 1 t i t i s and and bounding box vector w L , y =1 ( W s 1 i t i X r d x ( W frame = = ( R th i - a 1 - s γ ∈ i Our STVG task requires to capture the object relation- For the sentence � t i × sides, we obtain the frame features nation of the forward andAs backward the hidden STVG states of groundsor step objects interrogative described sentences, we in need the toresentation declarative extract from the query language rep- context.tity First, feature we select the en- where aware feature and 3.2. Spatio-Temporal Graph Encoder ships and developcially the for cross-modal interrogative understanding, sentences that espe- may only offer the point and object, for example, where feature of the entire frame from Faster R-CNN. embeddings into a bi-directional GRU [5]semantic to features learn the word for interrogative sentences, the featureis of chosen ”who” as or the ”what” aggregates entity the textual feature. clues from Next, language an context by attention method r … Graph Encoder Graph N Extractor Language Convolution Convolution Graph Spatial Temporal depicting man Temporal on - S

… it s watch Graph Cross-Modal ∈ Spatio regions denoted boy

Fusion s watch K Graph chair Explicit kiss spatio-temporal convolution layers directly retrieves the tubes from the region level. T Boy girl Relation man Object: Spatial Graph BiGRU and a sentence boy Queried is associated with its visual feature Implicit V chair t i

Faster r the the R-CNN ∈ girl? girl v -th frame corresponds to kissing t blond is the . A region boy of Temporal - Graph =1 K i little Construction } A head t i Spatio r { Given a video We first apply a pre-trained Faster R-CNN [30] to extract Figure 2. The Overall Architecturelearn of the the query Spatio-Temporal representation. GraphAfter Reasoning Next, it, Network a we (STGRN). spatio-temporal apply We localizer first a with extract spatio-temporal the graph region encoder features to and develop the multi-step cross-modal graph reasoning. 3. The Proposed Method by CNN, modeland the learn the language object-languageproaches expression matching. [42, through 13] Some decompose RNN recentcomponents the and ap- expression calculate matching into scores multiple And of each Deng module. and Zhuang etmechanism to al. build cross-modal [6, interactions. 51] Further, applyet Yang the al. co-attention [40, 39]improve explore the the relationships accuracy. betweenworks objects [50, As to 1, for 38, 4,scenarios. video 38, Zhou 4] grounding, and can existing Balajee onlyural et be language al. applied [50, in to 1] restricted a onlytranscriptions frame ground of and nat- the temporal video. alignmentand With Shi video et a clips, al. sequence [15, Huang of frames 33] by ground nouns the or weakly-supervised pronouns MILand in methods. Yamaguchi specific et And al. Chen [38, 4]tubes localize spatio-temporal from object trimmed videos,raw video but streams. cannot In this directlytask paper, deal we to propose further with a explore novel the STVG for spatio-temporal multi-form video sentences. grounding a set of regions for each frame, where a video contains 3.1. Video and Text Encoder an object, STVG is to retrieveure its 2 spatio-temporal illustrates tube. the Fig- overall framework. frames and the by interaction information of the unknown object with other balanced scalar. Here we simultaneously consider the ap- objects. Thus, we build a spatio-temporal region encoder pearance similarity and spatial overlap ratio of two regions. with T spatio-temporal convolution layers to capture the re- And the temporal distance |k − t| of two frames is used to gion relationships with temporal object dynamics and sup- limit the IoU score, that is, for distant frames, the linking port the multi-step cross-modal graph reasoning. score is mainly determined by feature similarity rather than t the spatial overlap. Next, for the region ri , we select the k 3.2.1 Graph Construction region rj with the maximal linking score from frame k to build an edge, and get 2M + 1 edges for each region (in- We first parse the video into a spatio-temporal region cluding the self-loop). The unlabeled temporal edges Etem graph, which involves the implicit spatial subgraph Gimp = have 3 directions: forward, backward and self-loop. (V, Eimp) , explicit spatial subgraph Gexp = (V, Eexp) in each frame and temporal dynamic subgraph Gtem = 3.2.2 Multi-Step Cross-Modal Graph Reasoning (V, Etem) across frames. Three subgraphs all treat regions as their vertexes V but have different edges. Note that we After graph construction, we incorporate the textual clues add the self-loop of each vertex in each subgraph. into this graph and develop the multi-step cross-modal Implicit Spatial Graph. We regard the fully-connected graph reasoning by T spatio-temporal convolution layers. region graph in each frame as the implicit spatial graph Cross-Modal Fusion. To capture the relationships asso- Gimp, where Eimp contains K × K undirected and unla- ciated with the sentence, we first use a cross-modal fusion beled edges in each frame (including self-loops). that dynamically injects textual evidences into the spatio- Explicit Spatial Graph. We extract the region triplet temporal graph. Concretely, we first utilize an attention t t t hri , pij, rji to construct the explicit spatial graph, where mechanism to aggregate the words features for each region. t t t t ri and rj are the i-th and j-th regions in frame t, and pij For a region ri , we calculate the attention weights over word L is the relation predicate between them. Each triplet can features {si}i=1, denoted by be regarded as an edge from i to j. Thus, explicit graph αt = w> tanh(Wmrt + Wms + bm), construction can be formulated as a relation classification ij m 1 i 2 j t t t L task [41, 45]. Concretely, given the feature [r ; b ] of re- exp(αij) X (3) i i αt = , qt = αt s , t t eij L i eij j gion i, feature [rj; bj] of region j and the united feature P exp(αt ) t t j=1 ij j=1 [rij; bij] of the union bounding box of i and j (also ex- m m m tracted by Faster R-CNN), we first transform three features where W1 , W2 are projection matrices, b is the bias > t via different linear leyers and then concatenate them into a and wm is the row vector. And qi is the region-aware tex- classification layer to predict the relations. Similer to ex- tual feature for each region i in frame t. isting works [41, 19], we train such a classifier on the Vi- Next, we build the textual gate that takes language in- sual Genome dataset [18], where we select top-50 frequent formation as the guidance to weaken the text-irrelevant re- predicates in its training data and add an extra no relation gions, given by class for non-existent edges. We then predict the relation- gt = σ(Wgqt + bg), vt = [rt gt; qt], (4) t t i i i i i i ships between ri and rj. Eventually, the edges Eexp have 3 directions (including i-to-j, j-to-i and i-to-i of self-loops) where σ is the sigmoid function, is the element-wise t dr and 51 types of labels (top-50 classes plus the self-loop). multiplication, gi ∈ R means the textual gate for re- t Temporal Dynamic Graph. While the spatial graphs gion ri . And we then concatenate the filtered region feature model region-region interactions, our temporal graph is to and textual feature to obtain the cross-modal region features t K N capture the dynamics and transformation of objects across {{vi}i=1}t=1. Next, we develop T spatio-temporal convo- frames. So we expect to connect the regions containing the lution layers for the multi-step graph reasoning. same object in different frames and then learn more expres- Spatial Graph Convolution. In each layer, we first de- sive and discriminative object features. For the frame t, we velop the spatial graph convolution to capture visual rela- connect its regions with adjacent 2M frames (M for for- tionships among regions in each frame. Concretely, with t K N ward frames and M for backward). Too distant frames can- cross-modal region features {{vi}i=1}t=1, we first adopt not provide the real-time dynamics. Concretely, we first the implicit graph convolution on Gimp that is undirected t k t and unlabeled, given by define the linking score s(ri , rj ) between ri from frame t rk k t X imp imp t and j from frame by vi = αij · (W vj), j∈N (rt) t k t k  t k i i s(ri , rj ) = cos(ri, rj ) + |k−t| · IoU(ri , rj ), (2) exp((Uimp[vt; bt])> · (Uimp[vt ; bt ])) (5) αimp = i i j j , where cos(·) is the cosine similarity of two features, IoU(·) ij P imp t t > imp t t exp((U [vi; bi]) · (U [vj; bj])) is the intersection-over-union of two regions, and  is the t j∈Ni(ri ) t t where Ni(ri ) are the regions connected with ri in Gimp. 3.3. Spatio-Temporal Localizer The implicit graph convolution can be regarded as a variant In this section, we devise a spatio-temporal localizer of self-attention and we compute the coefficient αimp by ij to determine the temporal tube boundaries and spatio- combining the visual features and region locations. temporal tubes of objects from the region level. Simultaneously, we develop the explicit graph convolu- Temporal Localizer. We first introduce the temporal tion. Different from the original undirected GCN [16, 34], localizer, which estimates a set of candidate clips and ad- we consider the direction and label information of edges on just their boundaries to obtain the temporal grounding [48]. the directed and labeled G , given by exp Specifically, we first aggregate the relation-aware region t P exp exp t exp graph into the frame level by an attention mechanism. With ˆvi = αlab(i,j) · (Wdir(i,j)vj + blab(i,j)), (6) t q j∈Ne(ri ) the query representation s , the region features of each exp frame are attended by where W(·) are optional matrices by the direction dir(i, j) exp t > f t f f β = w tanh(W m + W sq + b ), of edge (i, j), b(·) are optional bias by the label of edge i f 1 i 2 (i, j) i j j i i t K . Here, the edge has three directions ( -to- , -to- , - exp(β ) X (10) i N (rt) βet = i , mt = βetmt, to- ) and 51 types. e i are the regions connected with i PK t i i t exp i=1 exp(βi ) i=1 ri . Moreover, the relation coefficient α(·) can also be cho- sen by the label of edge (i, j). Different sentences describe where mt represents the relation-aware feature of frame different relations and their grounding is heavily dependent t. We then concatenate these features with their corre- on the specific relation understanding. Thus, the coefficient t N sponding global frame features {f }t=1, and apply another of explicit edges can be decided by query presentation sq, t N BiGRU to learn final frame features {h }t=1. Next, we given by define multi-scale candidate clips at each time step t as Bt = {(st, et)}P , where (st, et) = (t − w /2, t + w /2) αexp = Softmax(Wrsq + br), (7) i i i=1 i i i i are the start and end boundaries of the i-th clips, wi is the where αexp ∈ R51 corresponds to the coefficients of 51 temporal length of i-th clip and P is the clip number. After types of relationships. it, we estimate all candidate clips by a linear layer with the Temporal Graph Convolution. We next develop the sigmoid nonlinearity and simultaneously produce the off- temporal graph convolution on the directed and unlabeled sets of their boundaries, given by graph G to capture the dynamics and transformation of tem Ct = σ(Wc[ht; sq] + bc), δt = Wo[ht; sq] + bo,(11) objects across frames. We consider the forward, backward t t P and self-loop edges for each region ri with the cross-modal where C ∈ R corresponds to confidence scores of P can- t t 2P feature vi, denoted by didates at step t and δ ∈ R are the offsets of P clips. The temporal localizer has two losses: the alignment loss t X tem tem vei = αij · Wdir(i,j)vj, for the clip selection and a regression loss for boundary ad- t j∈Nt(ri ) justments. Concretely, for alignment loss, we first compute > (8) ˆt exp((Utemvt) · (Vtem v )) the temporal IoU score Ci of each clip with the ground tem i dir(i,j) j αij = , truth. And the alignment loss is denoted by P tem t > tem j∈N (rt) exp((U vi) · (Vdir(i,j)vj)) t i N P 1 X X t t t t tem tem Lalign = − (1−Cˆ )·log(1−C )+Cˆ ·log(C ), where W(·) and V(·) are matrices and dir(i, j) indicates NP i i i i t=1 i=1 the direction of edge (i, j) to select the corresponding pro- (12) jection matrix, where the temporal edges have three direc- ˆt tem where we use the temporal IoU score Ci rather than 0/1 tions. And αij is the semantic coefficient for each neigh- score to further distinguish high-score clips. Next, we fine- borhood region. tune the boundaries of the best clip with highest Ct, which Next, we combine the outputs of spatial and temporal i t(1) has the boundaries (s, e) and offsets (δs, δe). We first com- graph convolutions and obtain the result vi of the first pute the offsets of this clip from ground truth boundaries ˆ ˆ spatio-temporal convolution layer by (ˆs, eˆ) by δs = s − sˆ and δe = e − eˆ and define the regres-

t(1) t t t t sion loss by vi = ReLU(vi + ˆvi + vei + vi). (9) ˆ ˆ Lreg = R(δs − δs) + R(δe − δe), (13) In order to support multi-order relation modeling, we per- form the multi-step encoding by the spatio-temporal graph where R represents the smooth L1 function. encoder with T spatio-temporal convolution layers and Spatial Localizer. With the temporal grounding, we t K N learn final relation-aware region features {{mi}i=1}t=1 . next localize the target regions in each frame. For the t-th t K Table 1. Dataset Statistics about the Number of Declarative and frame with region features {mi}i=1, we directly estimate the matching scores of each region by integrating the query Interrogative Sentences. #Declar. Sent. #Inter. Sent. All representation sq and final frame feature ht, denoted by Training 36,202 44,482 80,684 t c t q t c Validation 3,996 4,960 8,956 Si = σ(W [mi; s ; h ] + b ), (14) Testing 4,610 5,693 10,303 t where Si is the matching score of region i of frame t. Simi- All 44,808 55,135 99,943 lar to temporal alignment loss, the spatial loss first compute ˆt the spatial IoU score Si for each region with the ground truth region, where the frames outside the temporal ground 4.1. Dataset Annotation truth are omitted. And the spatial loss is denoted by VidOR [32] is the existing largest object relation K dataset, containing 10,000 videos and fine-grained anno- 1 X X ˆt t ˆt t Lexp = − (1−Si )·log(1−Si )+Si ·log(Si ), tations for objects and their relations. Specifically, Vi- K|St| t∈St i=1 dOR annotates 80 categories of objects with dense bound- (15) ing boxes and annotates 50 categories of relation predi- where St is the set of frames in the temporal ground truth. cates among objects (8 spatial relations and 42 action re- Eventually, we devise a multi-task loss to train our pro- lations). Specifically, VidOR denotes a relation as a triplet posed STGRN in an end-to-end manner, given by hsubject, predicate, objecti and each triplet is associated with the temporal boundaries and spatio-temporal tubes of LST GRN = λ1Lalign + λ2Lreg + λ3Lexp, (16) subject and object. Based on VidOR, we can select the suitable triplets, and describe the subject or object with where λ , λ and λ are the hyper-parameters to control the 1 2 3 multi-form sentences. Taking VidOR as the basic dataset balance of three losses. has many advantages. On the one hand, we can avoid labor- 3.4. Dynamic Selection Method intensive annotations for bounding boxes. On the other hand, the relationships in the triplets can be simply incor- During inference, we first retrieve the temporal bound- porated into the annotated sentences. aries (T , T ) of the tube from the temporal localizer and s e We first split and clean the VidOR data, and then anno- then determine the grounded region for each frame by the tate the rest video-triplet pairs with multi-form sentences. spatial localizer. A greedy method directly selects the re- The cleaning process is introduced in the supplementary gions with highest matching scores St. However, such i material. For each video-triplet pair, we choose the subject generated tubes may not be very smooth. The bounding or object as the queried object, and then describe its ap- boxes between adjacent frames may have too large displace- pearance, relationships with other objects and visual envi- ments. Thus, to make the trajectory smoother, we introduce ronments. For interrogative annotations, the appearance of a dynamic selection method. Concretely, we first define queried objects is ignored. We discard video-triplet pairs the linking score s(rt, rt+1) between regions of successive i j that are too hard to give a precise description. And a video- frames t and t + 1 by triplet pair may correspond to multiple sentences. s(rt, rt+1) = St + St+1 + θ · IoU(rt, rt+1), i j i i i j (17) 4.2. Dataset Statistics t t+1 t where Si and Si are matching scores of regions ri and After annotation, there are 99,943 sentence descriptions t+1 ri , and θ is the balanced scalar which is set to 0.2. Next, about 79 types of queried objects for 44,808 video-triplet we generate the final spatio-temporal tube Y with the max- pairs, shown in Table 4. The average duration of videos is imal energy E given by 28.01s and the average temporal length of object tubes is 9.68s. The average lengths of declarative and interrogative E(Y) = 1 PTe−1 s(rt, rt+1), (18) |Te−Ts| t=Ts i j sentences are 11.12 and 8.98, respectively. Further, we pro- vide the distribution of 79 types of queried objects and some where (Ts, Te) are the temporal boundaries and we solve annotation examples in the supplementary material. this optimization problem using a Vitervi algorithm [10]. 5. Experiments 4. Dataset 5.1. Experimental Settings As a novel task, STVG lacks a suitable dataset as the benchmark. Therefore, we contribute a large-scale spatio- Implementation Details. In STGRN, we first sample temporal video grounding dataset VidSTG by augmenting 5 frames per second and downsample the frame number of the sentence annotations on VidOR [32]. overlong videos to 200. We then pretrain the Faster R-CNN Table 2. Performance Evaluation Results on the VidSTG Dataset. Declarative Sentence Grounding Interrogative Sentence Grounding Method m tIoU m vIoU [email protected] [email protected] m tIoU m vIoU [email protected] [email protected] Random 5.18% 0.69% 0.04% 0.01% 5.35% 0.60% 0.02% 0.01% GroundeR + TALL 9.78% 11.04% 4.09% 9.32% 11.39% 3.24% STPR + TALL 34.63% 10.40% 12.38% 4.27% 33.73% 9.98% 11.74% 4.36% WSSTG + TALL 11.36% 14.63% 5.91% 10.65% 13.90% 5.32% GroundeR + L-Net 11.89% 15.32% 5.45% 11.05% 14.28% 5.11% STPR + L-Net 40.86% 12.93% 16.27% 5.68% 39.79% 11.94% 14.73% 5.27% WSSTG + L-Net 14.45% 18.00% 7.89% 13.36% 17.39% 7.06% STGRN (Greedy) 18.99% 23.63% 13.48% 17.46% 20.02% 11.92% 48.47% 46.98% STGRN 19.75% 25.77% 14.60% 18.32% 21.10% 12.83% GroundeR + Tem. Gt - 28.80% 43.20% 22.74% - 26.11% 38.37% 18.34% STPR + Tem. Gt - 29.72% 44.78% 23.83% - 26.97% 39.89% 20.07% WSSTG + Tem. Gt - 33.32% 50.01% 29.98% - 30.05% 44.54% 25.76% STGRN + Tem. Gt - 38.04% 54.47% 34.80% - 35.70% 47.79% 31.41% on MSCOCO [20] to extract 20 region proposals for each The GroundeR is a frame-level approach, which originally frame (i.e. K = 20). The region feature dimension dr is grounds natural language in a still image. We apply it for 1,024 and we map it 256 before graph modeling. For sen- each frame of the clip and generate a tube. The STPR and tences, we use a pretrained Glove word2vec [28] to extract WSSTG are both tube-level methods and adopt the tube pre- 300-d word embeddings. As for the hyper-parameters, we generation framework. Specifically, the original STPR [38] set M to 5,  to 0.8, θ to 0.2 and set λ1, λ2, λ3 to 1.0, only grounds persons from multiple videos, we extend it 0.001 and 1.0, respectively. The layer number T of the to multi-type object grounding in a single clip. The orig- spatio-temporal graph encoder is set to 2. For the tempo- inal WSSTG [4] employs a weakly-supervised setting, we ral localizer, we set P to 8 and define 8 window widths extend it by applying a supervised triplet loss [39] to se- [8, 16, 32, 64, 96, 128, 164, 196]. We set the dimension of lect candidate tubes. So we obtain 6 combined baselines almost parameter matrices and bias to 256, including the GroundeR+TALL, STPR+TALL and so on. We also pro- exp exp f f W(·) , b(·) in the explicit graph convolution, W and b vide the temporal ground truth to form 3 baselines. We in the temporal localizer and so on. And the BiGRU net- show more baseline details in the supplementary material. works have 128-d hidden states for each direction. During training, we apply an Adam optimizer [7] to minimize the 5.2. Experiment Results multi-task loss LST GRN , where the initial learning rate is Table 2 shows the overall experiment results of all meth- set to 0.001 and the batch size is 16. ods, where STGRN(Greedy) uses the greedy region selec- Evaluation Criteria. We employ the m tIoU, m vIoU tion for the tube generation rather than the dynamic method. and vIoU@R as evaluation criteria [9, 4]. The m tIoU is The Random selects the temporal clip and spatial regions the average temporal IoU between the selected clips and randomly. Tem. Gt means that the temporal ground truth is ground truth clips. And we define SU as the set of frames provided. We can find several interesting points: contained in the selected or ground truth clips, and SI as the set of frames in both selected and ground truth clips. We • The GroundeR+{·} methods independently ground calculate vIoU by vIoU = 1 P IoU(rt, rˆt), where sentences in every frame and achieve worse perfor- |SU | t∈SI rt and rˆt are selected and ground truth regions of frame t. mance than STPR+{·} and WSSTG+{·} methods, val- The m vIoU is the average vIoU of samples and vIoU@R idating the temporal object dynamics across frames are is the proportion of samples which vIoU > R. vital for spatio-temporal video grounding. Baseline. Since no existing strategy can be directly ap- • The model performance on interrogative sentences plies to STVG, we extend the existing visual grounding is obviously lower than declarative sentences, which method GroundeR [31] and video grounding approaches shows the interrogative sentences with unknown ob- STPR [38] and WSSTG [4] as the baselines. Consider- jects are more difficult to ground. ing these methods all lack temporal grounding, we first ap- ply the temporal sentence localization methods TALL [8] • For temporal grounding, our STGRN achieves a better and L-Net [3] to obtain a clip and then retrieve the tubes performance than the frame-level localization method from the trimmed clip by GroundeR, STPR and WSSTG. TALL and L-Net, demonstrating the spatio-temporal Table 3. Ablation Results on the VidSTG Dataset. 0.49 0.19 Gimp Gexp Gtem m tIoU m vIoU [email protected] 0.48 X 44.81% 17.13% 21.08% 0.18 X 45.56% 17.58% 21.49% 0.47 m_tIoU 45.12% 17.53% 21.91% m_vIoU 0.17 X 0.46 45.99% 17.72% 22.07% Declarative Declarative XX Interrogative Interrogative 0.45 0.16 XX 46.70% 18.07% 22.23% 1 2 3 4 5 1 2 3 4 5 XX 47.12% 18.28% 22.52% Graph Layer Graph Layer XXX 47.64% 18.96% 23.19% Figure 3. Effect of the Number of Spatio-Temporal Graph Convo- lution Layers.

region modeling is effective to determine the temporal A little child grabs the hands of a man and jumps off the blue slide. boundaries of object tubes.

• For spatio-temporal grounding, our STGRN outper- Ground Truth forms all baselines on both declarative and interrog- WSSTG + L-Net STGRN ative sentences with or without temporal ground truth, A little child grabs the hands of a man and jumps off the blue slide which suggests our cross-modal spatio-temporal graph reasoning can effectively capture the object relation- ships with temporal dynamics and our spatio-temporal localizer can retrieve the object tubes precisely. Figure 4. A Example of the Spatio-Temporal Grounding Result.

• Our STGRN with the dynamic selection method outperforms the STGRN(Greedy) with the greedy each region feature tends to be identical. The performance method, showing the dynamic smoothness is benefi- changes on different criteria and sentence types are basi- cial to generate high-quality tubes. cally consistent, demonstrating the stable influence of T . 5.4. Qualitative Analysis 5.3. Ablation Study We display a typical example in Figure 4. The sentence In this section, we conduct the ablation studies on the describes two parallel actions ”grab the hands” and ”jump spatio-temporal region graph that is the key component of off the slide” of the boy in a short-term segment, requir- our STGRN. Concretely, the spatio-temporal graph includes ing the accurate spatio-temporal grounding. By intuitive the implicit spatial subgraph Gimp , explicit spatial subgraph comparison, our STRGN gives a more precise temporal seg- Gexp and temporal dynamic subgraph Gtem. We selec- ment and generates a more reasonable spatio-temporal tube tively discard them to generate ablation models and report than the baseline WSSTG+L-net. Moreover, the attention all ablation results in Table 3, where we do not distinguish method in the cross-modal fusion module builds a bridge the declarative and interrogative sentences. From these re- between visual and textual contents, and we visualize the sults, we can find that the full model outperforms all abla- weights of several key regions over the sentences here. We tion models, validating each subgraph is helpful for spatio- can see that semantic related region-word pairs have a larger temporal video grounding. If only a subgraph is applied, weight, e.g., the region of the boy and the word ”child”. the model with Gexp achieves the best performance, demon- strating the explicit modeling is the most important to cap- 6. Conclusion ture object relationships. And if two subgraphs are used, the model with Gexp and Gtem outperforms other models, which In this paper, we propose a novel spatio-temporal video suggests the spatio-temporal modeling play a crucial role in grounding task STVG and contribute a large-scale dataset relation understanding and high-quality video grounding. VidSTG. We then design a STGRN to capture region rela- Moreover, the layer number T is the essential hyperpa- tionships with temporal object dynamics and directly local- rameter of the spatio-temporal graph. We investigate the ef- ize the spatio-temporal tubes from the region level. fect of T by varying it from 1 to 5. Figure 3 shows the exper- Acknowledgments This work was supported by Zhejiang imental results on the criteria m tIoU and m vIoU for both Natural Science Foundation LR19F020006 and the Na- declarative and interrogative sentences. From the results, tional Natural Science Foundation of China under Grant we can find our STGRN has the best performance when T No.61836002, No.U1611461 and No.61751209, Sponsored is set to 2. The one-layer graph cannot sufficiently capture by China Knowledge Centre for Engineering Sciences the object relationships and temporal dynamics. And too and Technology and Alibaba-Zhejiang University Joint Re- many layers may result in region over-smoothing, that is, search Institute of Frontier Technologies. References [18] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- [1] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Object referring in videos with language and human gaze. Connecting language and vision using crowdsourced dense In Proceedings of the IEEE Conference on Computer Vision image annotations. International Journal of Computer Vi- and Pattern Recognition, pages 4129–4138, 2018. sion, 123(1):32–73, 2017. [2] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat- [19] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation- Seng Chua. Temporally grounding natural sentence in video. aware graph attention network for visual question answering. In EMNLP, pages 162–171. ACL, 2018. In ICCV, 2019. [3] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Luo. Localizing natural language in videos. In AAAI, 2019. Pietro Perona, Deva Ramanan, Piotr Dollar,´ and C Lawrence [4] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee K Zitnick. Microsoft coco: Common objects in context. In Wong. Weakly-supervised spatio-temporally grounding nat- ECCV, pages 740–755. Springer, 2014. ural sentence in video. 2019. [21] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng [5] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Liu. Weakly-supervised video moment retrieval via semantic Yoshua Bengio. Empirical evaluation of gated recurrent neu- completion network. AAAI, 2020. ral networks on sequence modeling. In NIPS, 2014. [22] Zhijie Lin, Zhou Zhao, Zhu Zhang, Zijian Zhang, and Deng [6] Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, Cai. Moment retrieval via cross-modal interaction networks and Mingkui Tan. Visual grounding via accumulated atten- with query reconstruction. IEEE Transactions on Image Pro- tion. In CVPR, pages 7746–7755, 2018. cessing, 29:3750–3762, 2020. [7] John Duchi, Elad Hazan, and Yoram Singer. Adap- [23] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Bao- tive subgradient methods for online learning and stochas- quan Chen, and Tat-Seng Chua. Attentive moment retrieval tic optimization. Journal of Machine Learning Research, in videos. In SIGIR, pages 15–24. ACM, 2018. 12(Jul):2121–2159, 2011. [24] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan [8] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Chen, and Tat-Seng Chua. Cross-modal moment localiza- TALL: temporal activity localization via language query. In tion in videos. In MM, pages 843–851. ACM, 2018. ICCV, pages 5277–5285. IEEE, 2017. [25] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana [9] Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Camburu, Alan L Yuille, and Kevin Murphy. Generation Nevatia. Turn tap: Temporal unit regression network for tem- and comprehension of unambiguous object descriptions. In poral action proposals. 2017. CVPR, pages 11–20, 2016. [10] Georgia Gkioxari and Jitendra Malik. Finding action tubes. [26] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy- In CVPR, pages 759–768, 2015. Chowdhury. Weakly supervised video moment retrieval from [11] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef text queries. In CVPR, pages 11592–11601, 2019. Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- [27] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- ments in video with natural language. In ICCV, pages 5803– eling context between objects for referring expression under- 5812, 2017. standing. In ECCV, pages 792–807. Springer, 2016. [12] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef [28] Jeffrey Pennington, Richard Socher, and Christopher Man- Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ning. Glove: Global vectors for word representation. In ments in video with temporal language. In EMNLP, pages EMNLP, pages 1532–1543, 2014. 1380–1390. ACL, 2018. [29] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, [13] Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Stefan Thater, Bernt Schiele, and Manfred Pinkal. Ground- Darrell, and Kate Saenko. Modeling relationships in refer- ing action descriptions in videos. Transactions of the Asso- ential expressions with compositional modular networks. In ciation of Computational Linguistics, 1:25–36, 2013. CVPR, pages 1115–1124, 2017. [30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. [14] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Faster r-cnn: Towards real-time object detection with region Kate Saenko, and Trevor Darrell. Natural language object proposal networks. In NIPS, pages 91–99, 2015. retrieval. In CVPR, pages 4555–4564, 2016. [31] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor [15] De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Darrell, and Bernt Schiele. Grounding of textual phrases Li Fei-Fei, and Juan Carlos Niebles. Finding ”it”: Weakly- in images by reconstruction. In ECCV, pages 817–834. supervised reference-aware visual grounding in instructional Springer, 2016. videos. In CVPR, June 2018. [32] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, [16] Thomas N Kipf and Max Welling. Semi-supervised classifi- and Tat-Seng Chua. Annotating objects and relations in user- cation with graph convolutional networks. In ICLR, 2016. generated videos. In ICMR, pages 279–287. ACM, 2019. [17] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and [33] Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all Juan Carlos Niebles. Dense-captioning events in videos. In frames are equal: Weakly-supervised video grounding with ICCV, pages 706–715, 2017. contextual similarity and visual clustering losses. In CVPR. [34] Petar Velickoviˇ c,´ Guillem Cucurull, Arantxa Casanova, visual object discovery through dialogs and queries. In Pro- Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph at- ceedings of the IEEE Conference on Computer Vision and tention networks. In ICLR, 2017. Pattern Recognition, pages 4252–4261, 2018. [35] Weining Wang, Yan Huang, and Liang Wang. Language- driven temporal activity localization: A semantic matching reinforcement learning model. In CVPR, pages 334–343, 2019. [36] Fanyi Xiao, Leonid Sigal, and Yong Jae Lee. Weakly- supervised visual grounding of phrases with linguistic struc- tures. In CVPR, pages 5945–5954, 2017. [37] Huijuan Xu, Kun He, L Sigal, S Sclaroff, and K Saenko. Multilevel language and vision integration for text-to-clip re- trieval. In AAAI, volume 2, page 7, 2019. [38] Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Spatio-temporal person retrieval via natural language queries. In ICCV, pages 1453–1462, 2017. [39] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal re- lationship inference for grounding referring expressions. In CVPR, pages 4145–4154, 2019. [40] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. In ICCV, pages 4644–4653, 2019. [41] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, pages 684–699, 2018. [42] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular at- tention network for referring expression comprehension. In CVPR, pages 1307–1315, 2018. [43] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, pages 69–85. Springer, 2016. [44] Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speaker-listener-reinforcer model for referring expres- sions. In CVPR, pages 7282–7290, 2017. [45] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global con- text. In CVPR, pages 5831–5840, 2018. [46] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In CVPR, pages 1247–1257, 2019. [47] Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. In CVPR, pages 4158–4166, 2018. [48] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao. Cross-modal interaction networks for query-based moment retrieval in videos. In SIGIR, 2019. [49] Zhu Zhang, Zhou Zhao, Zhijie Lin, Jingkuan Song, and Deng Cai. Localizing unseen activities in video via image query. In IJCAI, 2019. [50] Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly- supervised video object grounding from text by loss weight- ing and object interaction. arXiv preprint arXiv:1805.02834, 2018. [51] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton van den Hengel. Parallel attention: A unified framework for Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences –Supplementary Material–

7. Dataset Details temporal person retrieval among trimmed videos and only contains one type of objects (i.e. people), which is too sim- VidOR contains 7,000, 835 and 2,165 videos for train- ple for the STVG task. And VID-sentence dataset [4] con- ing, validation and testing, respectively. Since box annota- tains 30 categories but also offer the annotations on trimmed tions of testing videos are unavailable yet, we omit testing videos. Different from them, our VidSTG simultaneously videos, split 10% training videos as our validation data and offers temporal clip and spatio-temporal tube annotations, regard original validation videos as the testing data. Con- contains more sentence descriptions, has a richer variety of sidering a video may contain multiple same triplets that objects, and further supports multi-form sentences. have different temporal and bounding box annotations, we cut these videos into several short videos, where each short video contains a triplet that covers a segment of the short 8. Baseline Details video. We then delete unsuitable video-triplet pairs based Since no existing strategy can be directly applies to on three rules: (1) the video length is less than 3 seconds; STVG, we combine the existing visual grounding method (2) the temporal duration of the triplet is less than 0.5 sec- GroundeR [31] and spatio-temporal video grounding ap- onds; (3) the triplet duration is less than 2% of the video. proaches STPR [38] and WSSTG [4] with the temporal Next, because too many triplets are related to spatial rela- sentence localization methods TALL [8] and L-Net [3] as tions like ”in front of” and ”next to”, we delete 90% spatial the baselines. The TALL and L-Net first provide the tem- triplets to keep the types of relations balanced. poral clip of the target tube and the extended GroundeR, For each video-triplet pair, we choose the subject or STPR and WSSTG then retrieve the spatio-temporal tubes object as the queried object, and then describe its appear- of objects. ance, relationships with other objects and visual environ- We first introduce the TALL and L-Net approaches. The ments. We discard video-triplet pairs that are too hard to TALL applies a sliding window framework that first sam- give a precise description. And a video-triplet pair may cor- ples abundant candidate clips and then ranks them by esti- respond to multiple sentences. After annotation, there are mating the clip-sentence scores. During estimating, TALL 6,924 videos (5,563, 618 and 743 for training, validation incorporates the context features for the current clip to fur- and testing sets) and 99,943 sentences for 44,808 video- ther improve the localization accuracy. And L-Net devel- triplet pairs. we show some typical samples in Figure 5 with ops the evolving frame-by-word interactions for video and declarative and interrogative sentences. We can find that the query contents, and dynamically aggregates the matching objects may exist in a very small segment of the video and evidence to localize the temporal boundaries of clips ac- the sentences may only describe a short-term state of the cording to the textual query. queried object. Next, we illustrate the extended grounding methods Next, we show the distribution of different types of GroundeR, STPR and WSSTG based on the retrieved clip. queried objects as Figure 6. The original VidOR contains The GroundeR is a frame-level approach, which originally 80 types of objects, including 3 types of persons, 28 types grounds natural language in a still image. We apply it of animals and 49 types of other objects. After data cleaning for each frame of the clip to obtain the object region and and annotating, sentences in VidSTG describes 79 types of generate a tube by directly connecting these regions. The objects by the declarative or interrogative ways, including 3 drawback of this method is the lack of temporal context types of persons, 27 types of animals and 49 types of other modeling of regions. Different from it, original STPR objects. A rare type (i.e., stingray) is not contained in Vid- and WSSTG are both tube-level methods and adopt the STG. From Figure 6, we can find the sentences of person tube pre-generation framework. This framework first ex- types take up the largest proportion and sentence numbers tracts a set of spatio-temporal tubes from trimmed clips of other categories are relatively uniform. and then identifies the target tube. The original STPR [38] Moreover, we compare VidSTG with existing video only grounds persons from multiple videos, we extend it grounding datasets in Table 4. Previous temporal sentence to multi-type object grounding in a single clip. Specifi- grounding datasets like DiDeMo [11], Charades-STA [8], cally, we use the pre-trained Faster R-CNN to detect multi- TACoS [29] and ActivityCation [17] only provide the tem- type object regions to generate the candidate tubes rather poral annotations for each sentence and lack the spatio- than only generate person candidate tubes. And during temporal bounding boxes. As for existing video grounding training, we retrieve the correct tube from a video rather datasets, Persen-sentence [38] is originally used for spatio- multiple videos, where we do not change the loss func- Figure 5. Annotation Samples with Declarative or Interrogative Sentence Descriptions.

Human Animal Other 









 YQO IGZ IGX ZU_ VOM HGZ LOYN I[V ROUT YOTQ HOXJ JUM YULG JOYN IXGH HKGX IGQK ZXGOT ZOMKX INORJ J[IQ INGOX U\KT HGH_ YZUUR ZGHRK LX[OZY GJ[RZ ZUORKZ Z[XZRK NUXYK VOGTU YTGQK HXKGJ IGSKR M[OZGX XGHHOZ HUZZRK XGIQKZ HKTIN LG[IKZ VGTJG RGVZUV LXOYHKK HOI_IRK GOXIXGLZ IGSKXG YIUUZKX YW[OXXKR INOIQKT RKUVGXJ Y[OZIGYK VKTM[OT KRKVNGTZ GTZKRUVK NGTJHGM IXUIUJORK HGIQVGIQ YZUVEYOMT QGTMGXUU H[YZX[IQ Y[XLHUGXJ IKRRVNUTK HGH_EYKGZ ]GZKXIXGLZ IGZZRKIU] SOIXU]G\K \KMKZGHRKY SUZUXI_IRK YQGZKHUGXJ YTU]HUGXJ ZXGLLOIEROMNZ KRKIZXOIELGT XKLXOMKXGZUX YNKKVMUGZ NGSYZKXXGZ HGH_E]GRQKX HGRRYVUXZYEHGRR YIXKKTSUTOZUX Figure 6. The Distribution of Different Types of Queried Objects in the Entire VidSTG Dataset. tion of STPR. The original WSSTG [4] employs a weakly- sider single tube modeling and ignore the rela1ionships supervised setting, we extend it to the fully-supervised between objects. Finally, we obtain 6 combined base- form. Concretely, we discard the original ranking and di- lines GroundeR+TALL, STPR+TALL, WSSTG+TALL, versity losses and employs a classics triplet loss [39] on the GroundeR+L-Net, STPR+L-Net and WSSTG+L-Net. matching scores of the candidate tubes and sentence. The We also provide the temporal ground truth clip to STPR and WSSTG both have the drawbacks of the tube form 3 baselines GroundeR+Tem.Gt, STPR+Tem.Gt and pre-generation framework: (1) they are hard to pre-generate WSSTG+Tem.Gt. high-quality tubes without textual clues; (2) they only con- During training, we first train the TALL and L-Net based Table 4. Dataset Comparison. Dataset #Video #Sentence #Type Domain Temporal Ann. Box Ann. Multi-Form Sent. TACoS 127 18,818 - Cooking X DiDeMo 10,464 40,543 - Open X Charades-STA 6,670 16,128 - Activity X ActivityCaption 20,000 37,421 - Open X Person-sentence 5,293 30,365 1 Person X VID-sentence 4,318 7,654 30 Open X VidSTG 6,924 99,943 79 Open XXX

Table 5. Ablation Results of Directed GCN. Method m tIoU m vIoU [email protected] [email protected] modeling in Table 6, where we also apply the GRU+Object w/o. Explicit Subgraph 46.70% 18.07% 22.23% 13.12% Rec.+Attention to WSSTG+L-Net. Concretely, the object- Undirected GCN 46.81% 18.13% 22.28% 13.06% aware vector sq replaces the original sentence vector in final Undirected GAT 47.08% 18.21% 22.35% 13.20% Directed GCN (our) 47.64% 18.96% 23.19% 13.62% localization for L-Net and is added into visually guided sen- tence features for WSSTG. Table 6. Ablation Results of Query Modeling. Method Query Modeling m tIoU m vIoU [email protected] GRU 40.27% 13.85% 17.66% WSSTG+L-Net GRU+Object Rec.+Attention 41.22% 14.32% 20.08% GRU 46.93% 18.42% 22.41% STGRN GRU+Object Rec.+Attention 47.64% 18.96% 23.19% on the sentence-clip matching data and train GroundeR, STPR and WSSTG within the ground truth clip. But while inference, we first use TALL and L-Net to determine the clip boundaries and then employ GroundeR, STPR and WSSTG to localize the final tubes. To guarantee the fair comparison, GroundeR, STPR and WSSTG are built on the same TALL or L-Net models, and we apply the Adam opti- mizer to train all baselines.

9. More Ablation Study 9.1. Directed GCN To confirm the effect of the directed explicit GCN, we replace it with the original undirected GCN [16] and GAT [34]. In Table 5, our directed GCN has a better per- formance and the results of undirected GCN and GAT are close to the model without explicit subgraph modeling. The reason is that the undirected GCN and GAT have a similar ability with the implicit GCN and may lead to redundancy modeling. 9.2. Query Modeling Our input setting is consistent with previous grounding works but we adopt a different strategy for query mod- eling in STGRN. Previous works model the sentence by RNN as a whole query vector. Different from them, we use the NLTK library to recognize the first noun or interroga- tive word ”who/what” in the sentence, corresponding to the query object. We then select its feature se from RNN out- puts and adopt context attention to learn the object-aware query vector sq. We conduct an ablation study for query