Fish and Fisheries 14 (2013), 186-204. DOI: 10.1111/j.1467-2979.2012.00462.x Video multitracking of fi sh behaviour: a synthesis and future perspectives Johann Delcourt, Mathieu Denoël, Marc Ylieff & Pascal Poncin Laboratory of Fish and Amphibian Ethology, Behavioural Biology Unit, University of Liège, Liège, Belgium Abstract With the development of digital imaging techniques over the last decade, there are now new opportunities to study complex behavioural patterns in fish (e.g. schooling behaviour) and to track a very large number of individuals. These new technologies and methods provide valuable information to fundamental and applied science disciplines such as ethology, animal sociology, animal psychology, veterinary sciences, animal wel- fare sciences, statistical physics, pharmacology as well as neuro- and ecotoxicology. This paper presents a review of fish video multitracking techniques. It describes the possibilities of tracking individuals and groups at different scales, but also outlines the advantages and limitations of the detection methods. The problem of occlusions, during which errors of individual identifications are very frequent, is underlined. This paper summarizes different approaches to improving the quality of individual identification, notably by the deve- lopment of three-dimensional tracking, image analysis and probabilistic applications. Finally, implications for fish research and future directions are presented. Keywords 3D tracking; ethometry; fish tracking; occlusion; schools; video multitracking © 2012 Blackwell Publishing Ltd Background Aims and scope Basis of video-tracking techniques Experimental set-up and principles of the video tracking Detection thresholds Minimum threshold size Body size and arena size Activity threshold and frame rate The specificity of multitracking techniques: tracking groups General comments Detection Occlusions: difficulties and solutions Detection of occlusions Three-dimensional tracking Merge–split and straight-through approaches Identifying individuals during occlusions Probabilistic applications Application of multitracking Scale of analysis: from individual to group patterns Implications in fish research and future directions Acknowledgments References Correspondence: Johann Delcourt, Laboratory of Fish and Amphibian Ethology, Behavioural Biology Unit, University of Liège, 22 Quai van Beneden 4020 Liège, Belgium Tel.: +32 (0)4 366 5079 or 5080 Fax: +32 (0)4 366 5113 E-mail: [email protected] The present pdf is the author postprint (i.e., post-refereed version of the manuscript). The paginated published pdf is archived in an institutional repository (http://hdl.handle.net/2268/109041) and in the publisher website (http://dx.doi. org/10.1111/j.1467-2979.2012.00462.x) (Blackwell). Background structures of social systems (Camazine et al. 2001; Anderson 2002), multispecies interactions (Ward et Aristotle first reported schooling behaviour in fish al. 2002a; Mathis and Chivers 2003) , self-organi- over 2400 years ago (Masuda and Tsukamoto 1999). zation (Camazine et al. 2001; Anderson 2002; Par- However, the first real quantitative studies of their rish et al. 2002), social synchronization and social structures and dynamical properties were underta- amplification phenomena (Camazine et al. 2001; ken only <60 years ago (e.g. for the earlier paper : Anderson 2002; Canonge et al. 2009), and in the Breder 1954; Keenleyside 1955; Cullen et al. 1965; ontogeny of social behaviours (Masuda et al. 2003; Pitcher 1973; Pitcher and Partridge 1979; Partridge Fukuda et al. 2010). Specifically, in situations where et al. 1980). The initial measurements were manual a large number of individuals are involved, the use and laborious. Since the 1980s and 1990s (e.g. for of video-tracking data is essential, as manual ana- the earlier works: Aoki 1982, 1984; Reynolds 1987), lyses would be complicated, time-consuming and many attempts have been made to replicate the sometimes even impossible. Today, multitracking patterns of moving animal groups using computer allows us to observe directly the behaviours of simulations (e.g. Parrish et al. 2002; Czirók and groups and to determine the real interaction rules Vicsek 2006). Generally, this approach constituted by sampling data collected in nature or in the labo- an a posteriori study of these behaviours. It was ratory, without any a posteriori rules (e.g. Ballerini based on predetermined interaction rules between et al. 2008; Cavagna et al. 2010; Herbert-Read et al. individuals, which could be examined by compari- 2011; Katz et al. 2011). sons with natural behavioural patterns. However, Among living organisms, fish frequently form although a good fit was obtained between computed shoals, defined as a voluntary association of indi- and natural patterns, the interaction rules in natu- viduals, in both fresh and salt water environments ral fish groups remained less well explained. (Pitcher and Parrish 1993). It has been estimated With the recent developments in image analyses that more than 25% of the approximately 27 000 and computer sciences, there are now powerful tools species of teleosts adopt shoaling behaviours throu- that can substitute or complement traditional beha- ghout their life, and over 50% do this as juveniles vioural observation tools. Video tracking, by defi- (Shaw 1978). These behaviours are also reported in nition, is the tracking of moving objects (here fish many other animal taxa such as selachians (Klimley individuals) and the monitoring of their activities 1985), cephalopods (Boal and Gonzalez 1998), crus- by image sequences obtained from video cameras taceans (Evans et al. 2007) and amphibians (d’Heur- (Maggio and Cavallaro 2011). It is an automatized sel and Haddad 2002). In collective behaviours of procedure that determines animal position over fish, gradients from swarm (unpolarized shoals) time and gives the resulting tracks with a large array to school (polarized shoals) have been identified of data such as distance travelled, speed or space depending on the degree of polarization and the used (Noldus et al. 2001; Maggio and Cavallaro synchronization of speed (Pitcher 1983; Couzin et 2011). The quantitative approach of this methodo- al. 2002; Parrish et al. 2002). Small teleost fishes logy has made it possible to collect straightforward are used as laboratory models to study these collec- data in fields as varied as ecotoxicology (Kanen et tive behaviours (e.g. Wright and Krause 2006). For al. 2005; Jakka et al. 2007; Denoël et al. 2010), neu- instance, the zebrafish (Brachydanio rerio, Cypri- rotoxicology (Eddins et al. 2010), behavioural brain nidae), one of the most commonly used laboratory research (Mathur et al. 2010; Silverman et al. 2010), animals, which adopts shoaling and schooling beha- pharmacology (Pinhasov et al. 2005; Singh et al. viours, is the model organism par excellence for all 2009; Cachat et al. 2010), genetic and behavioural the above-mentioned topics (e.g. Gerlai 2003; Guo screening (Orger et al. 2004; Egan et al. 2009; Blaser 2004; Hill et al. 2005; Rubinstein 2006). et al. 2010), animal well-being studies (Navarro-Jo- As happened when single video techniques were ver et al. 2009; Winandy and Denoël 2011), beha- launched, video multitracking has been underused vioural ontogeny (Fontaine et al. 2008; Fukuda et even though it could be a pertinent tool for unders- al. 2010), social behaviour (Newlands and Porcelli tanding the mechanisms of animal interactions and 2008; Salierno et al. 2008), cognition (Bisazza et al. particularly of shoaling behaviours. Several initia- 2010) and ethology and behavioural ecology (Prit- tives of multitracking have been developed inde- chard et al. 2001; Speedie and Gerlai 2008). pendently, using a variety of technologies. With video multitracking, more than one indivi- dual can be tracked simultaneously. Sensu stricto, Aims and scope this refers to tracks of individuals within the same space, which is called an arena in video-tracking Our objective is not to create a comparative list of procedures. Video multitracking can tackle the new all available video multitracking systems, commer- challenge of integrating the interactive component cial or otherwise. In the present review, we provide in animal behaviour. This has important outcomes, a general outline of video multitracking methods because animals interact with others during their applied in fish studies. It is addressed to fish resear- lifetime, such as when defending territories, cour- chers, particularly those working on fish schools, to ting sexual partners and taking care of progeny. give them an overview of the available methodolo- There are also a large number of species living in gies but also to support them in their choice of the groups in which the individuals orient their beha- most appropriate methodologies. In the first sec- viour according to that displayed by other members tion, we define the different characteristics of video of the group. All these interactions between indi- systems in fish studies, their advantages and draw- viduals within groups are fundamental in the pro- backs. In the second section, we tackle the problems cesses of information transmission and social lear- of individual identifications of multitracking sys- ning (Brown and Laland 2003; Hoare and Krause tems but also present directions towards optimiza- 2003), collective decision-making (Conradt and tion of these procedures, particularly with regard to Roper 2003; Couzin et al. 2005), and in the func- occlusions