View metadata, citation and similar papers at core.ac.uk brought to you by CORE

provided by Directory of Open Access Journals

Page 1 of 7 Impulse: the Premier Journal for Undergraduate Publications in the Neurosciences 2006

Effects of a 5-HT3 and antagonist on inter-male aggression in Mus musculus

Matthew J. Crisp and Michael Kerchner Department of Psychology, Washington College, Chestertown, MD 21620

Research has revealed an inverse relationship between (5-HT) levels in the brain and aggressive behavior. However, effects on aggression at the level of the receptor have yet to be elucidated for many 5-HT receptor subtypes. This study examined the effects of the 5-HT3 receptor agonist m-chlorophenylbiguanide (mCPBG) and antagonist on inter-male aggression in mice. Using a resident-intruder paradigm designed to assess both offensive and defensive aggression, male C57BL/6J mice received 1 mg/kg i.p. injections of either mCPBG, ondansetron, or an inactive vehicle and were subsequently exposed to male AKR/J mice for a period of 10 minutes. Attack latency and the proportion of time engaged in a range of defensive behaviors were recorded. Subject C57BL/6J mice were then immediately run in an open field test for an additional 10 minutes to examine any anxiolytic or sedative effects of the drugs. Results show no significant differences between drug groups in either offensive or defensive behavior, nor in measures of open field activity. However, an overall significant difference was observed between subjects of the offensive and defensive conditions in each drug group with regard to the open field test. In conclusion, this study did not reveal any significant effects of the 5-HT3 agents on inter-male aggression, which may reflect a functional difference between the unique fast-acting 5-HT3 receptor-channel and the remaining G-protein coupled 5-HT receptors. However, this conclusion is limited by the large variance in behavior combined with small sample sizes, or the possibility of a drug dose insufficient for behavioral effects.

Keywords: mCPBG, ondansetron, resident-intruder paradigm, offensive, defensive, C57BL/6J, AKR/J

Introduction

Serotonin, or 5-hydroxytryptamine (5- in turn enabling investigators to specifically HT), is an important neurotransmitter in the target particular receptor subtypes for human nervous system that plays an integral part pharmacological action. This study examines in myriad neural functions, from nociception to the pharmacological and behavioral effects of mood and aggression. Serotonin’s multifaceted the specific serotonin subtype 3 (5-HT3) agonist roles in the nervous system are mediated by a m-chlorophenylbiguanide (mCPBG) and family of 7 different receptor subtypes, many of antagonist ondansetron on the complex series of which contain distinct subunits and differ in behaviors that constitute inter-male aggression. their mechanism of signal transduction. Such Previous research has revealed an receptor heterogeneity allows serotonin to exert inverse relationship between CNS levels of 5- a number of different effects in the nervous HT and aggressive behavior in both humans and system. Current research is focused on the rodents. Specifically, higher concentrations of molecular and behavioral characterization of 5-HT correspond to decreased aggression. A these receptor subtypes to understand their decrease in cortical 5-HT has been demonstrated individual roles in a variety of neurological, to occur during and following an aggressive cellular, biochemical, and behavioral processes, encounter in rats (van Erp & Miczek, 2000). In Page 2 of 7 Effects of a 5-HT3 agonist and antagonist on aggression in Mus musculus 2006 humans, levels of the 5-HT metabolite 5- interacting with either pre- or post-synaptic 5- hydroxyindoleacetic acid (5-HIAA) in the HT1A receptors, other 5-HT receptor subtypes, or cerebral spinal fluid of non-suicidal patients other neurotransmitter systems (Sanchez et al., with Axis I disorders – schizophrenia, 1993). depression, or bipolar disorder – display this Regarding the 5-HT1B receptor, inverse relationship with measures of aggressive knockout mice lacking this gene display behavior (Stanley et al., 2000). Suicide itself heightened aggression compared to wild-types, can be considered a form of self-directed implicating an important role for this receptor aggression. Increased expression of 5-HT2A subtype in isolation-induced inter-male receptors in the hippocampus and prefrontal aggression (Saudou et al., 1994). cortex has been documented in adolescent Pharmacological studies support this role, in that suicide victims, further implicating altered 5-HT1B served to reduce aggression in function in aggression (Pandey et vivo (Olivier et al., 1995). Studies examining al., 2002). Environmental factors, such as the pharmacology of the 5-HT2 receptor reveal smoking, have also been associated with lower that the selective 5-HT2A/ agonist 1-(2,5- serotonergic function and increased aggression dimethoxy-4-iodophenyl)-2-aminopropane in patients with Axis I disorders (Malone et al., decreases aggression at high doses, but also 2003). elicits a characteristic side effect of 5-HT Examination of the effects of 5-HT on stimulation termed “wet dog shaking.” (Sloviter aggression and other behaviors at the level of 5- et al., 1978) The relationship is not as clear with HT receptor subtypes using pharmacological and 5-HT2 antagonists. While Sanchez et al. (1993) knockout approaches in rodents has produced a report no effects of 5-HT2 antagonists on plethora of data, much of which focuses on the aggression, other studies by White et al. (1991) 5-HT1 and 5-HT2 receptors and their respective and Olivier et al. (1995) provide evidence that subtypes. Employing isolation-induced the specific 5-HT2 antagonists DOI and aggression and resident-intruder paradigms to reduce aggression. examine inter-male aggression, these studies In addition to examining agonist and have reliably demonstrated that exposure of antagonist effects on the 5-HT1 and 5-HT2 specific agonists for these receptor subtypes receptors, the studies by White et al. (1991) and generally leads to a reduction in inter-male Sanchez et al. (1993) also examined the 5-HT3 aggression. Conversely, inhibition of receptor antagonists ondansetron and , but function using specific antagonists generally concluded that these compounds have no effect promotes such aggression, although this on mediating rodent inter-male aggression. relationship is not as sound. For example, the 5- However, the anxiolytic, or anxiety-reducing, HT1A agonists , 8-OH-DPAT (8- and anti-emetic properties of these compounds hydroxy-2-(di-n-propyl-amino) tetralin), 5-Me- have been well documented (Gandara et al., ODMT (5-methoxy-N,N-dimethyltryptamine), 1998). Effects of 5-HT3 agonists on inter-male , and significantly reduce aggression have yet to be effectively studied. aggressive behaviors when administered prior to In situ hybridization has shown that 5- a hostile confrontation (Olivier et al., 1989; Bell HT3 receptors exist in the peripheral and central & Hobson, 1994). Expansion of this nervous system (CNS). Within the CNS, 5-HT3 relationship through additional studies between mRNA is concentrated in the hippocampus, selective agents for 5-HT1A receptors and cerebral cortex, amygdaloid complex, aggression revealed that many of these agonists hypothalamus, and cranial nerve nuclei in the reliably reduce aggression without affecting brainstem (Tecott et al, 1993). Localization of motor performance (White et al., 1991; Sanchez 5-HT3 receptors in these neural structures et al., 1993; Mendoza et al., 1999). However, suggests that this fast-excitatory receptor may these and additional studies found that a number play important roles in emotional and behavioral of 5-HT1A antagonists also lead to a decrease in responses, as well as other, less defined aggression (Mendoza et al., 1999). Such a functions such as sensory perception, memory, discrepancy may be the result of these agents and, possibly, motor control. The presence of 5- Page 3 of 7 Impulse: the Premier Journal for Undergraduate Publications in the Neurosciences 2006

HT3 receptors in the amygdaloid complex and C57BL/6J male was exposed to a conspecific cerebral cortex suggests that these receptors AKR/J male; however, the role of the C57BL/6J have the potential to mediate aggression. males in the aggressive encounter, whether the Paralleling research on 5-HT1A, 5-HT1B, resident or the intruder, was dependent on the and 5-HT2 receptors, as well as investigations on condition and the housing of the subject. All the anxiolytic effects of 5-HT3 antagonists, the encounters were recorded on video. For the results of this study were predicted to reflect an offensive resident-intruder condition, isolated inverse relationship between 5-HT3 activation C57BL/6J mice served as the resident males and and aggressive behavior, both offensive and were exposed to socially housed AKR/J males. defensive. Specifically, the 5-HT3 agonist Two weeks after arrival, a 5-minute training mCPBG was expected to attenuate inter-male session was conducted to expose the resident aggression in both offensive and defensive C57BL/6J males to the AKR/J intruders. conditions. Conversely, the 5-HT3 antagonist Baseline behavioral measurements (described ondansetron was hypothesized to enhance below) were recorded one week later over a aggression. However, the anxiolytic effects of three day period during 10-minute encounters this drug could outweigh any positive effects on between the resident C57BL/6J and an intruder aggression. AKR/J male. Immediately after the 10-minute encounter, the C57BL/6J mice were run in an open field test (PASF Software, San Diego Materials and Methods Instruments) for 10 minutes to assess general activity and the subject’s state of anxiety. Subjects Experimental measures were recorded one week Seven week old C57BL/6J (30) male after baseline. Isolated C57BL/6J males mice and AKR/J (6) male mice were obtained received one dose of either the inactive vehicle from The Jackson Laboratory (Bar Harbor, ME). (10 mL/kg distilled water), the 5-HT3 agonist m- The strains were chosen for their moderate chlorophenylbiguanide HCl (1 mg/kg), or the 5- aggression and ease of visual discrimination HT3 antagonist ondansetron HCl (1 mg/kg). during antagonistic encounters. Immediately Following a 15 minute acclimation period, an upon arrival, 15 C57BL/6J mice were housed in intruder AKR/J mouse was placed in the cage of isolation, while the remaining 15 were socially the drugged resident and recorded for 10 housed in cohorts of three. The difference in minutes. Open field measurements were taken housing defined the aggressive condition tested for the experimental resident immediately (described below). The C57BL/6J mice were following the antagonistic encounter. then randomly and blindly assigned to one of For the defensive resident-intruder three treatment groups. AKR/J mice were either condition, isolated AKR/J male mice served as socially housed in a group of three conspecifics the residents and were exposed to 15 socially or in isolation. Mice were housed in clear housed C57BL/6J males. Two weeks after Plexiglas cages in a temperature controlled room arrival, a 5-minute training session was with a 12-hr light/dark cycle. Food (Lab Diet conducted to ensure that the AKR/J males could 5015 Mouse Diet) and tap water were provided defeat each C57BL/6J intruder. Defeat of the ad libitum and cages were cleaned once per intruder was essential for the observation of a week. Animal handling protocols were carefully full repertoire of defensive behaviors. Both reviewed and accepted by the Institutional baseline and experimental measurements were Animal Care and Use Committee (IACUC) of taken following the same procedure and timeline Washington College. as the offensive resident-intruder condition, with the exception that the socially housed Behavioral Testing experimental C57BL/6J males served as the intruders. The open field test was conducted A resident-intruder paradigm was immediately following an encounter. employed to assess both offensive and defensive Both resident-intruder conditions aggression. In both conditions, an experimental allowed the assessment of a range of behaviors. Page 4 of 7 Effects of a 5-HT3 agonist and antagonist on aggression in Mus musculus 2006 Behaviors were analyzed directly from video statistical software. Attack latencies in the footage and classified according to de Boer et al. offensive resident-intruder condition between (1999). For the offensive resident-intruder baseline and experimental groups were analyzed condition, aggressive behavior was assessed by with respect to drug group using a one-way measuring the latency to initial attack by the ANCOVA with baseline attack latencies as a C57BL/6J resident. Assessment of defensive covariate. Activity in the open field was divided behavior was qualified by observing the into central, peripheral, and total activity. proportion of time spent in submissive postures, Activity among drug groups and resident- freezing, flight, and retaliation. The open field intruder conditions were compared using a two- test, a measure of general activity and animal way ANCOVA for each activity measure, with anxiety, was scored by how often the animal respective baseline activity measures used as tripped a grid of beams during the 10-minute covariates. Due to a significant difference trial. between the offensive and defensive conditions, one-way ANCOVAs with baseline activities as Drugs covariates were employed on each activity Ondansetron hydrochloride (1,2,3,9- measure in either aggressive condition to tetrahydro-9-methyl-3-[(2-methyl-1H-imidazol- determine any significant effects of drug groups. 1-yl)methyl]-4H-carbazol-4-one hydrochloride; Significance was judged at the α = .05 level. cat. no. O3639; lot no. 084K4714; molecular weight 293.37) was obtained from Sigma- Aldrich. M-chlorophenylbiguanide (cat. no. Results 0440; batch no. 4A/57696; molecular weight 248.11) was obtained from Tocris Bioscience. Offensive and defensive resident-intruder Both drugs were dissolved in distilled water (the conditions inactive vehicle) to a concentration of 0.100 Mean ± S.E. values for both offensive mg/mL and stored at 4°C. Drugs were used and defensive conditions with respect to drug within 10 days. Injections were administered group are graphically depicted in Figure 1. In i.p. at a volume of 10 mL/kg. the offensive condition, the mean (± S.E.) attack latency for resident C57BL/6J males exposed to Data Analysis the inactive vehicle, mCPBG, and ondansetron Data were analyzed using SPSS 13.0 were 279.0 (±108.1), 315.0 (± 117.6), and 337.2

Mean (± S.E.) Attack Latency (sec) Mean % (±S.E.) Defensive Behavior Time

500 30.00 A B

25.00 400

20.00

300

15.00

200

10.00

100 5.00

0 0.00 vehicle mCPBG ondansetron vehicle mCPBG ondansetron Drug Group

Figure 1. In the offensive condition (A), significant differences were not observed between the vehicle (n = 5), mCPBG (n = 5), and ondansetron (n = 5) with respect to attack latency. Similarly, no significant differences were seen between the vehicle (n = 5), mCPBG (n = 5), and ondansetron (n = 4) regarding the proportion of time spent in defensive behavior in the defensive condition (B). Page 5 of 7 Impulse: the Premier Journal for Undergraduate Publications in the Neurosciences 2006 (± 109.1) seconds, respectively. A one-way the brain, or may have been ineffective at the ANCOVA, controlling for initial group tested concentration. However, the large within- differences, revealed no significant differences subject variance, combined with a small sample between mean attack latencies of experimental size (4-5 subjects per drug group and condition), animals with respect to drug group, F < 1. make discovering any relatively small effects of Intruder C57BL/6J males in the defensive these drugs on aggression difficult. condition exposed to the inactive vehicle, mCPBG, and ondansetron spent an average of 1000 Condition Offensive

18.97%, 16.27%, and 13.06% in defensive 800 Central Defensive behaviors. No significant differences were 600 observed between drug groups using a one-way 400 * * *

ANCOVA, with differences in resident AKR/J 200 Mean (± S.E.) Activity aggression as covariates, F < 1. 0 Open Field Measure 1000 Peripheral Open field activity 800 Activity in the open field was divided 600 into central, peripheral, and total activity. A 400 200 one-way ANCOVA with baseline activity Mean (± S.E.) Activity measures as a covariate revealed that the 0 1000 offensive condition possessed significantly 800 Total greater activity than the defensive condition for 600 all activity measures: central F(1,26) = 4.642, p 400

< .05; peripheral F(1, 26) = 9.144, p < .05; total 200 Mean (± S.E.) Activity

F(1, 26) = 10.533, p < .05. However, when one- 0 way ANCOVAs controlling for baseline activity vehicle mCPBG ondansetron were performed for each condition, no Drug Group significant differences between drug groups Figure 2. Analysis of open field activity revealed that were found (F < 1 for all drug groups in each the offensive group (n = 15) possessed significantly condition, except offensive central activity F[2, greater activity than the defensive group (n = 14) in 12] = 1.247, p > .05). Mean ± S.E. activity each activity measure. However, no significant values for each open field measure, clustered by differences were observed between drug groups in any resident-intruder condition, are shown in Figure activity measure. (*denotes significance at the .05 level) 2. The single significant difference within the open field activity of the subjects in both Discussion offensive and defensive conditions may be explained by differences in housing or by the This study failed to show that the 5-HT3 effects of the hostile encounter. Considering the agents mCPBG and ondansetron have a reliable former, housing mice in isolation could account effect on either offensive or defensive murine for effects in an open field test, inter-male aggression. Similarly, no significant which would explain the larger difference in differences between drug groups with respect to peripheral activity versus central activity. open field activity were present. These results However, the effects of the latter cannot be ruled may be due to a number of factors. First, the 5- out. Research has documented altered HT3 receptor may not play a significant role in serotonergic function both during and following mediating inter-male aggression, which may an antagonistic encounter in rats (van Erp & reflect its radical structural differences compared Miczek, 2000). Perhaps this effect is dependent to other serotonin receptors. Pharmacologically, on the role the rodent plays in an encounter, these drugs may act as poor 5-HT3 agents in whether establishing dominance or suffering vivo, failing to reach and bind to the receptors in defeat. Such an effect may partially explain the Page 6 of 7 Effects of a 5-HT3 agonist and antagonist on aggression in Mus musculus 2006 observed differences in open field activity comparative pharmacological study with 8- between aggressive conditions. Future trials hydroxy-2-dipropylaminotetralin, comparing open field activity in isolated and ipsapirone, buspirone, eltoprazine, and socially housed mice to victorious or defeated WAY-100635. J Pharamacol Exp Ther mice could provide the data to further study this 288(3): 1125-1133. observed effect. Gandara, D.R., Roila, F., Warr, D., Edelman, Suggestions for future research include M.J., Perez, E.A., and Gralla, R.J. (1998). increasing the sample size for each drug group, Consensus proposal for 5HT3 antagonists in implementing video analysis software, and the prevention of acute emesis related to testing a range of doses for each 5-HT3 agent. highly emetogenic chemotherapy. Dose, Expanding the sample size for each drug group schedule, and route of administration. and resident-intruder condition would decrease Support Care Cancer 6(3): 237-243. the variance observed in behavior and activity. Malone, K.M., Waternaux, C., Haas, G.L., If the drugs did have a subtle effect on behavior, Cooper, T.B., Li, S., and Mann, J.J. (2003). this may only be observed with a large group. A Cigarette smoking, suicidal behavior, and larger subject pool would also enable a better serotonin function in major psychiatric selection of only moderately aggressive resident disorders. American Journal of Psychiatry mice of both strains, thus decreasing initial 160(4): 773-779. group differences between drugs and conditions. Olivier, B., Mos, J., van der Heyden, J., and Behavioral assessment using video analysis Hartog, J. (1989). Serotonergic modulation software, where behaviors are fed into a of social interactions in isolated male mice. computer at specified time intervals, would Psychopharmacology (Berl) 97(2): 154- serve to increase the accuracy and range of 156. observed behaviors. This would prove a Olivier, B., Mos, J., van Oorschot, R., and Hen, valuable tool since many behaviors occur R. (1995). Serotonin receptors and animal rapidly and intermittently and cannot be models of aggressive behavior. accurately measured by the human hand. Pharmacopsychiatry 28 Suppl 2: 80-90. Finally, developing a dose-response curve for Pandey, G.N., Dwivedi, Y., Rizavi, H.S., Ren, each drug by testing a wider dose range would X., et al. (2002). Higher expression of be ideal to determine the dose-dependent serotonin 5-HT2A receptors in the relationship between the tested 5-HT3 agents and postmortem brains of teenage suicide behavior. One of the reasons that this study did victims. American Journal of Psychiatry not support the original hypothesis may have 159(3): 419-429. been due to a drug dose too low for any Sánchez, C., Arnt, J., Hyttel, J., and Moltzen, behavioral effects to be noted. In that case, E.K. (1993). The role of serotonergic analyzing the effects of the drugs over a range of mechanisms in inhibition of isolation- doses may reveal subtle effects in behavior. induced aggression in male mice. Psychopharmacology (Berl) 110(1-2): 53- 59. References Saudou, F., Amara, D.A., Dierich, A., LeMeur, M., Ramboz, S., Segu, L., Buhot, M., and

Bell, R, and Hobson, H. (1994). 5-HT1A receptor Hen, R. (1994). Enhanced aggressive influences on rodent social and agonistic behavior in mice lacking 5-HT1B receptor. behavior: a review and empirical study. Science 265(5180): 1875-1878. Neuroscience and Biobehavioral Reviews Sloviter, R.S., Drust, E.G., and Conner, J.D. 18(3): 325-338. (1978). Specificity of a rat behavioral model de Boer, S.F., Lesourd, M., Mocaer, E., for serotonin receptor activation. J Koolhaas, J.M. (1999). Selective Pharamacol Exp Ther 206(2): 339-347. antiaggressive effects of in Stanley, B., Molcho, A., Stanley, M., Winchel, resident-intruder test are mediated via 5- R., Gameroff, M.J., Parsons, B., and Mann, hydroxytryptamine1A receptors: A J.J. (2000). Association of aggressive Page 7 of 7 Impulse: the Premier Journal for Undergraduate Publications in the Neurosciences 2006 behavior with altered serotonergic function in patients who are not suicidal. Am J Psychiatry 157(4): 609-614. Tecott, L.H., Maricq, A.V., and Julius, D. (1993). Nervous system distribution of the serotonin 5-HT3 receptor mRNA. Proc Natl Acad Sci USA 90(4): 1430-1434. van Erp, A.M. and Miczek, K.A. (2000). Aggressive behavior, increased accumbal , and decreased cortical serotonin in rats. J Neurosci 20(24): 9320-9325. White, S.M., Kucharik, R.F., and Moyer, J.A. (1991). Effects of serotonergic agents on isolation-induced aggression. Pharmacol Biochem Behav 39(3): 729-736.

Acknowledgements

Matthew would like to thank the Washington College psychology faculty for assistance with data analysis, Jaclyn Thornton for her help with data collection, and Dr. Jeffery Brown for his collaboration on the biochemistry behind 5-HT3 receptors.

Corresponding Author

Matthew J. Crisp Washington College Department of Psychology [email protected] 300 Washington Avenue Chestertown, MD 21620