Copyright © 2002 by Institute of Pharmacology Polish Journal of Pharmacology Polish Academy of Sciences Pol. J. Pharmacol., 2002, 54, 327341 ISSN 1230-6002 REVIEW IN THE SEARCH FOR SELECTIVE LIGANDS OF 5-HT5, 5-HT6 AND 5-HT7 SEROTONIN RECEPTORS Anna Weso³owska Department of New Drug Research, Institute of Pharmacology, Polish Academy of Sciences, Smêtna 12, PL 31-343 Kraków, Poland In the search for selective ligands of 5-HT#, 5-HT$ and 5-HT% serotonin receptors. A. WESO£OWSKA. Pol. J. Pharmacol., 2002, 54, 327–341. In recent years much attention has been focused on the functional impor- tance of 5-HT#, 5-HT$ and 5-HT% receptors in the pathogenesis of neuropsy- chiatric and other diseases. In this connection, intensive studies with ligands of these receptors are currently in progress. Recognition of the structural characteristics responsible for the binding of a ligand molecule to an appro- priate receptor, and development of an active complex have reached an ad- vanced stage in the search for selective compounds. This review was under- taken to summarize the results of structure-activity relationship studies with ligands of 5-HT#, 5-HT$ and 5-HT% receptors. Additionally, some data on lo- calization, pharmacological properties and the functional role of those recep- tors were reported. Key words: structure-activity relationship, 5-HT# ligands, 5-HT$ li- gands, 5-HT% ligands A. Weso³owska Receptors through which serotonin (5-HT) pro- to demonstrate effects on signal transduction sys- duces its physiological and pathological effects tems such as adenylate cyclase (AC) or phospholi- have been the subject of thorough investigation, pase C [38, 44, 63]. Although no second messenger initially using both in vitro and in vivo pharmaco- coupling could be detected for mouse 5-HT5A re- logical methods, and later on by means of radio- ceptor, Francken et al. [18] and Hurley et al. [29] ligand binding. However, a large body of new data reported functional coupling of the human 5-HT5A were accumulated in the past years, also thanks to receptor to G proteins and receptor-mediated inhi- molecular techniques including in situ hybridiza- bition of AC activity in HEK293 cells expressing tion, which makes modification of the above classi- human 5-HT5A receptors. fication justifiable. The Serotonin Club Receptor The nothern blot analysis, quantitative polyme- Nomenclature Committee has recently proposed rase chain reaction (PCR) and in situ hybridization new classification of 5-HT receptors, which requir- experiments showed the presence of 5-HT5A mes- es three fundamental properties of a receptor to be senger ribonucleic acid (mRNA) in human brain described to ensure a comprehensive classification: cortex, hippocampus (dentate gyrus, CA1,CA2 and its operational (drug-related), transductional (re- CA3), hypothalamic area, amygdala and cerebel- ceptor-coupling) and structural (primary amino lum. The 5-HT5A receptor mRNA is expressed in acid sequence) characteristics. When applied to the many regions of the rat brain, the highest levels currently recognized 5-HT receptors, the above cri- being found in the hippocampus and hypothalamus, teria indicate the existence of up to seven receptor with lower concentrations in the cortex, thalamus classes in the central nervous system (CNS) [4, 12]. and striatum. The presence of 5-HT5A mRNA in To date, the least time has been spent on 5-HT5, the mouse brain was shown in the cerebral cortex, 5-HT6 and 5-HT7 receptor studies, mostly due to hippocampus, cerebellum and olfactory bulb, but the lack of selective ligands of these receptors. This not in the kidney, liver, spleen, lung or heart [4, 41, review is focused on structure-activity relationship 42, 44, 45]. investigations with 5-HT5, 5-HT6 and 5-HT7 recep- The physiological function of 5-HT5A receptors tor ligands. Additionally, some data on localization, is still unclear. On the basis of their localization it is pharmacological properties and functional role of proposed that 5-HT5A receptors may be involved in 5-HT5, 5-HT6 and 5-HT7 receptors as well as their multiple functions of forebrain 5-HT, such as regu- potential contribution to neuropsychiatric and other lation of affective states, cognition, anxiety (and diseases have been presented. related behaviors), sensory perception and neuro- endocrine functions [41]. Limbic distribution of 5-HT receptors and their ligands 5-HT5A mRNA seems to suggest its role in learn- 5 ing, memory and emotional behavior [42, 44]. Ex- Two 5-HT5 receptor subtypes, i.e. 5-HT5A and pression of 5-HT5A immunoreactive cells in the 5-HT5B, have been recognized. Human and rodent substantia nigra points to involvement of this re- 5-HT5A complementary deoxyribonucleic acid ceptor in the regulation of nigrostriatal dopaminer- (cDNA) encodes a protein of 357 amino acid resi- gic transmission and sensorimotor integration. The dues with seven hydrophobic domains, two puta- above assumption is evidenced by an increased lo- tive N-linked glycosylation sites and several poten- comotor and exploratory behavior in 5-HT5A re- tial phosphorylation sites for protein kinase C and ceptor knockout mice [23]. Interestingly, 5-HT5A 3’,5’-cyclic adenosine monophosphate (cAMP)-de- immunoreactive cells are in abundance in the su- pendent protein kinase. Murine 5-HT5B receptor prachiasmatic nucleus, which suggests its potential cDNA codes for a protein of 370 amino acids with involvement in circadian rhythms [41]. The results one putative site for N-linked glycosylation and of genetic studies seem to testify to a dysfunction consensus sites for phosphorylation by protein ki- of the 5-HT system in complex psychiatric disor- nase C (PKC) and cAMP-dependent protein kinase. ders, since Birkett et al. [3] found that an allelic 5-HT5A and 5-HT5B receptors display a 68% amino variation in human 5-HT5A receptor gene may play acid identity [4, 38, 45]. a role in the development of schizophrenia and af- The expression of murine and human 5-HT5 re- fective disorder. ceptors has already been shown in various cell sys- 5-HT5B receptor mRNA was present exclusive- tems, however, most of the described studies failed ly in the CA1 field of the hippocampus, the habe- 328 Pol. J. Pharmacol., 2002, 54, 327–341 5-HT#, 5-HT$ AND 5-HT% LIGANDS nula and dorsal raphe in murine brain, but not in sumption is supported by the high affinity of ergo- peripheral organs such as, e.g. the kidney, heart, lines, such as LSD (Ki = 0.9 nM), which contain an lung or liver [4, 38]. The recent findings of Grailhe embedded conformationally-constrained trypta- et al. [22] showed that, in contrast to mouse 5-HT5B mine moiety. Ibogaine, which represents a different gene, human 5-HT5B gene does not encode a func- conformation of the tryptamine moiety, does not tional protein because its coding sequence is inter- bind at 5-HT5A receptors (Ki > 10000 nM); how- rupted by stop codons. Hence, the latter authors ever, its lack of affinity may be associated with suggested that the 5-HT5B receptor was lost during a bulk of the bridged ring (Fig. 1). evolution, after rodents and primates have di- Successive structure-5-HT5A receptor affinity verged. The 5-HT5B receptor is the first example of investigations consisted in modifications of a mole- a brain-specific protein not found in humans. cule of dimethyltryptamine (DMT) which showed COS-7 and NIH-3T3 cells expressing murine no significant affinity for 5-HT5A receptors (Ki = 5-HT5A or 5-HT5B receptors displayed high affinity 2815 nM) [54]. N1-methylation, transposition of binding of [125J]-2-iodo-lysergic acid diethylamide the nitrogen atom within the pyrrole moiety or its (LSD), that could be displaced by various non-se- removal, reduction of the pyrrole moiety as well as lective serotonergic substances including 2-bromo- homologation of a side chain of DMT did not yield -LSD, ergotamine, methysergide and methiothepin substances showing affinity for 5-HT5A receptors (Tab. 1) [4, 22, 44]. (Ki = 3000–25000 nM). Only introduction of a 5-me- thoxy group, but not a 4- or 6-methoxy one, into DMT enhanced about threefold the affinity (Ki = Table 1. The affinity of some compounds for 5-HT5A and 5-HT5B receptors [4] 850 nM for 5-methoxyDMT). Introduction of more bulky substituents to position 5 of DMT (e.g. for pKi 5-benzyloxy-DMT Ki = 2660 nM), as well as to the Compound amine group (Ki = 5075–10330 nM) yielded sub- 5-HT5A 5-HT5B stances with a reduced affinity for 5-HT5A recep- (mouse COS-7) (mouse COS-7) tors. In contrast, arylpiperazines showed a diverse 2-Bromo-LSD 8.7 8.5 affinity for 5-HT5A receptors (Fig. 2). In particular, Ergotamine 8.4 7.4 1-(1-naphthyl)piperazines represent high-affinity 5-Carboxyamidotryptamine 7.8 6.9 5-HT5A ligands (Ki = 40 and 3 nM for 1-(1-naph- thyl)piperazine (1-NP) and 7-hydroxy-1-NP, re- Methysergide 7.2 7.8 spectively). Although insufficient information is Methiothepin 7.0 6.6 available to permit a conclusion that the naphthyl 5-HT 6.6 6.4 ring of 1-NP binds to 5-HT5A receptors in the same RU24969 6.5 6.4 NH2 Up to the present, only Teitler et al. [54] have conducted preliminary structure-5-HT5A receptor HO affinity relationship studies. Beginning with the N structure of 5-HT, which binds only with modest H affinity to 5-HT receptors (K = 170 nM), minor 5A i 5-HT Ki = 170 nM [54] structural modifications were examined in a step- wise fashion. Methylation of 5-HT in position 2, O homologation of its side chain, dimethylation of its CH3 (C2H5)2N N N H amine group or removal of the pyrrole moiety H3CO yielded substances with a reduced affinity for N CH CH 5-HT5A receptors (Ki = 1290–10000 nM).