Theoretical and experimental studies of drug–receptor interactions: Determinants for recognition of 5‐hydroxytryptamine analogs

Experimental studies on 5‐hydroxytryptamine (5‐HT) and its congeners have shown these compounds to interact with the same receptors in peripheral tissues and in brain. To evaluate the importance of the relative position of the 5‐HT‐like recognition elements at these receptors, we studied two compounds structurally related to 5‐HT in which the structural elements involved in receptor recognition are positioned differently from 5‐HT: 5‐hydroxyaminotetrahydrobenzindole (FHATHBIN) in which the position of the side chain is fixed with respect to the indole, and 4(β‐aminoethyl)‐5‐hydroxyindole (FAEFHI) in which the side chain is flexible, and connected to the indole at the C4 position (rather than at C3 as in 5‐HT). Ab initio molecular orbital calculations of the molecules and model fragments were performed with STO ‐3G and 3‐21G basis sets, using structural optimization procedures. The results show that both structures possess the reactivity elements required for the interaction with the 5‐HT receptor, but that FAEFHI cannot be recognized at the 5‐HT receptor because the side chain is held in the wrong conformation with respect to the indole portion by a strong hydrogen bond between the side chain amine group and the hydroxyl at C5. We report results from competition experiments for binding at high affinity 5‐HT binding sites in brain membranes which support this conclusion by showing that FAEFHI has low affinity for these sites.