Bioinformatics Biocomputing and Perl An Introduction to Bioinformatics Computing Skills and Practice Michael Moorhouse Post-Doctoral Worker from Erasmus MC, The Netherlands Paul Barry Department of Computing and Networking, Institute of Technology, Carlow, Ireland Copyright 2004 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777 Email (for orders and customer service enquiries): [email protected] Visit our Home Page on www.wileyeurope.com or www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0-470-85331-X Typeset in 9.5/12.5pt Lucida Bright by Laserwords Private Limited, Chennai, India Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production. For my parents, who taught me the value of knowledge – MJM For three great kids: Joseph, Aaron and Aideen – PJB Contents Preface xv 1 Setting the Biological Scene 1 1.1 Introducing Biological Sequence Analysis 1 1.2 Protein and Polypeptides 4 1.3 Generalised Models and their Use 5 1.4 The Central Dogma of Molecular Biology 6 1.4.1 Transcription 6 1.4.2 Translation 7 1.5 Genome Sequencing 10 1.5.1 Sequence assembly 11 1.6 The Example DNA-gene-protein system we will use 12 Where to from Here 13 2 Setting the Technological Scene 15 2.1 The Layers of Technology 15 2.1.1 From passive user to active developer 16 2.2 Finding perl 17 2.2.1 Checking for perl 17 Where to from Here 18 I Working with Perl 19 3TheBasics 21 3.1 Let’s Get Started! 21 3.1.1 Running Perl programs 22 3.1.2 Syntax and semantics 23 3.1.3 Program: run thyself! 25 3.2 Iteration 26 3.2.1 Using the Perl while construct 26 3.3 More Iterations 30 3.3.1 Introducing variable containers 31 3.3.2 Variable containers and loops 32 viii Contents 3.4 Selection 34 3.4.1 Using the Perl if construct 35 3.5 There Really is MTOWTDI 36 3.6 Processing Data Files 41 3.6.1 Asking getlines to do more 43 3.7 Introducing Patterns 44 Where to from Here 46 The Maxims Repeated 46 4 Places to Put Things 49 4.1 Beyond Scalars 49 4.2 Arrays: Associating Data with Numbers 49 4.2.1 Working with array elements 51 4.2.2 How big is the array? 51 4.2.3 Adding elements to an array 52 4.2.4 Removing elements from an array 54 4.2.5 Slicing arrays 54 4.2.6 Pushing, popping, shifting and unshifting 56 4.2.7 Processing every element in an array 57 4.2.8 Making lists easier to work with 59 4.3 Hashes: Associating Data with Words 60 4.3.1 Working with hash entries 61 4.3.2 How big is the hash? 61 4.3.3 Adding entries to a hash 62 4.3.4 Removing entries from a hash 62 4.3.5 Slicing hashes 63 4.3.6 Working with hash entries: a complete example 64 4.3.7 Processing every entry in a hash 66 Where to from Here 68 The Maxims Repeated 68 5 Getting Organised 71 5.1 Named Blocks 71 5.2 Introducing Subroutines 73 5.2.1 Calling subroutines 73 5.3 Creating Subroutines 74 5.3.1 Processing parameters 76 5.3.2 Better processing of parameters 78 5.3.3 Even better processing of parameters 80 5.3.4 A more flexible drawline subroutine 83 5.3.5 Returning results 84 5.4 Visibility and Scope 85 5.4.1 Using private variables 86 5.4.2 Using global variables properly 88 5.4.3 The final version of drawline 89 5.5 In-built Subroutines 90 5.6 Grouping and Reusing Subroutines 92 5.6.1 Modules 93 5.7 The Standard Modules 96 5.8 CPAN: The Module Repository 96 5.8.1 Searching CPAN 97 5.8.2 Installing a CPAN module manually 98 Contents ix 5.8.3 Installing a CPAN module automatically 99 5.8.4 A final word on CPAN modules 99 Where to from Here 100 The Maxims Repeated 100 6 About Files 103 6.1 I/O: Input and Output 103 6.1.1 The standard streams: STDIN, STDOUT and STDERR 103 6.2 Reading Files 105 6.2.1 Determining the disk-file names 106 6.2.2 Opening the named disk-files 108 6.2.3 Reading a line from each of the disk-files 110 6.2.4 Putting it all together 110 6.2.5 Slurping 114 6.3 Writing Files 116 6.3.1 Redirecting output 117 6.3.2 Variable interpolation 117 6.4 Chopping and Chomping 118 Where to from Here 119 The Maxims Repeated 119 7 Patterns, Patterns and More Patterns 121 7.1 Pattern Basics 121 7.1.1 What is a regular expression? 122 7.1.2 What makes regular expressions so special? 122 7.2 Introducing the Pattern Metacharacters 124 7.2.1 The + repetition metacharacter 124 7.2.2 The | alternation metacharacter 126 7.2.3 Metacharacter shorthand and character classes 127 7.2.4 More metacharacter shorthand 128 7.2.5 More repetition 130 7.2.6 The ? and * optional metacharacters 130 7.2.7 The any character metacharacter 131 7.3 Anchors 132 7.3.1 The \b word boundary metacharacter 132 7.3.2 The ^ start-of-line metacharacter 133 7.3.3 The $ end-of-line metacharacter 133 7.4 The Binding Operators 134 7.5 Remembering What Was Matched 135 7.6 Greedy by Default 137 7.7 Alternative Pattern Delimiters 138 7.8 Another Useful Utility 139 7.9 Substitutions: Search and Replace 140 7.9.1 Substituting for whitespace 141 7.10 Finding a Sequence 142 Where to from Here 146 The Maxims Repeated 146 8 Perl Grabbag 147 8.1 Introduction 147 8.2 Strictness 147 x Contents 8.3 Perl One-liners 149 8.4 Running Other Programs from perl 152 8.5 Recovering from Errors 153 8.6 Sorting 155 8.7 HERE Documents 159 Where to from Here 160 The Maxims Repeated 161 II Working with Data 163 9 Downloading Datasets 165 9.1 Let’s Get Data 165 9.2 Downloading from the Web 165 9.2.1 Using wget to download PDB data-files 167 9.2.2 Mirroring a dataset 168 9.2.3 Smarter mirroring 168 9.2.4 Downloading a subset of a dataset 169 Where to from Here 171 The Maxims Repeated 171 10 The Protein Databank 173 10.1 Introduction 173 10.2 Determining Biomolecule Structures 174 10.2.1 X-Ray Crystallography 174 10.2.2 Nuclear magnetic resonance 176 10.2.3 Summary of protein structure methods 177 10.3 The Protein Databank 177 10.4 The PDB Data-file Formats 179 10.4.1 Example structures 180 10.4.2 Downloading PDB data-files 181 10.5 Accessing Data in PDB Entries 182 10.6 Accessing PDB Annotation Data 183 10.6.1 Free R and resolution 184 10.6.2 Database cross references 186 10.6.3 Coordinates section 188 10.6.4 Extracting 3D coordinate data 191 10.7 Contact Maps 192 10.8 STRIDE: Secondary Structure Assignment 196 10.8.1 Installation of STRIDE 197 10.9 Assigning Secondary Structures 197 10.9.1 Using STRIDE and parsing the output 200 10.9.2 Extracting amino acid sequences using STRIDE 204 10.10 Introducing the mmCIF Protein Format 205 10.10.1 Converting mmCIF to PDB 206 10.10.2 Converting mmCIFs to PDB with CIFTr 206 10.10.3 Problems with the CIFTr conversion 208 10.10.4 Some advice on using mmCIF 208 10.10.5 Automated conversion of mmCIF to PDB 208 Where to from Here 210 The Maxims Repeated 210 Contents xi 11 Non-redundant Datasets 211 11.1 Introducing Non-redundant Datasets 211 11.1.1 Reasons for redundancy 211 11.1.2 Reduction of redundancy 212 11.1.3 Non-redundancy and non-representative 212 11.2 Non-redundant Protein Structures 213 Where to from Here 217 The Maxims Repeated 217 12 Databases 219 12.1 Introducing Databases 219 12.1.1 Relating tables 220 12.1.2 The problem with single-table databases 222 12.1.3 Solving the one-table problem 222 12.1.4 Database system: a definition 224 12.2 Available Database Systems 224 12.2.1 Personal database systems 225 12.2.2 Enterprise database systems 225 12.2.3 Open source database systems 225 12.3 SQL: the Language of Databases 226 12.3.1 Defining data with SQL 226 12.3.2 Manipulating data with SQL 227 12.4 A Database Case Study: MER 227 12.4.1 The requirement for the MER database 231 12.4.2 Installing a database system 232 12.4.3 Creating the MER database 233 12.4.4 Adding tables to the MER database 235 12.4.5 Preparing SWISS-PROT data for importation