Biopython Tutorial and Cookbook Jeff Chang, Brad Chapman, Iddo Friedberg Last Update{24 October 01 Contents 1 Introduction 3 1.1 What is Biopython? ......................................... 3 1.1.1 What can I find in the biopython package ......................... 3 1.2 Obtaining Biopython ......................................... 4 1.3 Installation .............................................. 4 1.3.1 Installing from source on UNIX ............................... 4 1.3.2 Installing from source on Windows ............................. 5 1.3.3 Installing using RPMs .................................... 6 1.3.4 Installing with a Windows Installer ............................. 6 1.3.5 Installing on Macintosh ................................... 6 1.4 Making sure it worked ........................................ 7 1.5 FAQ .................................................. 7 2 Quick Start { What can you do with Biopython? 9 2.1 General overview of what Biopython provides ........................... 9 2.2 Working with sequences ....................................... 9 2.3 A usage example ........................................... 13 2.4 Parsing biological file formats .................................... 13 2.4.1 General parser design .................................... 13 2.4.2 Writing your own consumer ................................. 14 2.4.3 Making it easier ....................................... 16 2.4.4 FASTA files as Dictionaries ................................. 17 2.4.5 I love parsing { please don't stop talking about it! .................... 18 2.5 Connecting with biological databases ................................ 19 2.6 What to do next ........................................... 20 3 Cookbook { Cool things to do with it 21 3.1 BLAST ................................................ 21 3.1.1 Running BLAST over the internet ............................. 21 3.1.2 Parsing the output from the WWW version of BLAST .................. 22 3.1.3 The BLAST record class ................................... 23 3.1.4 Running BLAST locally ................................... 26 3.1.5 Parsing BLAST output from local BLAST ......................... 26 3.1.6 Parsing a file full of BLAST runs .............................. 27 3.1.7 Finding a bad record somewhere in a huge file ...................... 28 3.1.8 Dealing with PSIBlast .................................... 29 3.2 SWISS-PROT ............................................. 29 3.2.1 Retrieving a SWISS-PROT record ............................. 29 3.3 PubMed ................................................ 31 3.3.1 Sending a query to PubMed ................................. 31 3.3.2 Retrieving a PubMed record ................................ 31 3.4 GenBank ............................................... 32 1 3.4.1 Retrieving GenBank entries from NCBI .......................... 32 3.4.2 Parsing GenBank records .................................. 33 3.4.3 Making your very own GenBank database ......................... 34 3.5 Dealing with alignments ....................................... 34 3.5.1 Clustalw ............................................ 35 3.5.2 Calculating summary information ............................. 36 3.5.3 Calculating a quick consensus sequence .......................... 36 3.5.4 Position Specific Score Matrices ............................... 37 3.5.5 Information Content ..................................... 38 3.5.6 Translating between Alignment formats .......................... 39 3.6 Substitution Matrices ........................................ 40 3.6.1 Using common substitution matrices ............................ 40 3.6.2 Creating your own substitution matrix from an alignment ................ 40 3.7 More Advanced Sequence Classes { Sequence IDs and Features ................. 41 3.7.1 Sequence ids and Descriptions { dealing with SeqRecords ................ 42 3.7.2 Features and Annotations { SeqFeatures .......................... 43 3.8 Classification ............................................. 46 3.9 BioCorba ............................................... 46 3.10 Miscellaneous ............................................. 46 3.10.1 Translating a DNA sequence to Protein .......................... 46 4 Advanced 47 4.1 Sequence Class ............................................ 47 4.2 Regression Testing Framework ................................... 47 4.2.1 Writing a Regression Test .................................. 47 4.3 Parser Design ............................................. 48 4.3.1 Design Overview ....................................... 48 4.3.2 Events ............................................. 48 4.3.3 'noevent' EVENT ...................................... 48 4.3.4 Scanners ............................................ 48 4.3.5 Consumers .......................................... 49 4.3.6 BLAST ............................................ 49 4.3.7 Enzyme ............................................ 50 4.3.8 Fasta ............................................. 51 4.3.9 Medline ............................................ 51 4.3.10 Prosite ............................................. 52 4.3.11 SWISS-PROT ........................................ 52 4.4 Substitution Matrices ........................................ 53 4.4.1 SubsMat ............................................ 53 4.4.2 FreqTable ........................................... 56 5 Where to go from here { contributing to Biopython 57 5.1 Maintaining a distribution for a platform ............................. 57 5.2 Bug Reports + Feature Requests .................................. 58 5.3 Contributing Code .......................................... 58 6 Appendix: Useful stuff about Python 59 6.1 What the heck in a handle? ..................................... 59 6.1.1 Creating a handle from a string ............................... 59 2 Chapter 1 Introduction 1.1 What is Biopython? The Biopython Project is an international association of developers of freely available Python (http://www. python.org) tools for computational molecular biology. The web site http://www.biopython.org provides an online resource for modules, scripts, and web links for developers of Python-based software for life science research. Basically, we just like to program in python and want to make it as easy as possible to use python for bioinformatics by creating high-quality, reusable modules and scripts. 1.1.1 What can I find in the biopython package The main biopython releases have lots of functionality, including: The ability to parse bioinformatics files into python utilizable data structures, including support for • the following formats: { Blast output { both from standalone and WWW Blast { Clustalw { FASTA { GenBank { PubMed and Medline { Expasy files, like Enzyme, Prodoc and Prosite { SCOP, including 'dom' and 'lin' files { Rebase { UniGene { SwissProt Files in the supported formats can be iterated over record by record or indexed and accessed via a • Dictionary interface. Code to deal with popular on-line bioinformatics destinations such as: • { NCBI { Blast, Entrez and PubMed services { Expasy { Prodoc and Prosite entries Interfaces to common bioinformatics programs such as: • 3 { Standalone Blast from NCBI { Clustalw alignment program. A standard sequence class that deals with sequences, ids on sequences, and sequence features. • Tools for performing common operations on sequences, such as translation, transcription and weight • calculations. Code to perform classification of data using k Nearest Neighbors, Naive Bayes or Support Vector • Machines. Code for dealing with alignments, including a standard way to create and deal with substitution • matrices. Code making it easy to split up parallelizable tasks into separate processes. • GUI-based programs to do basic sequence manipulations, translations, BLASTing, etc. • Extensive documentation and help with using the modules, including this file, on-line wiki documen- • tation, the web site, and the mailing list. Integration with other languages, including the Bioperl and Biojava projects, using the BioCorba • interface standard (available with the biopython-corba module). We hope this gives you plenty of reasons to download and start using Biopython! 1.2 Obtaining Biopython Biopython's internet home is at, naturally enough, http://www.biopython.org. This is the home of all things biopython, so it is the best place to start looking around if you are interested. When you feel ready to dive in and start working with the code, you have three choices: 1. Release code { We made available both stable and developer's releases on the download page (http: //www.biopython.org/Download/). The stable releases are likely to be more well tested, while the development releases are closer to what is in CVS, and so will probably have more features. The releases are also available both as source and as installers (rpms and windows installers, right now), so you have some choices to pick from on releases if you prefer not to deal with source code directly. 2. CVS { The current working copy of the Biopython sources is always available via CVS (Concurrent Ver- sions Systems { http://www.cvshome.org/). Concise instructions for accessing this copy are available at http://cvs.biopython.org. Based on which way you choose, you'll need to follow one of the following installation options. Read on for the information you are interested in. 1.3 Installation 1.3.1 Installing from source on UNIX Biopython uses Distutils, which is the new standard python installation package. Copies are available