COVER STORY www.photocase.com

Creating network apps with Twisted TWISTING PYTHON

The Twisted framework makes networking capabilities to Python pro- Although Python’s standard libraries grams. could handle this (the asyncore module it so easy to create network- If you are coding an email client with has basic functionality for switching be- multiple protocol support (POP 3, SMTP, tween multiple I/ O channels within a aware applications in Python. IMAP), the Twisted framework will re- thread), Twisted implements this design move the need to start from scratch. at a higher level in its protocols, inter- Twisted speaks all the major Twisted Mail has all the major mail pro- faces, and components. This lets pro- Internet protocols, from mail tocols. Twisted also has ready-to-run grammers write network applications modules for SSH, SFTP, HTTP (includ- that do without additional processes and through chat, and it can han- ing HTTP/ 1.1), DNS, NNTP, and Jab- threads while at the same time handling ber. If you insist on re-inventing the multiple I/ O channels. dle encryption. We’ll show you wheel – by implementing your own protocols for example – the twisted. Highly Modular how to set up a personal web cred and twisted.spread Twisted mod- To use Twisted to make your own soft- ules can help to simplify the job. ware network-aware, you first need to server with Twisted. BY find out which part of the framework Asynchronous you will need. The developers split MARKUS FRANZ Networking Twisted into multiple sub-projects when Twisted is basically an asynchronous moving from version 1 to 2, with the aim network framework. In contrast to of adding more clarity. Table 1 has a list ost eventually other libraries, the Twisted functions do of the most important elements, and face the task of adding network not block when they are called. Applica- there is a complete list at [2]. Mcommunication capabilities to tions just keep on running until they are Twisted requires the Interface their applications. If it is simply a case of told that the required data is available. module, which implements interfaces manipulating web content, there are simple answers, but more complex func- Table 1: Major Twisted Modules tions, such as adding full-fledged email twisted Framework for asynchronous applications, the basis for all Twisted subprojects capabilities or a complete web server, in- twisted.conch Implementation of the SFTP and SSH protocols for clients and servers volve much more effort. twisted.web HTTP protocol for clients and servers Although the Python standard library twisted.web2 Support for the HTTP/ 1.1 protocol as a server framework; this package is still has modules to match most walks of a under development at present, and should not be used for critical applications ’s daily life (batteries in- twisted.mail Implementation of the SMTP, IMAP, and POP protocols for clients and servers cluded, in Python-speak), special appli- twisted.names DNS protocol support for clients and servers cations require external packages. twisted.news NNTP protocol for clients and servers Twisted [1] is a well-organized and pow- twisted.words Module for chat or instant messaging applications erful collection of modules for adding

30 ISSUE 64 MARCH 2006 W W W. L I N U X - M A G A Z I N E . C O M Twisted COVER STORY

that the Python language core lacks. If DB-API 2.0. This makes access to you just need to use a single project, MySQL, Oracle, or PostgreSQL databases Client (Browser) such as Twisted Web or Twisted Mail, child’s play. The module uses an asyn- you can download the required module chronous interface, which can run in from the project homepage. Alterna- multiple threads, without losing thread tively, Twisted Sumo [3] has all (stable) safety in an event-based Twisted main HTTP modules, including a Zope interface. loop. The main loop occurs within the After unpacking, give the python twisted.internet module and is known as setup.py install command. If the Zope the reactor. It implements the infinite »web.server.Site« interface is not installed, the installer loop of the program in which Twisted will display an error message and quit. handles various events. The reactor pro- The twisted.cred module handles au- vides the underlying interface to Twist- thentication in client-server communica- ed’s major internal capabilities, such as »web.server.Request« tions. It allows multiple network proto- network connections, threading, or cols to connect to a system, to authenti- event handling. cate, and to exchange data. For example, Other modules, such as twisted.proto- POP 3 support in Twisted provides a cols or twisted.manhole, are very rarely »web.resource.Resource« combination of the username and pass- needed in practical applications. Conch word to open the requested mailbox. implements version 2 of the Secure Shell The so-called Perspective Broker is im- protocol for Twisted. The how-to at [5] Twisted portant here; the broker provides access discusses the implementation of an SSH to remote objects and implements object client with Conch in a few simple steps. Figure 1: Web request process within the copying, referencing, and caching. Web or application servers are one of Twisted framework. Twisted’s most interesting fields of appli- Database Connection cation and use twisted.web or twisted. also has the full range of functions re- twisted.enterprise provides a database web2. The powerful template toolkit for quired for programming HTTP clients. interface that is compatible with Python- this task is titled Nevow [6]. Twisted The twisted.web API supports multiple

advertisement

WWW.LINUX - MAGAZINE.COM ISSUE 64 MARCH 2006 31 COVER STORY Twisted

abstraction layers: from simple web serv- in handy for downwardly compatible ap- ate scripts for installing and removing ers, through session support, interacting plications. the server. application servers, and distributed web- Listing 1 shows a simple web server sites – the options are many. Frozen Server and demonstrates how powerful Twisted When a client request reaches a web The Twisted framework has a collection is. You could use this code to enable server, the server creates a request object of command line programs, which pre- web-based configuration of your applica- and hands the object back to the re- pare Twisted applications for special sce- tion. The first couple of lines load the re- source system, which creates the re- narios [7]. The mktap and tapconvert quired modules. Line 6 creates a new sponse (Figure 1). tools create Tap, Tas, or Tax formatted server with a web root directory for Besides twisted.web, there is also the files from the Python source code. The HTML documents – /var/www/htdocs in twisted.web2 module, which is still code can then be used with the various our example. The following instruction under development. The major improve- Twisted servers, for example: Web, FTP, tells the reactor to listen for requests on ments are as follows: or IRC. port 7777. The reactor.run() call • HTTP 1.1 support The twistd tool brings Tap files to life. launches the web server. • Internal and predefined output filters, Strictly speaking, twistd is not required Listing 2 demonstrates a web server for gzip-compressed serving of web to run Twisted applications, but it does that adds a number of options to the pages make things easier, as it takes control of basic framework in Listing 1. First of all, • Separation of high and low level re- the reactor and handles starting and lines 9 through 11 enable script sup- quest handling quitting the application. Additionally, port: we want the server to hand files • Correct analysis of HTTP headers twistd supports the selection of a differ- with a .pl extension to the Perl inter- • Vastly improved URL rewriting, if used ent reactor type, allowing an application preter, /usr/bin/perl, and to return its in combination with a proxy to run in daemon mode or write logfiles. output. Support for PHP or other lan- Although there are major enhancements The tap2deb and tap2rpm programs guages could just as easily be enabled at in twisted.web2 in comparison to the compress finished server applications for this point. previous version, the developers do not distribution in Debian or RPM package The putChild() method in line 14 sets recommend using the new module at formats. The tools automatically gener- the CGI directory to /var/www/cgi-bin, present, one of the reasons being that the new module is slower than its prede- Listing 2: Extended Web Server cessor. The last major module is Twisted Mail, 01 # Load modules which implements SMTP, POP 3, and 02 from twisted.internet import reactor IMAP 4. Besides protocols, the module 03 from twisted.web import static, server, twcgi can also read and write the Maildir mail- 04 box format. There is also a preconfig- ured combination of SMTP and POP 3 05 # set root directory for mail servers and virtual hosting 06 my_server = static.File('/var/www/htdocs') systems. And Twisted Mail understands 07 most Sendmail options, which can come 08 # Evaluate Perl scripts Listing 1: Simple Web 09 class PerlScript(twcgi.FilteredScript): Server 10 filter = '/usr/bin/perl' # path to Perl interpreter 01 # Load modules 11 my_server.processors = {'.pl': PerlScript} 02 from twisted.internet import 12 reactor 13 # Set and enable CGI directory 03 from twisted.web import 14 my_server.putChild('cgis', twcgi.CGIDirectory('/var/www/cgi-bin')) static, server 15 04 16 # Directories for other targets 05 # Set root directory 17 my_server.putChild('doc', static.File('/var/www/doc')) 06 my_server = static.File('/var/ 18 www/htdocs') 19 # Index files 07 20 my_server.indexNames = ['index.html', 'index.htm', 'index.pl'] 08 # Launch web server on port 21 7777 22 # Launch web server on port 7777 09 reactor.listenTCP(7777, server.Site(my_server)) 23 reactor.listenTCP(7777, server.Site(my_server)) 10 reactor.run() 24 reactor.run()

32 ISSUE 64 MARCH 2006 WWW.LINUX - MAGAZINE.COM Twisted COVER STORY

thus allowing access to the scripts at more information on the available op- known protocols. This gives program- http://servername:7777/cgis. The tions. mers secure, fast, stable, and flexible method also specifies the path for the network applications at the click of a doc directory, /var/www/doc. The index- Simple but Powerful mouse. ■ Names variable specifies which files the This article can only give you a quick server looks for during a directory re- overview of Twisted. The framework, INFO quest. The order is defined by the file- along with Twisted Mail, Conch, Twisted [1] Twisted Matrix project page: names specified. Web(2), and the other sub-projects, form http:// www. twistedmatrix. com The following Twisted tools could be the basis for many professional network [2] Overview of all Twisted projects: used as an alternative to the web server applications – even NASA uses Twisted http:// www. twistedmatrix. com/ code: Matrix [8]. projects Helping Python applications reach for [3] Twisted Sumo: mktap web U the stars makes life a lot easier for over- http:// twistedmatrix. com/ projects/ core --path /var/www/htdocs U worked programmers back here on [4] Zope Interface: --port 7777 earth. Twisted clearly reduces the effort http:// www. zope. org/ Wikis/ Interfaces twistd --file web.tap involved in developing a client or server by removing the need to implement [5] SSH client with Conch: http:// twistedmatrix. com/ projects/ mktap creates a pre-configured Tap file. conch/ documentation/ howto/ conch_ In this case, mktap sets the root direc- Markus Franz runs a business advi- client. html tory to /var/www/htdocs and the port to sory service for companies involved [6] Nevow (template system): 7777. The results end up in the web.tap in IT projects. He develops Internet http:// divmod. org/ projects/ nevow file, which the twistd server program search engines, including Metager [7] Tools: http:// twistedmatrix. com/ uses as a parameter. After launching, the 2, www.metager2.de, which was im- projects/ core/ documentation/ howto/ server, the logfile, twistd.log, and the PID plemented in Python throughout. Markus is just 17 years old and still basics. html file, which can be used to kill the server

THEAUTHOR attends the Armin Knab High School [8] Twisted users: http:// twistedmatrix. (kill `cat twistd.pid`), reside in the cur- at Kitzingen, Germany. com/ services/ success rent directory. mktap web -help gives you

advertisement