Parallele Programmierung mit Python Martin v. Löwis Überblick • Threads • Nebenläufige Programmierung • Prozesse • Cluster MvL ParProg | Python 2 2011 Threads • Motivation: schneller • Motivation: scheinbar einfachere Programmierung nebenläufiger Probleme • Problem: thread safety • in der Anwendung • im Interpreter MvL ParProg | Python 3 2011 import thread • Seit Python 0.98 (1992; SGI threads + Sun LWP) • Designziel: "native" threads • unterstützte Thread-APIs: POSIX, NT, OS/2, Solaris, SGI, Windows CE, LWP, pth, AtheOS, BeOS • Python/thread_*.h • start_new_thread, exit_thread, allocate_lock, get_ident MvL ParProg | Python 4 2011 import threading • Python 1.5.2 (1999) • class Thread • Lock, RLock, Condition, Event, Semaphore, BoundedSemaphore • class local • dummy_thread.py • Queue.py MvL ParProg | Python 5 2011 GIL • Global Interpreter Lock • Ziel: thread safe interpreter • Geschützte Resourcen: • Referenzzähler • Container (dict, list) • Speicherverwaltung (obmalloc, GC) • Interpreter state • Current thread state MvL ParProg | Python 6 2011 GIL (2) • Nebenläufigkeit: • Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS • sys.get/setcheckinterval • 3.2: GIL drop request/eval breaker/sys.getswitchinterval MvL ParProg | Python 7 2011 Free Threading • Greg Stein (ca. 2000) • main problem: dictionary synchronization • Uniprocessor: speedup 0.6 • two processors: 1.2 • three processor: 1.6 MvL ParProg | Python 8 2011 Nebenläufigkeit • Python workload: web application • Django, Zope/Grok, TurboGears, web2py • mehrere gleichzeitige HTTP-Anfragen • I/O: Sockets, Datenbank, Dateisystem MvL ParProg | Python 9 2011 import asyncore • Sam Rushing, Python 2.0 • Koordinierte Verwaltung von sockets • select, poll • class asyncore.dispatcher (handle_read, handle_write, ...) • Reactor-Pattern • asyncore.loop() MvL ParProg | Python 10 2011 Twisted Python • J.P. Calderone, G. Lefkowitz, M. Zadka, I. Shtull-Tauring, ... • "networking engine" (web server, IRC server, mail server, DNS server, * client) • Programmiermodell für nebenläufige Verarbeitung von Datagrammen MvL ParProg | Python 11 2011 Twisted: Client • class EchoClient(protocol.Protocol): """Once connected, send a message, then print the result.""" def connectionMade(self): self.transport.write("hello, world!") def dataReceived(self, data): "As soon as any data is received, write it back." print "Server said:", data self.transport.loseConnection() def connectionLost(self, reason): print "connection lost" MvL ParProg | Python 12 2011 Twisted: Deferred from twisted.internet import reactor from twisted.web.client import getPage from twisted.internet.defer import DeferredList def listCallback(results): print results def finish(ign): reactor.stop() def test(): d1 = getPage('http://www.google.com') d2 = getPage('http://yahoo.com') dl = DeferredList([d1, d2]) dl.addCallback(listCallback) dl.addCallback(finish) test() reactor.run() MvL ParProg | Python 13 2011 Twisted: Applications • Bittorrent • Zope3 • BuildBot • Launchpad MvL ParProg | Python 14 2011 Stackless Python • Christian Tismer • Ziel: "unbegrenzte" Rekursion • Spaghettistack • Ziel: beliebig viele Threads • Ziel: "persistente" Threads MvL ParProg | Python 15 2011 Stackless: Implementierungsstrategie • Version 1: Umformulierung des Interpreters zur Vermeidung von C-Rekursion • Problem: Callbacks in C-Code (map, Tkinter) • Version 2: "Weg-Kopieren" des Stacks in den Heap • Anpassung des Stackpointers (setjmp, longjmp) MvL ParProg | Python 16 2011 Stackless: API • Tasklets (Microthreads): • t = stackless.tasklet(aCallable) • t.insert, t.remove, stackless.run() • Channels • Scheduler • Serialisierung MvL ParProg | Python 17 2011 Stackless: Anwendungen • Eve Online (CCP Games) • IronPort (Cisco) • Sylphis3D • ... MvL ParProg | Python 18 2011 Prozesse • import subprocess • import multiprocessing • Web: • Application Server? • Apache: mod_python, mod_wsgi, mod_fastcgi • CGI MvL ParProg | Python 19 2011 Futures (PEP 3148) • Brian Quinlan (Python 3.2) • Task Queues • concurrent.futures.submit • liefert Future Objekt (.done(), .cancel(), .result(timeout?), .exception (), .add_done_callback(cb)) • Thread pools, process pools MvL ParProg | Python 20 2011.