Victor Stinner’s Notes Documentation Release 1.0
Victor Stinner
Sep 22, 2021
Contents
1 My Projects 3
2 Talks 5
3 Old Projects (2000-2009) 11
4 Victor Stinner: Contact Me 15
5 My contributions to Python 19
6 My Contributions to Free Softwares 25
7 Analyze of PyPy warmup in performance benchmarks 31
8 C programming language 73
9 Unicode 81
10 Multithreading Programming 87
11 Benchmarks 91
12 FreeBSD 97
13 Survivor Guide to Develop on Windows 101
14 GDB: GNU debugger 109
15 Unsorted Notes 117
16 Inspect an ELF binary file 131
17 systemd 133
18 Python in Fedora 139
19 Red Hat and Python 141
20 Fedora 143
i 21 GNOME and Wayland 147
22 vim for developer 153
24 See also 157
25 Indices and tables 159
Index 161
ii Victor Stinner’s Notes Documentation, Release 1.0
Contents:
Contents 1 Victor Stinner’s Notes Documentation, Release 1.0
2 Contents CHAPTER 1
My Projects
Projects of Victor Stinner. See also: My talks, my contributions to Free Softwares and my old projects.
1.1 Websites
• Python Development (created in Juny 2018) • Victor Stinner’s Blog 3 (new blog, created in 2015) • Victor Stinner’s Notes (this site, created in 2014)
3 Victor Stinner’s Notes Documentation, Release 1.0
1.2 Documentation
• Faster CPython (source code: faster_cpython at github) • Programming with Unicode (source code: unicode_book at github)
1.3 Python Projects
• fatoptimizer: static optimizer for Python 3.6 using function specialization with guards. It is implemented as an AST optimizer. It is part of the FAT Python project. • bytecode: API to modify Python bytecode, and a peephole optimizer. • pyfailmalloc: Debug tool for Python injecting memory allocation faults to simulate a low memory system to test how your application handles MemoryError exceptions. • sixer: add Python 3 support to Python 2 applications using the six module. • Hachoir: Python library that allows to view and edit a binary stream field by field. In other words, Hachoir allows you to “browse” any binary stream just like you browse directories and files. A file is split in a tree of fields, where the smallest field is just one bit.
1.4 Other Projects
• “misc” repository: – my “dot” files, configuration files: bashrc, hgrc, gitconfig, etc. – some command line program: apply_patch.py, scm.py – some Python scripts – some shell scripts: apt_get.sh, fedora_new_install.sh
4 Chapter 1. My Projects CHAPTER 2
Talks
See also my projects and my old projects. All my talks at available at Github: vstinner/conf.
2.1 Conferences 2011-today
• May 2020, Pycon US, Online (because of the COVID-19) – Python Steering Council Community Address: Brett Cannon, Barry Warsaw, Carol Willing, Thomas Woul- ters, Victor Stinner. • July 2019, EuroPython at Basel (Switzerland) – “Python Performance: Past, Present and Future” keynote – Slides – Video (9h YouTube video, link to the start of the my talk, at 31:15) • May 2019, Python Language Summit (during Pycon US) at Cleveland (Ohio, USA) – Python Core Developer Mentorship with Chery Sabella and Pablo Galindo Salgado – slides • May 2019, Python Language Summit (during Pycon US) at Cleveland (Ohio, USA)
5 Victor Stinner’s Notes Documentation, Release 1.0
– Status of stable API and stable ABI in Python 3.8: lightning talk (5 min) • May 2019, Pycon US at Cleveland (Ohio, USA) – How to engage Python contributors in the long term? Tech is easy, people are hard. – video on Youtube – slides (PDF) • January 2019, Devconf.CZ at Brno (Czech Republic) – Howto engage Python contributors in the long term? – slides (PDF) • October 2018, Pycon FR at Lille (France) – Comment avoir plus de contributeurs à Python: PDF slides – Comment avoir plus de contributeurs à Python: video on Youtube, the video has subtitles in french (you can enable automatic translation to english, or maybe other languages) • July 2018, EuroPython at Edinburgh, Scotland, UK (keynote) – Python 3: ten years later. (same talk than FOSDEM 2018) • June 2018, PyLondonium at London, UK (keynote) – Python 3: ten years later. (same talk than FOSDEM 2018) • May 2018, Pycon US at Cleveland, Ohio, USA: 3 talks! – Python 3: ten years later (same talk than FOSDEM 2018): slides (PDF) and Youtube video – Mentoring and diversity for Python: slides (PDF) and LWN article. I focused on contributors and mentor- ing, then Mariatta Wijaya talked about diversity. – Stable ABI (slides, PDF) • April 2018, Pycon Italy at Firenze (keynote) – Python 3, ten years later: keynote. (same talk than FOSDEM 2018) • February 2018, FOSDEM at Brussels – Looking back at Python evolutions of the last 10 years – Slides (PDF) – Sorry for the bad sound in the video, there was an issue with sound during my talk and the video team and only fixed it after my talk: “we had some issues with microphone charging cables and other sillyness at some point”. • May 2017, Pycon US at Portland (Oregon, USA) – LWN article: Making Python faster – Talk: Optimizations which made Python 3.6 faster than Python 3.5 – Video (Youtube) – Slides (PDF): Optimizations which made Python 3.6 faster than Python 3.5 • May 2017, Language Summit during Pycon US at Portland (Oregon, USA) – LWN article: Keeping Python competitive – Talk: Python performance
6 Chapter 2. Talks Victor Stinner’s Notes Documentation, Release 1.0
– Slides (PDF): Python performance • February 2017, FOSDEM at Brussels (Belgium) – Video: How to run a stable benchmark – Slides (PDF): How to run stable benchmarks • July 2016, EuroPython at Bilbao (Spain) – Slides (PDF): FAT Python, a new static optimizer for CPython 3.6 – Video on Youtube: Victor Stinner - FAT Python: a new static optimizer for Python 3.6 • February 2016, FOSDEM at Brussels (Belgium): – Slides: FAT Python: New static optimizer for CPython 3.6 – website – (video lost, sorry) • 2015, /var/var at Toulon (France: – slides: (fr) Projet FAT Python: Optimiseur statique pour Python • 2015, EuroPython at Bilbao (Spain): – asyncio community, one year later – slides (PDF)(copy at europython website) – slides at slideshare • 2014, Pycon FR at Lyon (France): – Exploration de la boucle d’événements asyncio – slides (PDF) – slides at SpeackerDeck – video • 2014, Pycon “US” at Montréal (Canada): – Track memory leaks in Python – slides (PDF) – slides at SpeakerDeck – video • 2013, Pycon FR at Strasbourg (France): “Traquer les fuites mémoires Python” – slides (PDF) – slides at slideshare – video • 2013, FOSDEM at Brussels (Belgium): “Two projects to optimize Python” (astoptimizer, register-based byte- code) – slides (PDF) – slides at slideshare • 2012, Pycon FR at Paris (France):
2.1. Conferences 2011-today 7 Victor Stinner’s Notes Documentation, Release 1.0
– Processus de développement de CPython – Nouveautés de Python 3.3 • 2011, Pycon US at Atlanta (USA): – https://github.com/vstinner/talks/tree/master/2011-PyconUS-Atlanta – video (blip.tv) – video (pyvideo) • 2011, Pycon FR at Rennes (France): – “Développement de CPython”:
* slides (PDF) * slides at slideshare – Python : langage homogène, explicite et efficace
2.2 Conferences 2005-2009
• 2009, Pycon FR at Paris (France): – Comprendre les erreurs Unicode: slides, video – Contribuer à Python – Python bling bling: slides, video – Interview of myself • 2009, OSDC at Paris (France): – Génerer des nombres aléatoires avec Hasard. • 2009: FOSDEM at Brussels (Belgium): – Fusil the fuzzer – Video of the demo (fusil-python.ogg) – Video: FOSDEM 2009 Fusil fuzzing • 2008, RMLL at Mont de Marsan (France): – Assurance qualité avec Fusil le fuzzer – https://github.com/vstinner/talks/tree/master/2008-RMLL • 2008, Pycon FR at Paris (France): – PyPy – PyPy: video – Python 3 aka “Pytho 3000” – Python 3: video • 2007, SSTIC at Rennes (France): – Project Fusil • 2007, Pycon FR at Paris (France): https://github.com/vstinner/talks/tree/master/2007-Pycon-Paris
8 Chapter 2. Talks Victor Stinner’s Notes Documentation, Release 1.0
• January 2007, AAM (Appel À Mousser) at Strasbourg (France): – Hachoir • 2005, UTBM at Belfort (France), Lolut association: Atelier sécurite PHP and MySQL – PHP “crackme” exercices: vulnerable PHP pages – Failles en PHP et injection SQL • 2005, UTBM at Belfort (France), Lolut association: Atelier sécurité of C programming – Introduction générale à la sécurité informatique (french) – Mots de passe, chiffrement et signature – Aide-mémoire sur les failles en C – C exercices: vulnerable C programs • 2005, Gameover at Limoges (France): – Wormux (french), conference given with Lawrence Azzoug.
2.2. Conferences 2005-2009 9 Victor Stinner’s Notes Documentation, Release 1.0
10 Chapter 2. Talks CHAPTER 3
Old Projects (2000-2009)
See also: My current projects and my talks.
3.1 Websites
• La page de Haypo (2005) • Turbo Pascal (2004)
11 Victor Stinner’s Notes Documentation, Release 1.0
3.2 Old websites (dead)
• Wiki (last new article in december 2013). • Blog Haypo (last article in 2011) • Wordpress Blog (dead URL: http://www.haypocalc.com/wordpress/) (last article in 2011) These sites were hosted at the haypocalc.com domain created in december 2001, but the domain died in december 2017, 16 years later.
3.3 Articles on linuxfr.org
Some of my articles: • Python 3.4 est sorti avec 7 nouveaux modules (2014/03) • Justice Free publie enfin ses patchs sur les logiciels libres (2011/09) • Python 3.2 (2011/02) • Patch pour le noyau Linux améliorant l’interactivité entre les applications console et Xorg (2010/11) • Python 2.7 (2010/07) • Sortie de la version 2.11 de la bibliothèque standard C GNU (glibc) (2009/11) • Intel ne maintient plus le pilote Linux Poulsbo depuis un an et demi (2009/10) • Python arrive en version 3.1 (2009/07) • Debian remplace la glibc par eglibc (2009/05) • Nouvelle version majeure de Python (2.6) (2008/10)
3.4 Paper Articles
• Netfilter et le filtrage du protocole IPv6 (french): GNU/Linux Magazine HS 41 (April 2009) • Hors-série Linux Mag : Explorez les richesses du langage Python (january/february 2009). I wrote 4 articles: – Nouveautés de Python 2.6 – Nouveautés de Python 3.0 – Trucs et astuces – Ctypes et Python • “Pratique du fuzzing avec Fusil” (french), MISC magazine n°39 (september 2008) • Comment réaliser un fuzzer ? (french), MISC magazine n°36 (march 2008)
3.5 Python projects
• pytracemalloc: debug tool to trace memory blocks allocated by Python. The module is part of Python standard library since Python 3.4, I maintain a backport for Python 2.7 and 3.3 (it should work on Python 2.6-3.3).
12 Chapter 3. Old Projects (2000-2009) Victor Stinner’s Notes Documentation, Release 1.0
• faulthandler: Dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal. The module is part of Python 3.3 and newer. • pysandbox: Sandbox to run untrusted Python code. Project stopped because it is broken by design. • python-ipy: The project is still active, but I don’t maintain it anymore. Python classes and tools for handling of IPv4 and IPv6 addresses and networks. I don’t need this module anymore and so I am no more interested to maintain it, the new maintainer is Jeff Ferland aka autocracy. Python 3.3 now includes ipaddress, a concurrent module to handle IP addresses and networks. • registervm: My fork of Python 3.3 using register-based bytecode, instead of stack-code bytecode. Read REG- ISTERVM.txt • python-ptrace: Python binding of ptrace library to debug processes on UNIX and BSD.
3.6 Other projects
• Trollius: portage of the Tulip project (asyncio module, PEP 3156) on Python 2, asynchronous input/output library. • Warmux, previously known as “Wormux”: Open source clone of the famous 2D game “Worms” by Team17.
3.7 Old removed projects
• Hasard: pseudo-random number generator (PRNG) library. • Fusil: Fusil is a Python library used to write fuzzing programs. It includes fuzzers for Firefox, ClamAV, Mplayer, Python, etc. I am no more working actively on the project, but it still works. • aioeventlet: implementation of the asyncio API (PEP 3156) on top of eventlet. • aiogevent: implementation of the asyncio API (PEP 3156) on top of gevent. • astoptimizer: experimental optimizer for Python code working on the Abstract Syntax Tree (AST, high-level representration). It does as much work as possible at compile time. • macfly: tool allowing to run one or more programs with a shifted clock compared to system clock. project written at INL for the french CNES (National Centre for Space Studies) • HaypoCALC: A formal calculator in text terminal. Existing functions : ln,lg, basen, gcd, lcm, ncr, npr, fac- tor(int), cos,sin,tan, acos,asin,atan, derive,taylor, . . . Work on Linux and Windows, programmed in C++. • happyboom: Prototype of a library to write a turn-based game, it is mostly a network protocol.
3.8 Workshops Lolut
Dans le cadre du club Lolut auquel j’appartiens, j’ai organisé plusieurs ateliers : • En cours de préparation : Ateliers sécurité durant le semestre 2005/2006 (cf. ateliers automne 2005) • Atelier sécurité - Introduction générale et failles avec le langage C (6 octobre 2005) : Compte rendu, présenta- tion, exercices et liens sont disponibles. • Atelier création de sites web avec XML/XSL/CSS (08 avril 2004) • Compte rendu atelier Gimp (25 mars 2004)
3.6. Other projects 13 Victor Stinner’s Notes Documentation, Release 1.0
• ‘’Programmation C/C++ sous Linux” (18 avril 2002) : pas de compte rendu malheureusement. • ‘’Atelier PHP” (9 janvier 2003), pas de compte rendu, mais les documents sont disponibles ainsi que les exem- ples. Je l’ai organisé avec [[Damien Boucard]] et Laurent Adda (ce dernier a quitté l’UTBM la même année . . . ).
3.9 “Childhood”
I like programming. I started with examples copy-pasted manually from the magazine “Science & Vie Junior”, pro- grams written in “Basic” (Microsoft Quick Basic). Then I moved forward with the programming language “Pascal” (avec Turbo Pascal 7), completed with assembler Intel x86 (Borland Turbo Assembleur 1 and 2). The other major change was the move to the object oriented programming (OOP) with Borland C++ Builder and short tests with gcc on MS-DOS (it didn’t work well). When I started to study at UTBM (engineer school), I switched from Windows to Linux because the 4 computers running Linux were always available and I didn’t thave the Internet at home. I learnt HTML and XHTML (completed with Javascript), then PHP, and finally XML + XSLT + CSS. I learnt some Java at school, and Python at home. I also developed with Visual Basic and Delphi, but I didn’t do anything interesting with them.
14 Chapter 3. Old Projects (2000-2009) CHAPTER 4
Victor Stinner: Contact Me
I am paid by Red Hat to maintain Python upstream and downstream (RHEL, CentOS, Fedora & Software collections). They also cover the cost of travels to Pycon US and EuroPython: meeting people is as important if not more than code. My contributions to Python.
4.1 Contact me
• Email: [email protected] • IRC: my nickname is vstinner on Freenode, I’m usually connected to #python-fr and #python-dev. Previously, I used the nick haypo.
4.2 Public profiles
• Twitter account • Github profile • Bitbucket profile
15 Victor Stinner’s Notes Documentation, Release 1.0
• Flickr profile (photos) • Launchpad • SpeakerHub See also Victor Stinner in Python Core Developer Motivations and Affiliations. Note for myself: see also gravatar.com (ex: used by Fedora Pagure).
4.3 Code statistics
• My profile on Open HUB: statistics on programming
4.4 My projects and talks
See the pages of my projects and my talks.
4.5 About me
• Job Title at Red Hat: Senior Software Engineer • Twitter: “Opensource Python Hacker” • Pycon US 2014: Python core developer since 2010, I’m the author of various Python applications and libraries Python. See my profile on Bitbucket and and Github. I’m now working on OpenStack at Enovance (Paris). • FOSDEM 2013: Python code developer since 3 years, I love hacking free softwares to improve them, especially the Python project. • techs.enovance.com: Victor is a Senior Engineer at eNovance. Python core developer since January 2010, he eats bugs for breakfast and reviews patches for dinner. On his free time, he shares code of open source projects on Bitbucket and Github. • PyDev of the Week: Victor Stinner (February, 2017)
4.6 Awards
• In October 2017, Gregory P. Smith of Google nominated me for a Google Open Source Peer Bonus for my work on CPython • In April 2016, the OpenStack project nominated me for the The Infinite Rebase Shield Award for my work on porting OpenStack to Python 3: In the face of endless rebases, Stinner has tirelessly pushed OpenStack toward Python 3. I get the impression that it has been hard at times to get the necessary reviews for his Python 3 porting work; especially for the projects he’s not well known on, and I hate to see him discouraged. We’re lucky to have such a prolific expert leading the Python 3 porting effort for OpenStack, and I’d like him to know that his work is appreciated. • In August 2015, I got a PSF Community Service Award for my work on CPython
16 Chapter 4. Victor Stinner: Contact Me Victor Stinner’s Notes Documentation, Release 1.0
4.7 Other people talking about me
LWN: • Postponing some feature removals in Python 3.9 by Jake Edge (February, 2020) • Mentoring and diversity for Python, at 2018 Python Language Summit, by Jake Edge (June, 2018) • Python 3, ASCII, and UTF-8 by Jake Edge (December, 2017) • Rationalizing Python’s APIs by Jake Edge (July, 2017) • Making Python faster by Jake Edge (June, 2017) • Keeping Python competitive by Jake Edge (May, 2017) • Python’s os.urandom() in the absence of entropy by Jonathan Corbet (July, 2016) • The failure of pysandbox by Jake Edge (November, 2013) Articles: • Welcome to the Jungle: Under Discussion: The Performance of Python (May 2020), interview of Julien Danjou and me. • Ledende Python-coreudvikler: Eksplosion i popularitet har øget presset (February 26, 2020) by Magnus Boye: article in Danish. • Python Core Developer Mentorship (May 2019) by A. Jesse Jiryu Davis: Talk that I gave with Pablo Galindo Salgado and Cheryl Sabella at the Language Summit 2019 (during Pycon US 2019). • Interview in Julien Danjou’s book “Scaling Python” (Dec 2017) • Interview in Julien Danjou’s book “The Hacker’s Guide to Python” (May 2015) • 8 Python developers that inspired me in 2017 (Dec, 2017) • Understanding your benchmarks and easy tips for fixing them (July, 2017) • PythonBytes podcast. Episode #32: 8 ways to contribute to open source when you have no time (July, 2017); “Michael #2: Keeping Python competitive” part • PyDev of the Week: Victor Stinner (February, 2017) • Montreal Python User Group: Person of the Month: Victor Stinner #MP42 (December, 2013) • developer of note: Victor Stinner (June 2012) by Tshepang Lekhonkhobe
4.7. Other people talking about me 17 Victor Stinner’s Notes Documentation, Release 1.0
18 Chapter 4. Victor Stinner: Contact Me CHAPTER 5
My contributions to Python
5.1 Reports
• 2017 Q3: 2017 Q3 (part 1), 2017 Q3 (part 2), 2017 Q3 (part 3) • 2017 Q2: 2017 Q2 (part 1), 2017 Q2 (part 2), 2017 Q2 (part 3) • 2017 Q1 • 2016 Q4 • 2016 Q3 • 2016 Q2 • 2016 Q1 • 2015 Q4 • 2015 Q3
5.2 Python 3.9 Contributions
• New math.nextafter() and math.ulp() functions. • New os.waitstatus_to_exitcode(): convert a waitpid wait status to an exit code. • New random.randbytes() function. • Add –with-platlibdir option to the configure script and add sys.platlibdir attribute: used by Fedora and Open- SUSE Linux distributions to install files in /usr/lib64 rather than /usr/lib. • Remove tons of deprecated features and deprecated a few more functions. • C API Changes: new functions to access structure members, private functions removed or moved o the internal C API. Many macros converted to static inline functions.
19 Victor Stinner’s Notes Documentation, Release 1.0
5.3 Python 3.8 Contributions
• PEP 587: https://docs.python.org/dev/c-api/init_config.html • New sys.unraisablehook function • New threading.excepthook function • io.IOBase finalizer now logs close() exception using sys.unraisablehook() • _thread.start_new_thread() now logs thread function exception using sys.unraisablehook(), rather than sys.excepthook(), so the hook gets the function which created the thread and a more helpful error message.
5.4 Mentoring, bug triage permission, core developers
I promoted the following developers as core devs: • 2020-04-09: Dong-hee Na(vote) • 2019-09-23: Joannah Nanjekye(vote) • 2019-06-16: Paul Ganssle(vote) • 2019-04-08: Stéphane Wirtel(vote) • 2019-02-19: Cheryl Sabella(vote) • 2018-06-20: Pablo Galindo Salgado(vote) • 2017-12-08: Julien Palard • 2016-11-21: Xiang Zhang • 2016-06-03: Xavier de Gaye • 2011-05-19: Charles-François Natali I gave the bug triage permission to: • 2020-11-13: Hai Shi • 2019-06-06: Zackery Spytz • 2019-02-22: Andrés Delfino • 2019-02-15: Paul Ganssle (is now a core dev) • 2019-02-02: Alexey Izbyshev • 2019-02-01: Joannah Nanjekye (is now a core dev) • 2018-01-18: Pablo Galindo Salgado (is now a core dev) • 2017-12-06: Cheryl Sabella (is now a core dev) • 2017-12-06: Sanyam Khurana
5.5 Python Enhancement Proposals (PEP)
Lisf of my PEPs and PEPs I co-wrote.
20 Chapter 5. My contributions to Python Victor Stinner’s Notes Documentation, Release 1.0
5.5.1 Draft PEPs
PEP Python Status Title 3.10 Draft Hide implementation details from the C API PEP 620
5.5.2 Accepted PEPs
PEP Python Sta- Title tus 3.8 Final Python Initialization Configuration PEP 587 3.7 Final Add new time functions with nanosecond resolution (ex: time.time_ns()) PEP 564 — Final Python Documentation Translations – co-written with Juliend Palard and Naoki PEP IANADA 545 3.7 Final Add a new UTF-8 mode PEP 540 3.6 Final Make os.urandom() blocking on Linux PEP 524 3.6 Final Add a private version to dict PEP 509 3.5 Final Retry system calls failing with EINTR – co-written with Charles-François Natali PEP 475 3.4 Final Add a new tracemalloc module to trace Python memory allocations PEP 454 3.4 Final Make newly created file descriptors non-inheritable PEP 446 3.4 Final Add new APIs to customize Python memory allocators PEP 445 3.3 Final Add monotonic time, performance counter, and process time functions PEP 418
5.5. Python Enhancement Proposals (PEP) 21 Victor Stinner’s Notes Documentation, Release 1.0
5.5.3 Rejected PEPs
PEP Python Status Title — Rejected Organization of the Python community PEP 8015 3.9 Rejected Coordinated Python release PEP 608 3.9 Rejected Python Compatibility Version PEP 606 2.7 Rejected Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 – co-written with PEP 546 Cory Benfield 3.6 Rejected API for code transformers PEP 511 3.6 Rejected Specialize functions with guards PEP 510 3.6 Rejected Chain exceptions at C level PEP 490 3.x Super- Easier suppression of file descriptor inheritance – supersed by my accepted PEP PEP 433 seded 446 3.3 Rejected Add a frozendict builtin type PEP 416 3.3 Rejected Use decimal.Decimal type for timestamps PEP 410 3.3 Deferred Deprecate codecs.StreamReader and codecs.StreamWriter PEP 400
5.5.4 Other contributions to PEPs
• PEP 460: I wrote the first version of the PEP 460 (bytes % args), then rewritten by Antoine Pitrou, to be later superseeded by the PEP 461 written by Ethan Furman. • PEP 471 (os.scandir): I helped Ben Hoyt to implement, test and benchmark his PEP 471
5.6 Major work
• Python 3.6: – PEP 524: Make os.urandom() blocking on Linux – PEP 509: Add a private version to dict • Python 3.5: – I added the following functions: os.get_blocking(fd) and os.set_blocking(fd). See issue #22054: Add os.get_blocking() and os.set_blocking() functions. I added these functions because it’s a common pattern in applications and signal.set_wakeup_fd() now raises an exception if the fd is blocking (I also added this check). – I worked with Charles-François Natali to implement the PEP 475: “Retry system calls failing with EINTR”. The implementation was much more complex than expected, like the implementation of socket.connect() which requires many syscalls (I refactored Modules/socketmodule.c for that). – I wrote the first version of the PEP 460 (bytes % args), then rewritten by Antoine Pitrou, to be later superseeded by the PEP 461 written by Ethan Furman. – I helped Ben Hoyt to implement, test and benchmark his PEP 471 (os.scandir)
22 Chapter 5. My contributions to Python Victor Stinner’s Notes Documentation, Release 1.0
– On Windows, signal.set_wakeup_fd() now also supports socket handles. – The time.monotonic() function is now always available. – New API for C memory allocators to support also calloc() – The __name__ attribute of generator is now set from the function name, instead of being set from the code name. Use gen.gi_code.co_name to retrieve the code name. Generators also have a new __qualname__ attribute, the qualified name, which is now used for the representation of a generator (repr(gen)). – New private _PyTime API to handle timestamps with a resolution of 1 nanosecond. – os.urandom() now uses getrandom() on Linux 3.17 and newer, and getentropy() on OpenBSD 5.6 and newer. – New private _Py_CheckFunctionResult() function to ensure that the C API is used correctly when calling a C function. – Enhance Py_FatalError()
* Display the current Python stack if an exception was raised but the exception has no traceback * Disable faulthandler if an exception was raised (before it was only disabled if no exception was raised) * To display the current Python stack, call PyGILState_GetThisThreadState() which works even if the GIL was released
* Try to flush stdout and stderr. – Issue #23353: complex bug related to exception handling with generators • Python 3.4: – new tracemalloc module (PEP 454) – better handling of MemoryError exceptions – PEP 446: Make newly created file descriptors non-inheritable • Python 3.3: – new faulthandler module – new time functions: time.monotonic, time.perf_counter, time.process_time (PEP 418) • Python 3.0 - 3.2 – Major work on Unicode support to handle all platforms and all corner cases
5.7 April Fool
• [Python-Dev] The next major Python version will be Python 8 • https://mail.python.org/pipermail/python-dev/2016-March/143603.html • https://hg.python.org/cpython/rev/9aedec2dbc01
5.8 Old contributions to Python
Fuzzing on Python using my fuzzer “Fusil”. Accepted patches:
5.7. April Fool 23 Victor Stinner’s Notes Documentation, Release 1.0
• 2008-07-06: invalid ref count on locale.strcoll() error. Patch appliqué dans la révision 65134. • 2008-07-09: bugs in scanstring_str() and scanstring_unicode() of _json module. Patch inspiré du mien commité dans la révision 65147. • 2008-07-06: segfault on gettext(None). Patch appliqué dans la révision 65133. • 2008-07-07: bugs in _sqlite module. Patch appliqué dans la révision 65040 • 2008-07-06: Use Py_XDECREF() instead of Py_DECREF() in MultibyteCodec and MultibyteStreamReader. Patch appliqué dans révision 65038 • 2008-07-07: dlopen() error with no error message from dlerror(). Patch appliqué dans rev 64976, rev 64977 et 64978 • 2008-07-07: missing lock release in BZ2File_iternext(). Appliqué dans le commit 64767. • 2008-07-06: DoS when lo is negative in bisect.insort_right() / _left(). Appliqué dans le commit 64845. • 2008-07-06: audioop.findmax() crashs with negative length. Appliqué dans le commit 64775. • 2008-07-06: invalid call to PyMem_Free() in fileio_init(). Appliqué dans le commit 64758 • 2007-08-13: Improved patches for sndhdr and imghdr • 2007-08-10: Fix the ctypes tests, corrige ctypes pour le passage de str/unicode à bytes/str. • 2007-04-10: Segfaults quand la mémoire est épuisée (rapport de bug avec patch) => patch appliqué (avec un léger changement) dans le commit 54757 (par georg.brandl). • 2007-02-27: trace.py needs to know about doctests. Patch applied the 23 Nov 2007. • 2006-09-06: Bug locale.getdefaultlocale(), lorsque le module _locale est absent, la fonction lo- cale.getdefaultlocale() retourne un charset errorné avec mes locales. Corrigé dans Python 2.5.1. • 2006-08-23: Bug report with patch, La fonction setup() du module distutils refusait un tuple (au lieu d’une liste) pour la commande « register » (le patch a été retouché pour fonctionner sur Python 2.1) • 2005-11-25: bug report + patch. La méthode seek(0,2) d’un objet du module bz2 était boguée dans Python 2.4.2 Other patches (fixed as well): • 2008-07-06: block operation on closed socket/pipe for multiprocessing • 2008-07-06: invalid check of _bsddb creation failure • 2008-07-06: invalid object destruction in re.finditer() • 2007-07-23: Unable to register or upload project (http error 302: moved) • 2007-07-17: Problem with socket.gethostbyaddr() and KeyboardInterrupt
24 Chapter 5. My contributions to Python CHAPTER 6
My Contributions to Free Softwares
See also my projects and my contributions to Python.
6.1 Contributions in 2016
• Fedora Cloud Image, bug reports: – https://bugzilla.redhat.com/show_bug.cgi?id=1405391 – https://fedorahosted.org/fedora-websites/ticket/415 • testrepository •PR Fix two ResourceWarning warnings • Issue Subprocesses pipes (stdout, stderr) are not explicitly closed: ResourceWarning warnings are logged • pip – Issue: pip should use the distro module: platform.linux_distribution() has been deprecated in Python 3.5 – PR: Fix a ResourceWarning in setuptools_build • setuptools – Don’t use deprecated ‘U’ flag to read manifest • python-json-patch – Use inspect.signature() on Python 3 – Add tox.ini • numpy – Use PyMem_RawMalloc on Python 3.4 and newer – Assertion error when running tests on Python 3.6 compiled in debug mode – https://github.com/numpy/numpy/issues/7360
25 Victor Stinner’s Notes Documentation, Release 1.0
• eventlet: – PR #303: Add hubs.default_clock and use time.monotonic – Bug report: Python 3: wsgi doesn’t handle correctly partial write of socket send() when using writelines()
6.2 Contributions in 2015
• monotonic – clock_gettime() is not thread-safe – Don’t use CLOCK_MONOTONIC_RAW #13 – both merged into monotonic 0.5 released at 2015-12-27 • routes: – Add tox.ini – Fix BytesWarning in Mapper.generate() • pep8: fix BytesWarning on Python 3 • eventlet: Python 3 patches – PR #275: Issue #274: Fix GreenSocket.recv_into(). Issue: On Python 3, sock.makefile(‘rb’).readline() doesn’t handle blocking errors correctly – PR #257: Fix GreenFileIO.readall() for regular file – Issue #248: eventlet.monkey_patch() on Python 3.4 makes stdout non-blocking: pull request Fix Green- FileIO.write() – Issue #230: Fix patcher.original, don’t load a module twice when eventlet monkey-patched is not used – tox 1.8 or newer is required – Fix threading monkey-patching on Python 3.4 – Fix threading.Condition with monkey-patching on Python 3.3 and newer #187 on Python 3.3 and newer when monkey patching is used)
6.3 Contributions in 2014
• pip: Fix issue #1433: parse requirements in markers #1472. Since pip 6.0, it is possible to write futures; python_version < '2.7' in requirements.txt. Feature very useful to specify requirements specific to Python 2, or only for an old Python version. • eventlet: Python 3 patches – Fix monkey-patched os.open(): add dir_fd parameter #170 – enhance socketpair code in tpool #167 – Fix monkey_patch() on Python 3 #168 • Mercurial bug reports: – Bug 4516 - Rename information lost after merge+edit+commit amend – Bug 4306 - histedit blocked if i exchange and edit a commit (unknown revision / no node)
26 Chapter 6. My Contributions to Free Softwares Victor Stinner’s Notes Documentation, Release 1.0
6.4 Old Work (2004-2008)
First, see my old projects.
6.4.1 Accepted patches in other projects
• 2008-05-08, PyPy: modules pwd et syslog implémentés avec ctypes (bon maintenant j’ai un compte Subversion chez PyPy, alors j’accepte mes propres contrib’ :-)) • 2008-03-05, PyPy: _locale module implementation in ctypes • 2008-02-21, PyPy: resource module implementation using ctypes • 2007-12-03, Apache: Fix XSS in error page #413. Voir le commit dans Subversion. • 2006-09-06, PyPy: Corrige le module codec pour la casse des charsets (pour être compatible avec CPython) • 2006-08-21, urwid: Patch ‘’setuptools” (appliqué dans la version 0.9.6) • 2006-04-27, Dia : http://bugzilla.gnome.org/show_bug.cgi?id=334771 Patch qui corrige un plantage alétoire lors du “dégroupage” d’un objet] (appliqué dans Dia 0.95) • 2005-06-16, Gnome : Patch pour libgnomeui. Nautilus utilisait 500 Mo de mémoire pour générer une miniature d’une image SVG de 28 Ko ! Mon patch limite au maximum le gaspillage de mémoire. (appliqué dans la version 2.11)
6.4.2 Pending patches
• 2008-07-07, PHP: count_chars() crashs if both arguments are the same reference • 2007-08-16, yui: container css: “cursor: pointer” instead of “cursor: hand”
6.5 INL/EdenWall
During my work at INL/EdenWall, I contributed to many open source softwares: • 2007, iptables: #7080: Don’t silenty exit on failure to open /proc/net/{ip,ip6}_tables_names • libnfnetlink: #6741: fix autogen.sh (sh syntax for string comparaison) • libnetfilter_conntrack: #6721: fix a crash on setting the counters of a conntrack, implement getter for the ATTR_USE attribute • 2006, libnetfilter_conntrack: #6719: Fix XML output syntax • libnfnetlink: #6718: Initialize callback structure • libnetfilter_conntrack: #6716: Fix new API test program (replace ntohs by htons), introduce NFCT_O_PLAIN flag • gcrypt (july 2006): Fix missing initializer warning in gcrypt.h • Microoptimize destruction of unused statitically initialized mutexes • 2005, (lxml library) Invalid use of xmlIO: crash on xmlCharEncCloseFunc() • (CPython) Bugfix for crashes on low-memory conditions
6.4. Old Work (2004-2008) 27 Victor Stinner’s Notes Documentation, Release 1.0
• (Python ctypes) ctypes: wrong calling convention for _string_at. See issue #3554, 3900 was a duplicate of this bug :-/ • PHP: bug report #42817 • Dia: Bug #334771 (Ungroup crashes) fixed • libc: Bug report made by Victor Stinner: vfprintf() segfault with multibyte string and long precision. Ulrich Drepper fixed the bug: see vfprintf patch v1.136 Security vulnerabilities: • 2007-05-22: CVE-2007-2754: FreeType Integer Overflow in TT_Load_Simple_Glyph() • 2007-05-11: CVE-2007-2650: ClamAV OLE2 Parser Denial of Service • 2007-05-10: CVE-2007-2645: Libexif Integer Overflow Vulnerability in exif_data_load_data_entry()
6.6 Fuzzing
Thanks to my project Fusil, I found and sometimes fixed many bugs in various softwares. See the list of crashes found by Fusil.
6.7 Bug reports
Fixed: • 2007-05-07, ImageMagick: Crash in EXIF parser with invalid IFD count. The file also crash gwenview appli- cation. • 2007-04-30, libc: vfprintf() segfault with multibyte string and long precision. • Le bug a été corrigé par Ulrich Drepper : patch vfprintf v1.136 • Rapport de bug Fedora Core • Rapport de bug Debian • 2007-04-28, FreeType: Another bug in TTF (cmap), voir le patch sfnt/sfobjs.c version 1.128 • 2007-04-27, FreeType: Bug in fuzzed TTF file. Voir le patch (dans CVS). Open: • 2008-02-21: PyPy, large-file support and file.seek() • 2008-01-28: Firefox, Venkman crashs on profiling after clearing profile data • 2008-01-28: command-not-found, phpize is missing from program.d database • 2007-10-01: PHP, buffer under- and overflow on clone(null)+array_push() • Diff sur zend_vm_execute.h • Tests de non regression : bug36071.phpt, bug42817.phpt, bug42818.phpt • 2007-07-05, ClamAV: – #561: OLE2: Long (slow) loop in ole2_walk_property_tree() with huge prop_index value – #560: bitset_realloc() is not atomic (avec patch et testcase) – #559: OLE2: Allocate too much memory with invalid file (avec patch et testcase)
28 Chapter 6. My Contributions to Free Softwares Victor Stinner’s Notes Documentation, Release 1.0
• 2007-04-18, ClamAV: Bug in OLE2 file parser (DoS found with fuzzing), dans bugzilla: Bug #466 (fermé au public) • 2007-04-20, ImageMagick: Bug report in TGA and XCF files (DoS found with fuzzing) • 2005-06-16, gdb : Display libc function names instead of address?
6.8 Other
• I contributed to some articles on the french Wikipedia, like: Sténographie.
6.8. Other 29 Victor Stinner’s Notes Documentation, Release 1.0
30 Chapter 6. My Contributions to Free Softwares CHAPTER 7
Analyze of PyPy warmup in performance benchmarks
Contents
• Analyze of PyPy warmup in performance benchmarks – Goal – General notes – Glossary – 2to3: loops=1, warmups=0, values=30 – chameleon: loops=32, warmups=6, values=10 – chaos: loops=16, warmups=20, values=25 – crypto_pyaes: loops=16, warmups=46, values=45 – deltablue: loops=256, warmups=14, values=55 – django_template: loops=4, warmups=12, values=36 – dulwich_log: loops=2, warmups=21, values=18 – fannkuch: loops=1, warmups=59, values=40 – float: loops=4, warmups=25, values=40 – genshi_text: loops=8, BUG! – genshi_xml: loops=2, BUG! – go: loops=2, warmups=87, values=80 – hexiom: loops=64, warmups=36, values=50 – hg_startup: loops=1, warmups=4, values=10 – html5lib: loops=2, warmups=50, values=50
31 Victor Stinner’s Notes Documentation, Release 1.0
– json_dumps: loops=16, warmups=38, values=25 – json_loads: loops=256, warmups=27, values=50 – TODO
7.1 Goal
The purpose of this analysis if to choose the number of warmups and samples for performance benchmarks for PyPy 5.7 on the speed-python server. • small enough to reduce total benchmark runtime • get a mean-stdev of the sample close enough to mean-stdev of all values • if performance has a cycle (usually with a spike at the start or end): compute 5 cycles to use the average of a cycle
7.2 General notes
The analysis is based on a single data set: pypy2_571_warmups.json.gz. • PyPy2 5.7.1 (revision 1aa2d8e03cdf): 64-bit, static binary • perf 1.2 (dev), performance 0.5.5 (dev) • performance hacked to run exactly 10 worker processes, each computes 250 values with 0 warmup • Date: 2017-04-13 22:19 - 2017-04-14 09:01 (10h45) • 67 benchmarks After computing these data, the following 4 microbenchmarks have been removed from performance benchmark suite and so will be ignored here. • call_method • call_method_slots • call_method_unknown • call_simple speed-python server: • cpu: 2 HP DL380 G7 Intel® Xeon® X5680 (3.33GHz/6-core/130W/12MB) FIO Processor Kit • memory: 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Registered CAS-9 Memory Kit • OS: Ubuntu 16.04.1 LTS • Kernel: GNU/Linux 4.4.0-47-generic Script used to get the number of loops computed by perf: import perf suite= perf.BenchmarkSuite.load('pypy.json.gz') for bench in suite: loops= [run._get_loops() for run in bench.get_runs() if run.values] if len(set(loops)) !=1: (continues on next page)
32 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) raise Exception(bench) loops= loops[0] print("%s: loops=%s"% (bench.get_name(), loops))
Must read paper: Virtual Machine Warmup Blows Hot and Cold (december 2016). See also the R changepoint package.
7.3 Glossary
• Loops: number of outer loop iteration, number of times the benchmark function is called to compute one “value”. perf calibrates the number of iterations to get a loop taking at least 100 ms. • Value: Average wall-clock time elapsed to call the number loops times. Pseudo-code:
t0= perf_counter() for _ in range(loops): func() dt= perf_counter() value= float(dt)/ loops
• Warmups: number of ignored values at the beginning of a benchmark. The number is chosen to ignore first values were are usually much higher than the mean of the “steady state” • Run: execution of one worker processes which computed values. In this analysis, each worker process computed exactly 250 values. Example: a result with 10 runs, 8 values, 0 warmups and 250 values means that the benchmark function was called 10x8x250 = 20,000 times in total, and 8x250 = 2,000 times per process, but the result only contains 10x250 = 2,500 values in total (or 250 values per process) since values are average time per loop iteration.
7.4 2to3: loops=1, warmups=0, values=30
• many spikes up to +16% of the mean • no cycle • moving average(50) increases from 1.816 sec to 1.824 sec Overall:
7.3. Glossary 33 Victor Stinner’s Notes Documentation, Release 1.0
Moving average (50 values) of runs:
34 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
7.5 chameleon: loops=32, warmups=6, values=10
Overall:
7.5. chameleon: loops=32, warmups=6, values=10 35 Victor Stinner’s Notes Documentation, Release 1.0
Short cycle of 2 values (skip first 6 values, limit to 30 values):
36 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Large cycle of 42 values (Moving average 1 value, skip first 6 values):
7.5. chameleon: loops=32, warmups=6, values=10 37 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=32, warmups=6, values=10: 7.05 ms +- 0.23 ms • loops=32, warmups=6, values=42: 7.02 ms +- 0.20 ms • LIMIT: loops=32, warmups=6: 7.01 ms +- 0.20 ms
7.6 chaos: loops=16, warmups=20, values=25
Overall:
38 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 5 values (average of runs, skip 20, limit to 50):
7.6. chaos: loops=16, warmups=20, values=25 39 Victor Stinner’s Notes Documentation, Release 1.0
Glitch at values 35..38. Mean: • loops=16, warmups=20, values=25: 6.50 ms +- 0.22 ms • LIMIT: warmups=20: 6.48 ms +- 0.22 ms
7.7 crypto_pyaes: loops=16, warmups=46, values=45
Overall:
40 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 9 values (average of runs, skip 46, limit to 45):
7.7. crypto_pyaes: loops=16, warmups=46, values=45 41 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=16, warmups=46, values=45: 9.81 ms +- 0.24 ms • LIMIT: loops=16, warmups=46: 9.81 ms +- 0.24 ms
7.8 deltablue: loops=256, warmups=14, values=55
Overall:
42 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 11 values (average of runs, skip 14, limit to 55):
7.8. deltablue: loops=256, warmups=14, values=55 43 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=256, warmups=14, values=55: 441 us +- 20 us • LIMIT: loops=256, warmups=14: 440 us +- 19 us
7.9 django_template: loops=4, warmups=12, values=36
Overall:
44 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 7.3 values (average of runs, skip 12, limit to 36):
7.9. django_template: loops=4, warmups=12, values=36 45 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=4, warmups=12, values=36: 36.9 ms +- 1.6 ms • loops=4, warmups=12, values=73: 36.9 ms +- 1.6 ms • LIMIT: loops=4, warmups=12: 36.9 ms +- 1.6 ms
7.10 dulwich_log: loops=2, warmups=21, values=18
Overall:
46 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 3.6 values (average of runs, skip 21, limit to 18):
7.10. dulwich_log: loops=2, warmups=21, values=18 47 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=2, warmups=21, values=18: 98.6 ms +- 4.7 ms • LIMIT: loops=2, warmups=21: 96.8 ms +- 4.4 ms
7.11 fannkuch: loops=1, warmups=59, values=40
Overall:
48 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Moving average of 25 values (skip 59), very small absolute variation (see the Y scale):
7.11. fannkuch: loops=1, warmups=59, values=40 49 Victor Stinner’s Notes Documentation, Release 1.0
Long cycle of 40 values. Not easy to see using moving average, spikes depend on the width of the moving window. Mean: • loops=1, warmups=59, values=40: 171 ms +- 1 ms • LIMIT: loops=1, warmups=59: 171 ms +- 1 ms
7.12 float: loops=4, warmups=25, values=40
Use the suboptimal step 2 to reduce total benchmark runtime, even if the step 3 is a little bit faster. Overall:
50 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Step 2, after warmup (average of runs, skip 25, limit 40):
7.12. float: loops=4, warmups=25, values=40 51 Victor Stinner’s Notes Documentation, Release 1.0
Step 3: cycle of 16 values (average of runs, skip 119, limit 80):
52 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Mean: • Step 2: loops=4, warmups=25, values=40: 41.0 ms +- 0.4 ms • Step 3: loops=4, warmups=119: 39.9 ms +- 1.6 ms
7.13 genshi_text: loops=8, BUG!
BUG! 19 ms at value 0 => 92 ms at value 250, steady slowdown! Overall:
7.13. genshi_text: loops=8, BUG! 53 Victor Stinner’s Notes Documentation, Release 1.0
7.14 genshi_xml: loops=2, BUG!
BUG! 70 ms at value 0 => 200 ms at value 250, steady slowdown! Overall:
54 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
7.15 go: loops=2, warmups=87, values=80
Overall:
7.15. go: loops=2, warmups=87, values=80 55 Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 5 values (average of runs, skip 87, limit 80):
56 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Step 2 after warmup (skip 87):
7.15. go: loops=2, warmups=87, values=80 57 Victor Stinner’s Notes Documentation, Release 1.0
Contiguous optimization (moving average of 50 values, skip 87), but only minor optimization (look at the Y scale):
58 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=2, warmups=87, values=32: 87.4 ms +- 4.9 ms • loops=2, warmups=87, values=80: 87.3 ms +- 5.0 ms • LIMIT: loops=2, warmups=87: 87.2 ms +- 5.0 ms
7.16 hexiom: loops=64, warmups=36, values=50
Only compute 2 cycles instead of 5 to limit the benchmark total runtime, since the cycle of long (25 values). Overall:
7.16. hexiom: loops=64, warmups=36, values=50 59 Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 25 values (average of runs, skip 36 , limit 127):
60 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=64, warmups=36, values=50: 2.32 ms +- 0.04 ms • LIMIT: loops=64, warmups=36: 2.33 ms +- 0.05 ms
7.17 hg_startup: loops=1, warmups=4, values=10
Overall:
7.17. hg_startup: loops=1, warmups=4, values=10 61 Victor Stinner’s Notes Documentation, Release 1.0
Step 2 (skip 4, limit 10):
62 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=1, warmups=4, values=10: 243 ms +- 1 ms • LIMIT: loops=1, warmups=4: 243 ms +- 1 ms
7.18 html5lib: loops=2, warmups=50, values=50
On 250 values, it seems like PyPy optimizes the code multiple times. Tere are at least 3 steps: • Warmup: 0..56 • Step 1: 57..158 • Step 2: 159..249 Use the step 1 which is suboptimal to reduce the benchmark total runtime. Overall:
7.18. html5lib: loops=2, warmups=50, values=50 63 Victor Stinner’s Notes Documentation, Release 1.0
Moving average of 25 values:
64 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle (runs average, skip 57, limit 50):
7.18. html5lib: loops=2, warmups=50, values=50 65 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • loops=2, warmups=50, values=50: 63.8 ms +- 3.6 ms • LIMIT: loops=2, warmups=50: 63.3 ms +- 3.5 ms
7.19 json_dumps: loops=16, warmups=38, values=25
Overall:
66 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Cycle of 5 values (runs average, skip 38, limit 50):
7.19. json_dumps: loops=16, warmups=38, values=25 67 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • Sample (5 cycles): loops=16, warmups=38, values=25: 7.38 ms +- 0.15 ms • Sample (10 cycles): loops=16, warmups=38, values=50: 7.39 ms +- 0.16 ms • LIMIT: loops=16, warmups=38: 7.39 ms +- 0.16 ms
7.20 json_loads: loops=256, warmups=27, values=50
Overall, +16% spike at value 90, +10% spike at value 190:
68 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
Step 2 (skip 27, limit 50) with a pike at value 51:
7.20. json_loads: loops=256, warmups=27, values=50 69 Victor Stinner’s Notes Documentation, Release 1.0
Mean: • Sample: loops=256, warmups=27, values=50: 28.4 us +- 0.2 us • Limit: loops=256, warmups=27: 28.4 us +- 0.4 us
7.21 TODO
• logging_format: loops=2048 • logging_silent: loops=134217728 • logging_simple: loops=4096 • mako: loops=8 • meteor_contest: loops=2 • nbody: loops=4 • nqueens: loops=4 • pathlib: loops=8 • pickle: loops=64 • pickle_dict: loops=64
70 Chapter 7. Analyze of PyPy warmup in performance benchmarks Victor Stinner’s Notes Documentation, Release 1.0
• pickle_list: loops=256 • pickle_pure_python: loops=128 • pidigits: loops=1 • pyflate: loops=1 • python_startup: loops=8 • python_startup_no_site: loops=8 • raytrace: loops=8 • regex_compile: loops=2 • regex_dna: loops=1 • regex_effbot: loops=4 • regex_v8: loops=1 • richards: loops=64 • scimark_fft: loops=16 • scimark_lu: loops=64 • scimark_monte_carlo: loops=32 • scimark_sor: loops=128 • scimark_sparse_mat_mult: loops=1024 • spambayes: loops=4 • spectral_norm: loops=16 • sqlalchemy_declarative: loops=1 • sqlalchemy_imperative: loops=8 • sqlite_synth: loops=32768 • sympy_expand: loops=1 • sympy_integrate: loops=4 • sympy_sum: loops=2 • sympy_str: loops=1 • telco: loops=8 • tornado_http: loops=2 • unpack_sequence: loops=2048 • unpickle: loops=128 • unpickle_list: loops=1024 • unpickle_pure_python: loops=128 • xml_etree_parse: loops=1 • xml_etree_iterparse: loops=1 • xml_etree_generate: loops=2 • xml_etree_process: loops=2
7.21. TODO 71 Victor Stinner’s Notes Documentation, Release 1.0
72 Chapter 7. Analyze of PyPy warmup in performance benchmarks CHAPTER 8
C programming language
See also: • GDB • Visual Studio
8.1 i++ and –i
• x = ++i; is equivalent of ++i; x = i; or i += 1; x = i; • x = i++; is equivalent of x = i; i++; or x = i; i += 1; • x = --i; is equivalent of --i; x = i; or i -= 1; x = i; • x = i--; is equivalent of x = i; i--; or x = i; i -= 1;
8.2 Lack of namespaces
• Common prefix for identifiers to prevent conflicts: – glib: g_ – Gtk: gtk_ – Python: Py_, _Py_ (“private”) or PY_ (define) • static keyword – function only accessible from the current file, not exported – variable only accessible from the current file, not exported – variable local to a function • #include: C preprocessor
73 Victor Stinner’s Notes Documentation, Release 1.0
Python has the make smelly tool to detect exported symbols which doesn’t start with Py_ nor _Py_.
8.3 Issues with types
• comparison between signed and unsigned: who wins? cast signed to unsigned, or cast unsigned to signed? => obvious integer overflow • integer overflow: undefined behaviour, the compiler is free to optimize. Check if an operation will overflow is hard, especially for signed numbers. GCC added builtin functions to check that in a portable and efficient way. • portability: int==long==void*, no warning. if int and long have the same size, downcasting a long into an int is allowed, and don’t emit any warning. • int stored in void* • void* vs intptr_t vs uintptr_t • Use “int” to access an item of an array in a loop: the compiler may have to handle integer overflow. size_t or ssize_t preferred here?
8.4 Portability
• Perl still uses C89 • Python only started to move to C99 with Python 3.6, only CPython is restricted to a subset of C99, the most portable parts of C99. . . • GNU extensions of GCC • glibc vs all other C libraries (ex: musl libc and uclibc). GNU extensions, again. Implementation bugs. Errno is sometimes set to 0 on success, sometimes it is not set on failure, it depends on the called function and the libc. •
8.5 Issues with C++
• C code used in C++ has to be careful with exceptions: need to compile C with -fexceptions? • extern { trickery for header files
8.6 Undefined Behaviour
• memcpy(dst, src, 0) is safe (if dst is a valid address): https://stackoverflow.com/questions/29844298/ is-it-legal-to-call-memcpy-with-zero-length-on-a-pointer-just-past-the-end-of-an • What Every C Programmer Should Know About Undefined Behavior #1/3 • What Every C Programmer Should Know About Undefined Behavior #2/3 • What Every C Programmer Should Know About Undefined Behavior #3/3
74 Chapter 8. C programming language Victor Stinner’s Notes Documentation, Release 1.0
• GCC Undefined Behavior Sanitizer, ubsan: -fsanitize=undefined • Clang UndefinedBehaviorSanitizer
8.7 Strict Aliasing
• Understanding Strict Aliasing (Mike Acton, June 1, 2006) • Demystifying The Restrict Keyword (Mike Acton, May 29, 2006) • Type punning isn’t funny: Using pointers to recast in C is bad. (April, 2008) by Matt Gallagher Magic UNION_CAST() macro:
#define UNION_CAST(x, destType) \ (((union {__typeof__(x) a; destType b;})x).b)
8.8 C aliasing
• -fstrict-aliasing vs -fno-strict-aliasing • __restrict__ (or just restrict) • clang “bug”: aliasing bug with union in clang 4.0 • Understanding Strict Aliasing (Mike Acton, June 1, 2006) • Demystifying The Restrict Keyword (Mike Acton, May 29, 2006) • Type-punning and strict-aliasing (Thiago Macieira, June 2011) • Python 2 slower: the C code base doesn’t respect strict aliasing and so must be compiled with -fno-strict-aliasing (to avoid bugs when the compiler optimizes the code) which is inefficient. The structure of Python C type has been deeply rewritten to fix the root cause. • gcc-help: Missed optimization when using a structure (2013-04) • GCC -fstrict-aliasing documentation • GCC Union documentation • Detecting Strict Aliasing Violations by P. Cuoq et. al. Change which fixed a crash after the merged of the new dict implementation on a specific platform (don’t recall which one!): https://github.com/python/cpython/commit/186122ead26f3ae4c2bc9f6715d2a29d339fdc5a Example:
#include
sp[1]= hi; (continues on next page)
8.7. Strict Aliasing 75 Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) sp[0]= lo;
return (arg); } int main(void) { uint32_t x= 0xabcd1234; uint32_t y= swap_words(x); printf("x=%lx\n", (long unsigned int)x); printf("y=%lx\n", (long unsigned int)y); return 0; }
Bug:
$ LANG= gcc -O3 x.c -o x -fstrict-aliasing -Wstrict-aliasing=2 && ./x x.c: In function 'swap_words': x.c:7:3: warning: dereferencing type-punned pointer will break strict-aliasing rules
˓→[-Wstrict-aliasing] uint16_t* const volatile sp = (uint16_t*)&arg; ^~~~~~~~ x=abcd1234 y=abcd1234
8.9 volatile volatile is discouraged in the Linux kernel in favor of smaller locks: https://github.com/torvalds/linux/blob/master/ Documentation/process/volatile-considered-harmful.rst
8.10 GCC warnings
• -Wall: some warnings • -Wall -Wextra: more warnings • -Wall -Wextra -O3: even more warnings. Some warnings are only emitted when the compiler optimizes the code, like dead code or unused variables. • There are even more. GCC is able to emit even more warnings, but they must be enabled explictly! – -fstrict-aliasing -Wstrict-aliasing=2
8.11 Platforms #define
• AIX: #ifdef _AIX • FreeBSD: #ifdef __FreeBSD__ • HP-UX: #ifdef __hpux • Linux: #ifdef __linux__
76 Chapter 8. C programming language Victor Stinner’s Notes Documentation, Release 1.0
• NetBSD: #ifdef __NetBSD__ • Solaris: #ifdef sun • Windows: _WIN32 or _WIN64 • macOS: #ifdef __APPLE__
8.12 Compiler defines
• GCC: #if defined(__GNUC__) && ((__GNUC__ > 4) || ((__GNUC__ == 4) && (__GNUC_MINOR__ > 5))) • Clang: #ifdef __clang__ • Visual Studio: #if defined(_MSC_VER) && _MSC_VER >= 1800
8.13 GCC flags
https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/
8.14 Compile in 32-bit mode on Fedora
• dnf install glibc-devel.i686 • gcc -m32 Example:
$ echo 'int main() { return sizeof(void *); }' > x.c $ gcc x.c -o x -m32 && ./x; echo $? 4
Configure in 32-bit:
./configure CFLAGS="-m32" LDFLAGS="-m32"&& make
For Python, install also libffi, openssl and zlib:
dnf install-y libffi-devel.i686 openssl-devel.i686 zlib-devel.i686
8.15 Compiler and linker options
• https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/ • https://wiki.debian.org/Hardening
8.12. Compiler defines 77 Victor Stinner’s Notes Documentation, Release 1.0
8.16 C macros (preprocessor)
• typeof(expr): C99 • offsetof(type, member):
#define BUILD_ASSERT_EXPR(cond) \ (sizeof(char [1 - 2*!(cond)]) - 1)
Magic ARRAY_LENGTH() macro by Rusty Russell, compilation error with GCC if the argument is not an array but a pointer:
#if (defined(__GNUC__) && !defined(__STRICT_ANSI__) && \ (((__GNUC__ == 3) && (__GNUC_MINOR__ >= 1)) || (__GNUC__ >= 4))) /* Two gcc extensions. &a[0] degrades to a pointer: a different type from an array */ #define ARRAY_LENGTH(array) \ (sizeof(array) / sizeof((array)[0]) \ + BUILD_ASSERT_EXPR(!__builtin_types_compatible_p(typeof(array), \ typeof(&(array)[0])))) #else #define ARRAY_LENGTH(array) \ (sizeof(array) / sizeof((array)[0])) #endif
Is a type signed or unsigned?
#define IS_TYPE_UNSIGNED(type) (((type)0 - 1) > 0)
8.16.1 Libc
• __GLIBC__: GNU libc
8.16.2 Convert to a string
STRINGIFY(expr) macro:
#define _XSTRINGIFY(x) #x
/* Convert the argument to a string. For example, STRINGIFY(123) is replaced with "123" by the preprocessor. Defines are also replaced by their value. For example STRINGIFY(__LINE__) is replaced by the line number, not by"__LINE__". */ #define STRINGIFY(x) _XSTRINGIFY(x)
#define __CONCAT(x,y) x ## y #define __STRING(x) #x
But __CONCAT and __STRING are not portable. For example, NetBSD says “only works with ANSI C”. Comment on Linux: “For these things, GCC behaves the ANSI way normally, and the non-ANSI way under -traditional.”
78 Chapter 8. C programming language Victor Stinner’s Notes Documentation, Release 1.0
8.17 const
First read: Why const Doesn’t Make C Code Faster (August 2019) by Simon Arneaud. General: • const int *x is the same than int const *x: it only matters if const is before or after * Single *, constant x, but *x is mutable:
int * const x=(int * const)1; x=(int * const)2;/ * compilation error */ *x=3;/ * ok */
Single *, constant *x, but x is mutable:
const int *x= (const int *)1; x= (const int *)2;/ * ok */ *x=3;/ * compilation error */
Single *, constant x and constant *x:
const int * const x= (const int * const)1; x= (const int * const)2;/ * compilation error */ *x=3;/ * compilation error */
Problem of casting char ** to const char **: http://c-faq.com/ansi/constmismatch.html GCC: use -Wcast-qual option.
8.18 Atomic variables
• C11
8.17. const 79 Victor Stinner’s Notes Documentation, Release 1.0
• __ATOMIC_RELAXED: Implies no inter-thread ordering constraints. See also GCC: Legacy __sync Built-in Functions for Atomic Memory Access like __sync_fetch_and_add(&var, 1).
8.19 C FAQ http://c-faq.com/
8.20 Thread Local Storage (TLS)
• GCC and clang extension: __thread. Example: __thread int i;. See GCC Thread Local documenta- tion. GCC and clang use the FS register on x86-64. • pthread – pthread_getspecific(), pthread_setspecific() – pthread_key_create(), pthread_key_delete()
80 Chapter 8. C programming language CHAPTER 9
Unicode
Read my free ebook: Programming with Unicode!
9.1 Encodings
• Windows – OEM code page: used by stdin, stdou and stderr in the Windows console – ANSI code page: used by all other Windows “ANSI” functions. Some examples: filenames, command line arguments, environment variables, etc. • UNIX: “Locale encoding” – LC_CTYPE locale – used for filenames, command line arguments, environment variables, the console (stdin, stdout, stderr) • Common encodings: – UTF-8 – ISO 8859-1 aka Latin1 or Windows code page 1252 – ASCII
81 Victor Stinner’s Notes Documentation, Release 1.0
9.2 Python
See my conference (in french) “Comprendre les erreurs Unicode” (Pycon FR 2009 at Paris): slides (PDF) and video.
9.2.1 Narrow and wide builds, PEP 393
Python 3.3 introduced the Flexible String Representation (PEP 393) and supports the whole Unicode range (U+0000 - U+10ffff) on all platforms. Older Python versions had a “narrow or wide” compilation option: • UNIX and Mac OS X uses wide mode: the unicode type uses 32-bit code points. In Unicode, it is called the UCS-4 encoding. • Windows uses narrow mode: the unicode type uses 16-bit code points, non-BMP characters (unicode range U+10000 - U+10ffff) are used as a surrogate pair (two 16-bit code points). In Unicode, it is called the UTF-16 encoding. This mode is preferred on Windows because Windows kernel uses also the UTF-16 encod- ing internally. Use sys.maxunicode == 0xffff to check if Python is compiled in narrow mode. Otherwise, sys. maxunicode is equal to 0x10ffff.
9.2.2 Python 2
• str type and "abc" are strings of bytes, unicode type is a string of characters • “Default encoding” – sys.getdefaultencoding() – used by unicode.encode() and str.decode() when no encoding is specified – ASCII by default, must not by modified (sys.setdefaultencoding()) • File system encoding – sys.getfilesystemencoding() – used to encode filenames and environment variables – used on UNIX by os.listdir(unicode) to decode filenames – ANSI code page (mbcs) on Windows, utf-8 on Mac OS X, the locale encoding on UNIX • Locale encoding – locale.getpreferredencoding() – used by default by io.TextIOWrapper – ANSI code page on Windows, LC_CTYPE locale on UNIX • OEM code page (Windows only) – sys.stdin.encoding, sys.stdout.encoding and sys.stderr.encoding
82 Chapter 9. Unicode Victor Stinner’s Notes Documentation, Release 1.0
9.2.3 Python 3
• bytes type is a string of bytes, str type and "abc" are strings of characters • UTF-8 – used for the default encoding of the source code • “Locale encoding” – locale.getpreferredencoding() – ANSI code page on Windows, LC_CTYPE locale on UNIX – used by sys.stdin, sys.stdout, sys.stderr, and by default by open() (and io.TextIOWrapper) • “File system encoding” – sys.getfilesystemencoding() – ANSI code page (mbcs) on Windows, utf-8 on Mac OS X, the locale encoding on UNIX – used for filenames, command line arguments, environment variables • “Default encoding” – sys.getdefaultencoding(), hardcoded to utf-8 – used by bytes.decode() and str.encode() when no encoding is specified • OEM code page (Windows only) – sys.stdin.encoding, sys.stdout.encoding and sys.stderr.encoding
9.2.4 Issues
• GB2312 codec is using a wrong covert table: WONTFIX, It’s a bug, but one which is present in a lot of other systems as well, so we’d potentially make it impossible to write GB2312 data which is supposed to be read back by these other systems.
9.3 Test non-ASCII characters with locales
It seems like FreeBSD 11 doesn’t support all encodings: only Latin1 and UTF-8 seem to be implemented. At least, KOI8-R, Big5 and CP1131 are not implemented properly. Windows locales: “fr-FR”, “en-US”, “ja-JP”, etc. On Windows, before setlocale(LC_CTYPE, “”) is called, LC_CTYPE uses the Latin1 encoding in practice (see Python issue #29571). Call setlocale(LC_CTYPE, “”) to use the ANSI code page. My tools: • test_all_locales.py: test Python implementation of locales. Only support a few operating systems. • all_locales.py: script to list all working locales, can be different than “locale -a” • c_locale.c: basic info on the “C” locale
9.3. Test non-ASCII characters with locales 83 Victor Stinner’s Notes Documentation, Release 1.0
9.3.1 Use cases
• Latin1 or UTF-8 encoding (locale different than C and POSIX) • C or POSIX locale: ASCII encoding on Linux, Latin1 encoding on FreeBSD/Solaris (but ASCII announced by nl_langinfo(CODESET)) • LC_NUMERIC != LC_CTYPE: the fun localeconv() bug, https://bugs.python.org/issue31900 • python 3.7 -X utf8 • macOS and Android UTF-8
9.3.2 Locale, announced encoding, effective encoding
Inconsistent:
Operating system Locale Announced encoding Effective encoding FreeBSD C, POSIX US-ASCII ISO-8859-1 FreeBSD zh_TW.Big5 Big5 ? (not Big5) macOS C, POSIX US-ASCII ISO-8859-1 macOS zh_TW.Big5 Big5 ? (not Big5)
Consistent, announced encoding = effective encoding:
Operating system Locale Encoding Fedora 27 C, POSIX ASCII FreeBSD fr_FR.UTF-8 UTF-8 macOS fr_FR.UTF-8 UTF-8 Fedora 27 fr_FR.UTF-8 UTF-8 Fedora 27 zh_TW.Big5 Big5
Tested operating systems: • macOS 10.13.2: • FreeBSD 11.1 • Fedora 27 (glibc 2.26)
9.3.3 localeconv()
Fedora 27:
LC_ALL Encod- Field Bytes Text locale ing es_MX.utf8 UTF-8 thousands_sep 0xE2 0x80 0x89 U+2009 fr_FR.UTF-8 UTF-8 cur- 0xE2 0x82 0xAC U+20AC (C) rency_symbol ps_AF.utf8 UTF-8 thousands_sep 0xD9 0xAC U+066C () uk_UA.koi8u KOI8-U cur- 0xC7 0xD2 0xCE U+0433 U+0440 U+043d U+002E rency_symbol 0x2E (.) uk_UA.koi8u KOI8-U thousands_sep 0x9A U+00A0
84 Chapter 9. Unicode Victor Stinner’s Notes Documentation, Release 1.0 macOS 10.13.2:
LC_ALL locale Encoding Field Bytes Text ru_RU.ISO8859- ISO8859- cur- b'\xe0\xe3\xd1. U+0440 U+0443 U+0431 U+002e 5 5 rency_symbol ' (.)
FreeBSD 11:
LC_ALL En- Field Bytes Text locale cod- ing ar_SA.UTF- UTF-8 deci- b'\xd9\xab' U+066b (’’) 8 mal_point ar_SA.UTF- UTF-8 thou- b'\xd9\xac' U+066c (’’) 8 sands_sep ar_SA.UTF- UTF-8 cur- b'\xd8\xb1.\xd8\xb3. U+0631 U+002e U+0633 U+002e 8 rency_symbol \xe2\x80\x8f' U+200f (‘..u200f’) zh_TW.Big5 Big5 cur- b'\xa2\xdc\xa2\xe2\xa2\x43'u'\uff2e\uff34\uff04' () rency_symbol zh_TW.Big5 Big5 deci- b'\xa1\x44' u'\uff0e' () mal_point zh_TW.Big5 Big5 thou- b'\xa1\x41' u'\uff0c' () sands_sep
Note: On FreeBSD with LC_CTYPE=”zh_TW.Big5”, mbstowcs() doesn’t use Big5 but a different encoding and so returns mojibake. Windows 7.1:
LC_ALL locale Encoding Field Bytes Text fr-FR cp1252 currency_symbol b'\x80' U+20AC fr-FR cp1252 thousands_sep b'\xA0' U+00A0
9.3.4 strftime(), tzname
Fedora 27:
LC_ALL locale Encoding Month %b Bytes Text fr_FR Latin1 December b'd\xe9c.' 'd\xe9c.' (déc.)
Windows 8.1:
LC_ALL locale Encoding Date, format Bytes Text fr-FR cp1252 December, %b b'd\xe9c.' 'd\xe9c.' (déc.) ja-JP cp932? Monday, %a N/A '\u6708'
Python2:
9.3. Test non-ASCII characters with locales 85 Victor Stinner’s Notes Documentation, Release 1.0
vstinner@apu$ python2 >>> import time, locale >>> locale.setlocale(locale.LC_ALL, "fr_FR") 'fr_FR' >>> time.strftime("%A, %d %B %Y", time.localtime(time.mktime((2018, 2, 1, 12, 0, 0, 0,
˓→ 0, 0)))) 'jeudi, 01 f\xe9vrier 2018'
• non-ASCII tzname on Windows: “’ ()’ means ‘Tokyo (Standard Time)’ in Japanese.” • https://bugs.python.org/issue5905 • https://bugs.python.org/issue13560 • https://bugs.python.org/issue16322 • Commit af02e1c8: Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() “Fix time.strftime() (if wcsftime() is missing): decode strftime() result from the current locale encoding, not from the filesystem encoding.” • Commit 720f34a3: Issue #5905: “time.strftime() is now using the locale encoding, instead of UTF-8, if the wcsftime() function is not available.”
9.3.5 strerror()
LC_ALL lo- Encod- Bytes Text cale ing fr_FR.ISO8859-ISO- b'Fichier ou r\xe9pertoire 'Fichier ou r\xe9pertoire 1 8859-1 inexistant' inexistant'
Links: • non-ASCII strerror: “os.strerror(23) = ‘Trop de fichiers ouverts dans le syst\xe8me’.” • https://bugs.python.org/issue13560 • Commit 1f33f2b0: “Issue #13560: os.strerror() now uses the current locale encoding instead of UTF-8”
9.4 Political and regional differences
Unicode provides a single standard and so cannot have special cases depending on country or recent political changes. Examples: • 2018: lower() on Turkish letter “I”˙ returns a 2-chars-long string • 2017: Germany made the upper case ß official. ‘ß’.upper() should now return .
86 Chapter 9. Unicode CHAPTER 10
Multithreading Programming
Writing code with threads is hard. Race conditions are likely and common in multithreaded applications and usually very hard to reproduce.
10.1 Threads and Signals
Threads are bad. Threads with signals are worse :-) Before OpenBSD 5.2, threads were implemented in user space on OpenBSD. Sending signals was not reliable at all. Before FreeBSD 8, threads+signals had a lot of issues too.
10.2 Spawn Child Processes
Many tests in test_threading are skipped on FreeBSD <= 6, HP UX11, NetBSD5: Between fork() and exec(), only async-safe functions are allowed (issues #12316 and #11870), and fork() from a worker thread is known to trigger problems with some operating systems (issue #3863): skip problematic tests on platforms known to behave badly.
10.2.1 Fork, Exec and File Descriptors
I wrote PEP 446 - Make newly created file descriptors non-inheritable which breaks backward compatibility in Python 3.4 “for the good of mankind” (wrote Guido van Rossum). This PEP makes all file descriptors non-inheritable to fix a race condition: • two threads create a child process • thread A creates a pipe • thread B creates a child process 1
87 Victor Stinner’s Notes Documentation, Release 1.0
• thread A makes the pipe non-inheritable • thread A creates a child process 2 Without the PEP, the child process 1 created by the thread B inherit the file descriptor created by the thread A. In fact, the race condition is only avoided if the thread A is able to make the file descriptor non-inheritable atomically, using O_CLOEXEC or SOCK_CLOEXEC flag for example.
10.2.2 Python subprocess Module
In Python 2, the subprocess module is not thread-safe: unexpected file descriptors can be inherited, and the subprocess has other sily issues.
10.3 Process-wide
Multithreading is hard because many functions of system C library (libc): • modify a state for the whole process: change process wide • is not reentrant • rely on a global state • is not “async signal safe” On Unix and other platforms, many variables are “process-wide”, in opposition of “per thread”. Examples of process-wide states: • Current working directory aka cwd – Modified by chdir(), read by getcwd() – Most “legacy” filesystem functions taking a filename rely on the “current working directory” (cwd), espe- cially using relative path. Exampes: open() or chmod(). – The new Linux “at” functions don’t rely on the current working directory. Examples: openat() or chmodat(). • Locales – Modified by setlocale() – For example, used by strftime() and localeconv(). – Functions using “wide character strings” (wcs) avoid some issues. Example: wcsftime(). – Some libraries don’t rely on a global locale but expect a locale argument – Recent glibc has a new API to get per-thread locale: XXX • Unix signals – Example of signals: SIGINT, SIGSEGV – Signal handlers are registered by signal() and sigaction() – raise() or kill() to send a signal to a process – Per thread API: pthread_kill() (send a signal to a thread), pthread_sigmask() (block signals) • Clock: clock_gettime(CLOCK_PROCESS_CPUTIME_ID), but there is: CLOCK_THREAD_CPUTIME_ID
88 Chapter 10. Multithreading Programming Victor Stinner’s Notes Documentation, Release 1.0
• umask: File mode creation mask – Get and set by umask(). Others: • File descriptors: not really an issue in practice if a FD is only used in a single thread. • Heap memory, malloc()/free(): modern malloc() implementations scales on threads/CPUs. Not an issue if a memory block is only used in a single thread. • User and groups See also Ghosts of Unix Past: a historical search for design patterns by Neil Brown (October, 2010).
10.3. Process-wide 89 Victor Stinner’s Notes Documentation, Release 1.0
90 Chapter 10. Multithreading Programming CHAPTER 11
Benchmarks
11.1 My articles
• My journey to stable benchmark, part 1 (system) (May 21, 2016) • My journey to stable benchmark, part 2 (deadcode) (May 22, 2016) • My journey to stable benchmark, part 3 (average) (May 23, 2016) • Visualize the system noise using perf and CPU isolation (June 16, 2016) • Intel CPUs: P-state, C-state, Turbo Boost, CPU frequency, etc. (July 15, 2016) • Intel CPUs (part 2): Turbo Boost, temperature, frequency and Pstate C0 bug (September 23, 2016) • Analysis of a Python performance issue (November 19, 2016)
11.2 Factors impacting benchmarks
Factors impacting Python benchmarks: • Linux Address Space Layout Randomization (ASRL), /proc/sys/kernel/randomize_va_space: – 0: No randomization – 1: Conservative randomization – 2: Full randomization • Python random hash function: PYTHONHASHSEED • Command line arguments and environmnet variables: enabling ASLR helps here (?) • CPU power saving and performance features: disable Intel Turbo Boost and/or use a fixed CPU frequency. • Temperature: temperature has a limited impact on benchmarks. If the CPU is below 95°C, Intel CPUs still run at full speed. With a correct cooling system, temperature is not an issue.
91 Victor Stinner’s Notes Documentation, Release 1.0
• Linux perf probes: /proc/sys/kernel/perf_event_max_sample_rate • Code locality, CPU L1 instruction cache (L1c): Profiled Guided Optimization (PGO) helps here • Other processes and the kernel, CPU isolation (CPU pinning) helps here: use isolcpus=cpu_list and rcu_nocbs=cpu_list on the Linux kernel command line • . . . Reboot? Sadly, other unknown factors may still impact benchmarks. Sometimes, it helps to reboot to restore standard performances. Commands to check these factors: • python3 -m perf system show to show the system state • python3 -m perf system tune tunes the system to run benchmarks
11.3 Links
• Biased Benchmarks (honesty is hard) • How to get consistent results when benchmarking on Linux? (Aug 2019) by Denis Bakhvalov • Numbers Everyone Should Know (2009) • How long does it take to make a context switch? (2010) • Rethinking optimization for size (LWN, 2013) • Intel Languages Performance
11.4 Python resources
• [email protected] mailing list. My threads: – External sources of noise changing call_simple “performance” – CPU speed of one core changes for unknown reason – When CPython performance depends on dead code. . . – Disable hash randomization to get reliable benchmarks – Linux tip: use isolcpus to have (more) reliable benchmark – Tool to run Python microbenchmarks
11.5 Random performance of modern Intel CPU https://github.com/cyring/corefreq
11.5.1 Intel CPUs https://en.wikipedia.org/wiki/List_of_Intel_CPU_microarchitectures • 2015: Skylake (SKL) • 2013: Haswell (HSW)
92 Chapter 11. Benchmarks Victor Stinner’s Notes Documentation, Release 1.0
• 2013: Silvermont • 2011: Sandy Bridge (SNB) • 2008: Bonnel – 2012: Ivy Bridge (IVB) • 2008: Nehalem (NHM) – 2010: Gulftown or Westmere-EP • 2006: Intel Core
11.5.2 CPU Pipeline
Modern Intel CPUs don’t execute CISC machine instructions (complex instructions) but decode them into RISC in- structions (simple instructions). The RISC instructions are also reordered to reduce the latency of memory load and store instructions.
11.5.3 Branch prediction
The CPU pipeline must decode and reorder instructions faster than the units executing instructions. To keep the CPU pipeline full, the CPU predicts branches (“if/else”). If its prediction is wrong, the CPU pipeline must be flushed and it has a cost on performance.
11.5.4 Linux perf
The Linux perf program gives access to low-level events: • stalled-cycles-frontend: “The cycles stalled in the front-end are a waste because that means that the CPU does not feed the Back End with instructions. This can mean that you have misses in the Instruction cache, or complex instructions that are not already decoded in the micro-op cache.” • stalled-cycles-backend: “The cycles stalled in the back-end are a waste because the CPU has to wait for resources (usually memory) or to finish long latency instructions (e.g. transcedentals - sqrt, logs, etc.).” “Another stall reason is branch prediction miss. That is called bad speculation. In that case uops are issued but they are discarded because the BP predicted wrong.” Source: https://stackoverflow.com/questions/22165299/what-are-stalled-cycles-frontend-and-stalled-cycles-backend-in-perf-stat-resul
11.5.5 Memory caches, L1, L2, L3, L4, MMU, TLB
Memory accesses are between slow and very slow compared to the speed of the CPU. To be efficient, there are multiple levels of caches: L1 (fastest, on the CPU die), L2, L3, and sometimes even L4 (slowest, but also the largest). Applications don’t handle directly physical addresses of the memory but use “virtual” addresses. The MMU (Memory management unit) is responsible to convert virtual addresses to physical addresses. When the Linux kernel switches to a different application, the TLB (Translation lookaside buffer) cache of the MMU must be flushed.
11.5. Random performance of modern Intel CPU 93 Victor Stinner’s Notes Documentation, Release 1.0
11.6 Micro optimisation
• Linux kernel: The problem with prefetch: “So the conclusion is: prefetches are absolutely toxic, even if the NULL ones are excluded.” • Linux kernel likely() / unlikely() based on GCC __builtin_expect() • How new-lines affect the Linux kernel performance by Nadav Amit
11.7 Memory
• What Every Programmer Should Know About Memory – HTML version (first article which ends with links to the following articles) – PDF version
11.8 Help compiler to optimize
• const keyword? • aliasing: -fno-strict-aliasing or __restrict__
11.9 Linux perf
Basic: perf stat command
Record: perf record-o trace.data-g command # -g to record call graph: you may recompile your code with -fno-omit-frame-pointer
Report: perf report-i trace.data
Links: • https://perf.wiki.kernel.org • http://www.brendangregg.com/perf.html • http://stackoverflow.com/questions/12601474/what-are-perf-cache-events-meaning • http://web.eece.maine.edu/~vweaver/projects/perf_events/perf_event_open.html
94 Chapter 11. Benchmarks Victor Stinner’s Notes Documentation, Release 1.0
11.10 Valgrind: Callgrind and Cachegrind
11.10.1 Callgrind
Command:
PYTHONHASHSEED=0 taskset-c7 valgrind--dsymutil=yes--tool=callgrind--callgrind-
˓→out-file=callgrind.out.slow2.25--dump-instr=yes--collect-jumps=yes./slow../
˓→benchmarks/performance/bm_call_simple.py-n 50--timer perf_counter
• Record at instruction level (not function level) • Record conditional jumps Open with Kcachegrind: kcachegrind callgrind.out.slow.25.
Or: callgrind_annotate callgrind.out.slow.25
See also: Callgrind documentation.
11.10.2 Cachegrind
Record traces:
PYTHONHASHSEED=0 time taskset-c2 valgrind--dsymutil=yes--tool=cachegrind--
˓→cachegrind-out-file=cachegrind.out.fast.25./fast../benchmarks/performance/bm_call_
˓→simple.py-n 25--timer perf_counter
See also: Cachegrind documentation.
11.10. Valgrind: Callgrind and Cachegrind 95 Victor Stinner’s Notes Documentation, Release 1.0
96 Chapter 11. Benchmarks CHAPTER 12
FreeBSD
12.1 Minimum packages to develop on FreeBSD
Install: sudo pkg install vim-console git
12.2 Misc
• Coredump filename: sudo sysctl -w 'kern.corefile =%N.%P.core'
12.3 Upgrade
Upgrade the system: sudo freebsd-update fetch sudo freebsd-update install sudo pkg upgrade-y
12.4 Configuration
• Rerun the installer configuration, run:: bsdconfig. • Change the keyboard layout: run kbdmap.
97 Victor Stinner’s Notes Documentation, Release 1.0
12.5 Upgrade to newer FreeBSD
Upgrade to FreeBSD 12.0-RC2: sudo freebsd-update fetch sudo freebsd-update upgrade-r 12.0 sudo freebsd-update install sudo reboot # after reboot sudo freebsd-update install
12.6 Repair pkg
Repair pkg, if needed: sudo pkg bootstrap-f
If something goes wrong, reinstall everything installed by pkg: sudo pkg-static upgrade-f
12.7 Install FreeBSD VM
12.7.1 Install VM image
• https://www.freebsd.org/where.html : Download amd64/qcow2 virtual machine image, • Uncompress the image: unxz file.qcow2.xz • Move the image to /var/lib/libvirt/images/ • Create a FreeBSD VM using this disk image
12.7.2 Classic installer
• Download ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/11.0/FreeBSD-11. 0-RELEASE-amd64-disc1.iso.xz • Uncompress: unxz FreeBSD-11.0-RELEASE-amd64-disc1.iso.xz • Create a new VM: – Name: FreeBSD – Boot from an ISO: specify the path to the .iso file – System: select Show all, select UNIX, pick FreeBSD 11 – 1 cpu, 1 GB of RAM – Disk size: 20 GB – Select network: shared interface, br0 • FreeBSD installer:
98 Chapter 12. FreeBSD Victor Stinner’s Notes Documentation, Release 1.0
–
12.7.3 Enlarge qcow2 image
• Shutdown the VM • On the host: sudo qemu-img resize /var/lib/libvirt/images/freebsd.qcow2 40G • Boot the VM, in FreeBSD: sudo /etc/rc.d/growfs onestart
12.7.4 Configure as root
• Log as root • kbdcontrol -l fr.iso • pkg install sudo bash tmux vim-console • visudo: uncomment %whell ALL.. without password • adduser: add user, add it to the wheel group • to add wheel group to an user: pw usermod -n vstinner -G wheel • Enable the SSH server: • Add sshd_enable=”YES” to /etc/rc.conf • service sshd start • https://www.freebsd.org/doc/handbook/openssh.html • Log out
12.7. Install FreeBSD VM 99 Victor Stinner’s Notes Documentation, Release 1.0
12.7.5 Configure as your user
• Log in as the your user • chsh -s /usr/local/bin/bash • Log out and log in again to get bash • sudo pkg install git
100 Chapter 12. FreeBSD CHAPTER 13
Survivor Guide to Develop on Windows
Guide written for Linux developers.
13.1 Useful tools
• ConEmu • Firefox • 7-zip: archive manager (.zip, .tar.gz, .7z, .rar, etc.) • Gvim: text editor gvim (Gtk+ version for Windows) • Python • MSYS: terminal using bash for Windows • WinSCP: secury file copy on the network (SSH protocl) • tortoisehg
13.2 Windows console
• Kill a blocked command (harder than CTRL+c): CTRL + Scroll Lock key. (send a SIGBREAK signal) Note: On my Lenovo T430 laptop, I have to use the “Fn” key: • Fn + B: Break • Fn + P: Pause • Fn + S: SysRq Alternative terminals for Windows: • ConEmu
101 Victor Stinner’s Notes Documentation, Release 1.0
• mintty
13.3 cmd.exe (Windows “shell”, Windows console, the MS-DOS black window)
• Change prompt: set PROMPT=$P$G • Redirect stdout and stderr into the file outlog.log: command >output.log 2>&1
Windows command UNIX command Comment set env Display all environment variables type file.exe cat file.txt Display the content of file.txt echo %PATH% echo $PATH Display the value of the PATH environment variable RMDIR /S /Q dir rm -rf dir Remove a directory and its content cmd > log cmd > log Redirect command stdout into a new log file cmd >log 2>&1 cmd >log 2>&1 Redirect command stdout and stderr into a new log file cmd >NUL cmd >/dev/null Ignore command stdout (redirect it to null) echo %errorlevel% echo $? Display the exit code of the previous command export PS1='$ ' Change the command line prompt to ‘‘$ ‘‘ ‘‘set PROMPT=$$ ‘‘ dir NAME /s /p find -name NAME Find a file by its name in subdirectories shutdown /p /f sudo poweroff Turn off the computer findstr PATTERN grep PATTERN Search PATTERN in files with name ending with .c *.c *.c cd pwd Current directory
Get usage: findstr /? grep in subdirectories: findstr/S PATTERN *.c
To emulate top on Windows, run powershell.exe and type: while (1) { ps| sort-desc cpu| select-first 15; sleep-seconds2; cls}
Press CTRL+c to stop it.
13.4 Configure vim on Windows
• Right click on gvim: Run as administrator • Open /program files (x86)/vim/_vimrc • Comment the lines source $VIMRUNTIME/mswin.vim and behave mswin • Add custom config
102 Chapter 13. Survivor Guide to Develop on Windows Victor Stinner’s Notes Documentation, Release 1.0
13.5 Mount Windows directory on Linux
Command to mount the Widows “test” directory locally to ~/mnt, local files will be owned by the user haypo:haypo: sudo mount.cifs'//192.168.0.14/test'~/mnt-o'user=USERNAME,pass=PASSWORD,uid=haypo,
˓→gid=haypo'
13.6 Visual Studio
Flavors: • Express • Professional: enough to build Python • Ultimate Versions:
Visual Studio MSVC++ _MSC_VER 2017 14.1 1910 2015 14.0 1900 2013 12.0 1800 2012 11.0 1700 2010 10.0 1600 2008 9.0 1500 2005 8.0 1400 2003 7.1 1310 — 7.0 1300 — 6.0 1200 — 5.0 1100
Configure a shell to use the VS C compiler in 64-bit mode:
"%VS140COMNTOOLS%\..\..\VC"\vcvarsall.bat amd64
Argument: • x86: compile in 32-bit mode • amd64: compile in 64-bit mode • x86_amd64: cross-compile to 64-bit mode on a 32-bit system
13.7 Configuration
13.7.1 Git configuration file
Filename: C:\Users\haypo\.gitconfig. Run cmd.exe as administrator to be allowed to create symbolic links.
13.5. Mount Windows directory on Linux 103 Victor Stinner’s Notes Documentation, Release 1.0
13.7.2 Windows console, cmd.exe
Right click on the title, Properties: set Buffer Size of Command History to 999 (default: 50).
13.8 See also
• Operating systems
13.9 Windows variants
To develop on CPython: get a “multi-version” of Windows 10 (no N, KN or VL variant) and use a “Pro - Retail” product key. Create of a VM with 60 GB of disk (prevously I used 40 GB, it was too small). Flavors: • Family: basic feature set • Pro: more features • Entreprise: even more features Variants: • “N”: Not with Media Player; for Europe. • “KN”: specially designed for Korean market and does not include Windows Media Player (WMP) and an instant messenger. • “VL”: Volume License, a single license key can be used to activate multiple installations of Windows 10. This is usually used by large enterprises. • “S”: “Windows 10 S can only run apps from the Windows Store”. Windows 10 S is designed to run well even on lower-end laptops. Windows 10 S is focused on speed, better battery life, and higher performance.
13.10 Some Windows error codes
• 5: ERROR_ACCESS_DENIED: Access is denied. • 996: ERROR_IO_INCOMPLETE: Overlapped I/O event is not in a signaled state. • 10060: WSAETIMEDOUT See the full list of Windows System Error Codes.
13.11 Windows exceptions
• EXCEPTION_ACCESS_VIOLATION = STATUS_ACCESS_VIOLATION = c0000005 (hex) = 3221225477 or -1073741819 • CONTROL_C_EXIT = STATUS_CONTROL_C_EXIT = C000013A (hex) = 3221225786 • STATUS_STACK_BUFFER_OVERRUN = C0000409 (hex) = 3221226505 • https://github.com/wine-mirror/wine/blob/master/include/winbase.h
104 Chapter 13. Survivor Guide to Develop on Windows Victor Stinner’s Notes Documentation, Release 1.0
• https://github.com/wine-mirror/wine/blob/master/include/winnt.h
13.12 OpenSSH server
• Go to Settings • Search for “Manage Optional Features” • Add an optional feature: “OpenSSH Server” • Install it. • Go to the Windows Start menu, search for Services. • In Services, search for the OpenSSH SSH Server. • Right click, properties: – Starting type: Automatic. – Click: OK • Right click: Start. To use the OpenSSH server from Microsoft (the “Optional feature”), you need at least Windows 10 build 1803. Before, this flavor was unusable. • Go to settings, search for “Manage Optional Features”: enable OpenSSH • In my case, I had to run \Windows\System32\OpenSSH\ssh-keygen -A • The SSH private key is stored in %ProgramData%\ssh\ssh_host_ed25519_key. This file must be owned by SYSTEM and the only permission must be that SYSTEM is allowed to Read this file. • To allow incoming TCP connections to port 22 (SSH), run PowerShell as administrator and type:
New-NetFirewallRule-Name sshd-DisplayName'OpenSSH SSH Server'-Enabled True -
˓→Direction Inbound-Protocol TCP-Action Allow-LocalPort 22
• Copy your SSH public key into C:\Users\vstinner\.ssh\authorized_keys (replace vstinner with your username!) • Go to Windows Menu>search for “Services”. In Services, search for “OpenSSH Server”: click on Start. • If OpenSSH server doesn’t work, look into %ProgramData%\ssh\Logs\sshd.log • If the server works, you can change the Service start from Manual to Automatic. To debug, you can install psexec, open a shell as SYSTEM with psexec -i -s -d cmd.exe and then type: C:\Windows\System32\OpenSSH\sshd.exe to run the SSH server in foreground. Files and directories: • C:\Windows\System32\OpenSSH\sshd.exe: the SSH server program • C:\ProgramData\ssh\ssh_host_ed25519_key: SSH server private key • C:\ProgramData\ssh\sshd_config: SSH server configuration file • C:\ProgramData\ssh\Logs\sshd.log: SSH server logs
13.12. OpenSSH server 105 Victor Stinner’s Notes Documentation, Release 1.0
13.13 Misc
• Get system load:: wmic cpu get loadpercentage
13.14 Disable Windows Defender Realtime protection
On an idle Windows VM, the VM uses between more than 150% of the CPU. If I move the mouse cursor, the CPU usage goes to 50%. It is the msmpeng.exe process which uses the CPU. “ps” in PowerShell and the Task Manager don’t agree on the CPU usage: 50% according to ps, 2% according to the Task Manager. . . If I disable Real Time protection in Windows Defender, the feature is enabled again at next reboot. . . I had to add a key into registry to ensure that Windows doesn’t reenable Real Time protection after reboot: • run regedit.exe • Go to HKEY_LOCAL_MACHINESOFTWAREPoliciesMicrosoftWindows Defender • Create a “DWORD (32-bit)” key called “DisableAntiSpyware”, set its value to 1. • Done.
13.15 Maximum path length
• https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file • MAX_PATH = 260 characters • Does the system support long path? Query ntdll.RtlAreLongPathsEnabled() Application manifest to opt-in for “long path”:
˓→
An application only supports long if the system supports long path and the application opts in for long path.
13.16 C languages: Windows types
Windows Data Types: • LPCTSTR: CONST WCHAR * if UNICODE defined, CONST CHAR * otherwise • UINT:: unsigned int
13.17 C Runtime library (CRT)
Visual Studio provides a C Runtime library (CRT). Its source code can be found in: “%ProgramFiles(x86)%Windows Kits10Source10.0.[version]ucrtenv”.
106 Chapter 13. Survivor Guide to Develop on Windows Victor Stinner’s Notes Documentation, Release 1.0
13.18 Debug Windows Update failure
• Google the error code • Open cmd.exe as an adminitrator and run: sfc /scannow (SFC fixes system files integrity) • Open cmd.exe as an adminitrator and run: DISM /Online /Cleanup-Image /RestoreHealth (re- pair corrupted files of installed packages) Not tested: CHKDSK C: /F /R (/F repairs errors, /R checks for bad sectors).
13.19 MSDN
• Download Windows 10 ISO: https://www.microsoft.com/en-us/software-download/windows10ISO • Get a Windows Product key: https://my.visualstudio.com/productkeys
13.20 Time
Kernel ticks: • The kernel uses an interruption at 64 Hz: 15.625 ms per tick • NtQueryTimerResolution() gives the min/max and current resolution of the tick Wait: • WaitForSingleObject(): resolution of 1 tick • Sleep(): resolution of 1 tick System clock: • GetSystemTimeAsFileTime(): resolution of 1 tick • GetSystemTimePreciseAsFileTime() (Windows 8 and newer) Monotonic clock: • GetTickCount64(): resolution of 1 tick Performance counter: • QueryPerformanceCounter(): resolution of 1 / frequency (100 ns on Windows 10) • QueryPerformanceFrequency(): 10 MHz on Windows 10 Timer: • CreateWaitableTimer() • SetWaitableTimer(): resolution of 100 ns Multimedia API, winmm.lib and timeapi.h: • Frequency up to 1 kHz: 1 ms per tick • timeBeginPeriod() * “Setting a higher resolution can improve the accuracy of time-out intervals
13.18. Debug Windows Update failure 107 Victor Stinner’s Notes Documentation, Release 1.0
in wait functions. However, it can also reduce overall system performance, because the thread sched- uler switches tasks more often. High resolutions can also prevent the CPU power management system from entering power-saving modes. Setting a higher resolution does not improve the accuracy of the high-resolution performance counter.” See also: • The Windows Timestamp Project
108 Chapter 13. Survivor Guide to Develop on Windows 109 Victor Stinner’s Notes Documentation, Release 1.0
CHAPTER 14
GDB: GNU debugger
110 Chapter 14. GDB: GNU debugger Victor Stinner’s Notes Documentation, Release 1.0
14.1 Symbols
By default, gdb processes an expression as C code. func@plt is valid. Use quotes to pass special symbols. Wrong:
(gdb) p PyTuple_New.constprop.0 A syntax error in expression, near `.0'.
Good:
(gdb) p 'PyTuple_New.constprop.0' $5 = {PyObject *(Py_ssize_t)} 0x47cb60
14.2 Function with multiple locations
Example:
(gdb) b PyTuple_New Breakpoint2 at 0x47cb60: PyTuple_New.(2 locations)
(gdb) info breakpoints Num Type Disp Enb Address What 2 breakpoint keep y
˓→internal/pycore_pystate.h:141 2.2y 0x000000000047d540 in PyTuple_New at./Include/
˓→internal/pycore_pystate.h:141
(gdb) info symbol 0x000000000047cb60 PyTuple_New.constprop.0 in section.text
(gdb) info symbol 0x000000000047d540 PyTuple_New in section.text gdb picks the first in the symbol table:
(gdb) info address PyTuple_New Symbol "PyTuple_New" is a function at address 0x47cb60.
(gdb) p PyTuple_New $6 = {PyObject *(Py_ssize_t)} 0x47cb60
.constprop.0 stands for “constant propagation”. GCC specialized PyTuple_New(size) for size==0. There is also .isra.3 which comes from -fipa-sra: Perform interprocedural scalar replacement of aggregates, removal of unused parameters and replace- ment of parameters passed by reference by parameters passed by value. Try also: • set print demangle off • set print asm-demangle • info functions PyTuple_New
14.1. Symbols 111 Victor Stinner’s Notes Documentation, Release 1.0
14.3 gdb commands
• info sharedlibrary: list loaded shared libraries (see also sharedlibrary command to explicitly load shared libraries) • info sources: list code source files
14.4 Debug methodolody
The first step is to debug is to find a reliable or highly reliable scenario to reproduce the bug. In the worst case, it can take several days to get such reliable scenario :-/
14.4.1 Code bisection
If a bug occurred recently and it didn’t exist in previous versions, it’s called a regression and it should be easier to debug. If the code is tracked by a Version Control System (VCS like Git or Mercurial), you can “bisect” the source code history to find the changeset introducing the bug. Git and Mercurial have a builtin bisect command: • hg bisect • git bisect The best is to have a reliable script reproducing the bug which returns 0 on success (test succeeded, no bug) or non-zero exit code (test failed, bug). Example of shell script:
make|| exit 125 ./python-m test-v test_sys
make || exit 125 fails with the exit code 125 if the compilation fails. It’s useful to skip a revision if the project cannot be compiled at this revision. hg bisect and git bisect skip a revision if the command exit code is 125. The idea is similar to dichotomy: reduce the quantity of code that have to be read to find a bug.
14.4.2 Dichotomy
Dichotomy or “divide to conquer” ;-) The idea of the “dichotomy” methodology is to reduce the quantity of executed code before the bug is triggered. To reduce the quantity of code, you can: • comment function calls • disable most features: turn off logging, turn off audio and/or video playback, etc. • comment large partion of code The goal is not the reduce the code to a single line of code reproducing the bug. The goal is the reduce the quantity of code that should be read manually to find the bug. The debug method can sometimes take several hours, so it’s nice to have “milestones”: backup points where the bug can still be reproduced. If you use a Version Control System (VCS like Git or Mercurial), you can use local commits (ex: create a local branch in git). Don’t worry of the change content or the commit message, plop is a great commit message for such changes :-) These milestones are important to be able to go backward if the bug cannot be reproduced anymore when you disabled too much code and features.
112 Chapter 14. GDB: GNU debugger Victor Stinner’s Notes Documentation, Release 1.0
14.4.3 Add printf
Debuggers are great, convenient and powerful. But. Sometimes, a bug cannot be reproduce in the debugger for an unknown reason, or the control flow is too complex to run the application in a debugger. For example, it’s hard to debug an event based application where a single logical “function” (or coroutine) is splitted into several small callbacks. In such case, an easy method is to add “print” calls (ex: printf() in C or print in Python) in the code to “dump” the control flow, to try to bisect manually the code. Example of a Python function body:
func1(a) func2(b) return func3(c)
Add print calls to see where the bug is triggered:
print("func1") func1(a) print("func2") func2(b) print("func3") res= func3(c) print("exit") return res
The last instruction was splitted into two instructions to see if the bug occurs before the call to func3() or after. If you see func1 and func2 but not func3, the bug occurred after the line print("func2") and before print("func3"), so the bug occurred on the func2() call. You can now remove the print("func1") line (to have a less verbose output), and continue to add print calls inside the func2() function. Iterate until you have a few functions to read to find the bug. It’s common to get the failing line in less than 5 iterations, the technic looks very basic, but it’s fast if the scenario to reproduce the bug is reliable. Debugging can be painful, so don’t hesitate to make it more funny by using less boring messages than func1 or func3 :-) I’m using: • LA (here in french) • 1 • 1b • pouet • @@@@@@@@@@@ • etc.
Note: If the source code is not tracked by a Version Control System (VCS like Git or Mercurial), don’t forget to create backup of files to easily remove these print calls when the bug is identified.
14.5 Gdb
• Pretty printer: set print pretty on • Enter TUI/exit TUI: CTRL+x a
14.5. Gdb 113 Victor Stinner’s Notes Documentation, Release 1.0
• https://sourceware.org/gdb/onlinedocs/gdb/TUI-Keys.html • CTRL+x o: change active window • Display full print value: set print elements 1024 (or set print elements 0 if you are brave) • Print variable type: whatis variable • Dump the structure of a variable: ptype variable • LD_LIBRARY_PATH: gdb -iex "set env LD_LIBRARY_PATH=$PWD" --args ./python Lib/test/gdb_sample.py • Autoload Python: set auto-load python-scripts on • Load a Python script: source script.py
14.6 x86_64 assembler, gdb https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI Stack aligned on 16 bytes boundary. Calling convention: • arg1: RDI • arg2: RSI • arg3: RDX • arg4: RCX • arg5: R8 • arg6: R9
14.7 gdb
• Truncated string: set print elements 0 • TUI: – CTRL+x a: enable/disable TUI – (Inside TUI) CTRL+x o: switch to the next TUI window – See also https://sourceware.org/gdb/onlinedocs/gdb/TUI-Keys.html • Stop on PyType_Ready() but only if type->tp_name is the string “_ModuleLock”:
(gdb) b PyType_Ready Breakpoint 2 at 0x4faa9c: file Objects/typeobject.c, line 4980.
(gdb) run ... Breakpoint 2, PyType_Ready (type=0x953ba0
˓→typeobject.c:4980 4980 if (type->tp_flags & Py_TPFLAGS_READY) {
(gdb) condition 2 strcmp(type->tp_name, "_ModuleLock")==0 (continues on next page)
114 Chapter 14. GDB: GNU debugger Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) (gdb) cont
Breakpoint 2, PyType_Ready (type=0x9ecf78) at Objects/typeobject.c:4980 4980 if (type->tp_flags & Py_TPFLAGS_READY) { (gdb) p type->tp_name $6 = 0x7ffff7f83080 "_ModuleLock"
• Breakpoint on a value:
(gdb) watch type->tp_init Hardware watchpoint4: type->tp_init
(...)
Hardware watchpoint4: type->tp_init
Old value= (initproc) 0x0 New value= (initproc) 0x4f3e6e
˓→typeobject.c:4944
• Run until line 4988:
(gdb) u 4899 inherit_slots (type=0x9ecf78, base=0x953ba0
˓→typeobject.c:4899
14.8 Write a core dump in disk
Fedora catchs fatal errors like segmentation faults with its application ABRT. To develop, sometimes it helps to get a core dump. It’s possible to write a core dump on disk with: ulimit-c unlimited sudo bash-c"echo' %e-%p.core' > /proc/sys/kernel/core_pattern"
Test:
$ python3 >>> import faulthandler; faulthandler._sigsegv() Erreur de segmentation (core dumped) $ ls *core* python3-27542.core
14.9 Display a wchar_t string
Use this macro: define wc_print echo " set $c = (wchar_t*)$arg0 while ( *$c ) (continues on next page)
14.8. Write a core dump in disk 115 Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) if ( *$c > 0x7f ) printf "[%x]", *$c else printf "%c", *$c end set $c++ end echo "\n end
14.10 Breakpoint in GDB
Write following code into bp.py: class MyBreakpoint(gdb.Breakpoint): def stop(self): caller= gdb.newest_frame().older() caller_name= caller.name() if caller else 'none' return (caller_name !='func_dealloc')
MyBreakpoint('func_clear')
In gdb, type: source bp.py. It puts a breakpoint on the function func_clear() but stop if the the caller function name is not func_dealloc().
14.11 Reverse
• http://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial • “Process record does not support instruction 0xc5 at address . . . ”: gdb doesn’t support AVX. Workaround? – LD_HWCAP_MASK=0?
116 Chapter 14. GDB: GNU debugger CHAPTER 15
Unsorted Notes
15.1 GitHub
• https://github.com/pulls/review-requested URLs: • Add ?w=1 in a pull request to ignore whitespace changes • Add .patch to a pull request to get the change as an unified diff • In a message,
117 Victor Stinner’s Notes Documentation, Release 1.0
15.2 Code search
• https://github.com/search/ • http://searchcode.com/ • https://codesearch.debian.net/ • http://stackoverflow.com/search?q=codecs.encode • specific to OpenStack: http://codesearch.openstack.org/
15.3 Git
15.3.1 Remove latest commit git reset--hard HEAD~1
15.3.2 List tags containing a specific commit
$ git tag --contains 94a3b83f9f1fd52a78b9d49b32ddfae40182f852 12.0.0.0b1 12.0.0a0 2014.2 2014.2.1 ...
15.3.3 Remote branches
• List remote branches: git branch -r • Create a new branch fix_1369426_icehouse tracking the remote branch origin/stable/ icehouse:
git branch--track fix_1369426_icehouse origin/stable/icehouse
• (Track and) Pull a remote branch:
git branch--track NAME_REMOTE_BRANCH git fetch--all # or: git pull --all
15.4 Shell script
• bash8: A pep8 equivalent for bash scripts • checkbashisms: static analysis tool for shell scripts. It looks for particular patterns which indicate a script might be relying on /bin/sh being bash. • shellcheck: static analysis and linting tool for sh/bash scripts • $'...' interprets escape sequences (like \n) in '...'
118 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
• <<<"HELLO" syntax, known as “here-string”, creates a temporary file which contains the string HELLO and uses this file as the child process stdin (fd 0). • "$(...)" syntax allows to pass the output of a command to a program with newline characters:
python3-c"$(echo -e" for i in range(3):\n print(i)")"
Example: haypo@selma$ echo $'a\rb'|hexdump -C 00000000 61 0d 62 0a |a.b.| 00000004
• sh is supposed to be the minimalist shell (faster, but less feature) • bash has more feature and is quite common, but not available by default on FreeBSD for example. • dash is a minimalist shell used as ‘sh’ on Debian Test: • [ is a program: /usr/bin/[ on Linux • man test • man [ # sometimes display bash manual page • [[ . . . ]] is a bash built-in, so specific to bash Replace name.py string with name, remove .py suffix: script="name.py" # display "name" echo ${script:0:-3}
Misc: • https://pypi.org/project/bashate/
15.5 Friends
• http://blog.sileht.net/ • http://www.florentflament.com/ • http://yeknan.free.fr/dc2/ Fun: • http://tumourrasmoinsbete.blogspot.fr/ • http://www.commitlogsfromlastnight.com/
15.6 Google
What Google knowns on you: • https://myactivity.google.com/ • https://myaccount.google.com/
15.5. Friends 119 Victor Stinner’s Notes Documentation, Release 1.0
• https://maps.google.fr/locationhistory/ • https://takeout.google.com/
15.7 Operating systems macOS (Mac OS X) versions:
macOS Name Darwin Version Release Year macOS 10.13 High Sierra 17.x 2017 (June) macOS 10.12 Sierra 16.x 2016 macOS 10.11 El Capitan 15.x 2015 macOS 10.10 Yosemite 14.x 2014 macOS 10.9 Mavericks 13.x 2013 macOS 10.8 Mountain Lion 12.x 2012 macOS 10.7 Lion 11.x 2010 macOS 10.6 Snow Leopard 10.x 2008 macOS 10.5 Leopard 9.x 2006 macOS 10.4 Tiger 8.x 2004
Use sw_vers in the command line to get macOS version. • Linux kernel versions: – 4.0: 2015 (under development) – 3.0: 2011 – 2.6: 2003 – 2.4: 2001 • Ubuntu releases: – 16.10: Yakkety Yak (not released yet, scheduled for 2016-10-20) – 16.04 LTS: Xenial Xerus, 2016-04-21 – 15.10: Wily Werewolf, 2015-10-22 – 15.04: Vivid, 2015-04 – 14.10: Utopic, 2014-10 – 14.04 LTS: Trusty, 2014-04 – 12.04 LTS: Precise, 2012-04 • Fedora releases: – Fedora 24: 2016-06-21 – Fedora 23: 2015-11-03 – Fedora 22: 2015-05-26 – Fedora 21: 2014-12 – Fedora 20: 2013-12, Heisenbug – Fedora 19: 2013-07, Schrödinger’s Cat
120 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
• Debian releases: – Debian 9 “Stretch”: June 17th, 2017 – Debian 8 “Jessie”: April 26th, 2015 FreeBSD releases, and Unsupported FreeBSD Releases:
FreeBSD Release End of life FreeBSD 11.0 2016-10 2021-09-30 FreeBSD 10.0 2014-01 2018-10-31 FreeBSD 9.0 2012-01 2016-12 FreeBSD 8.1 2010-07 2012-07 FreeBSD 7.0 2008-02 2009-04 FreeBSD 6.2 2007-01 2008-05
Microsoft Windows versions(version numbers):
Windows Version Release End of mainstream support Extended support Windows 10 10.0 2015-07 2020-10 2025-10 Windows 8.1 6.3 2013-10 2018-01 2023-01 Windows 8 6.2 2012-10 2016-01 2016-01 Windows 7 6.1 2009-10 2015-01 2020-01 Windows Vista 6.0 2007-01 2012-04 2017-04 Windows XP Professional x64 5.2 2005-04 2009-04 2014-04 Windows XP 5.1 2001-10 2009-04 2014-04
Note: For applications that have been manifested for Windows 8.1 or Windows 10. Applications not manifested for Windows 8.1 or Windows 10 will return the Windows 8 OS version value (6.2). To manifest your applications for Windows 8.1 or Windows 10, refer to Targeting your application for Windows.
Linux kernel: • Active kernel releases
Linux kernel Released Projected EOL 4.14 2017-11-12 2020-01 4.9 2016-12-11 2019-01 4.4 2016-01-10 2022-02 4.1 2015-06-21 2018-05 3.16 2014-08-03 2020-04 3.2 2012-01-04 2018-05 2.6 2003-12-17 2011-08
15.8 Gnome-Terminal
Configure Gnome-Terminal to select a full URL double-click: dconf write /org/gnome/terminal/legacy/profiles:/:${Profile_ID}/word-char-exceptions
˓→'@ms "-,.;/?%&#_=+@~·:"'
15.8. Gnome-Terminal 121 Victor Stinner’s Notes Documentation, Release 1.0
Replace ${Profile_ID} with the profile identifier. To get it:
$ gsettings get org.gnome.Terminal.ProfilesList list ['b1dcc9dd-5262-4d8d-a863-c897e6d979b9']
Example: dconf write/org/gnome/terminal/legacy/profiles:/:b1dcc9dd-5262-4d8d-a863-
˓→c897e6d979b9/word-char-exceptions'@ms"-,.;/?%&#_=+@~·:"'
To see notifications on irssi, use XTerm color theme, rather than the default “Tango” theme: XTerm theme has a better contrast.
15.9 Android
Avoid music applications (Spotify, radio) to stop when idle (phone locked): • Parameters > Network > Save data > select application: allow your music applications • Parameters > Batterie > Applications: allow your music applications
15.10 IRC
Give operator and owner permission to mdk:
/msg chanserv FLAGS #python-fr mdk +AFRefiorstv
Kick a spammer with a link to AFPy charter:
/msg ChanServ AKICK #python-fr ADD spammer_nickname !T 1h https://www.afpy.org/docs/
˓→charte
List operators of channel:
/msg ChanServ access #python-fr list
#python-dev flags to prevent people who are not logged in to an account from talking:
/mode #python-dev -q $~a
15.11 SSH keygen
Create an SSH key: ssh-keygen-t ed25519-o-a 100-C"haypo2017"-f ssh_key
• -t: key type, http://ed25519.cr.yp.to/ • -a 100: use 100 rounds of the key derivation function for the passphrase, increase resistance to brute-force password cracking • -C: comment
122 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
• -f: filename • -o: save private keys using the new OpenSSH format, increased resistance to brute-force password cracking (in fact, -t ed25519 already enables this option) Issues with ed25519: • Launchpad doesn’t support ed25519: Launchpad is implemented on top of Twisted which doesn’t support ed25519 yet. https://bugs.launchpad.net/launchpad/+bug/1282220 • gnome-keyrign doesn’t support the new SSH key format used by ed25519 by default: https://bugzilla.gnome. org/show_bug.cgi?id=723274 https://bugzilla.gnome.org/show_bug.cgi?id=641082 Links: • https://stribika.github.io/2015/01/04/secure-secure-shell.html • https://wiki.archlinux.org/index.php/SSH_keys SSH agent: • Modify /etc/pam.d/* to lines containing “pam_gnome_keyring.so” • Make sure that login still works after the change!!! Gnome and SSH passphrase:
sudo dnf install-y openssh-askpass
15.12 tmux
• tmux attach • tmux ls • CTRL+b . . . – [: navigation (scroll), ‘q’ to quit navigation mode – d: detach – c: new window – n / p: next/previous window – :: open the command line (“prompt”) – ,: name the window – w: window list – &: kill the window • Command line or “prompt” (opened by CTRL+b :): – list-sessions • tmux shortcuts & cheatsheet
15.12. tmux 123 Victor Stinner’s Notes Documentation, Release 1.0
15.13 Rounding
Wikipedia: https://en.wikipedia.org/wiki/Rounding Rounding modes for floating point numbers: • ROUND_FLOOR: Round towards minus infinity (-inf). – C: floor() – Python: math.floor(float) – Python: math.floor(-0.1) == -1 – Python: math.floor(0.9) == 0 – For example, used to read a clock. • ROUND_CEILING: Round towards infinity (+inf). – Python: math.ceil(float) – Python: math.ceil(0.1) == 1 – Python: math.ceil(-0.1) == 0 • ROUND_HALF_EVEN: Round to nearest with ties going to nearest even integer. – For example, used to round from a Python float. – Python: round(float) – Python: round(0.5) == 0 – Python: round(1.5) == 2 – Python: round(2.5) == 2 – This is the default rounding mode used in IEEE 754 floating-point operations. • ROUND_UP: Round away from zero. – For example, used for timeout. ROUND_CEILING rounds -1e-9 to 0 milliseconds which causes bpo- 31786 issue. ROUND_UP rounds -1e-9 to -1 millisecond which keeps the timeout sign as expected. select.poll(timeout) must block for negative values. • ROUND_DOWN: Round towards zero. – C: (int)double, ex: (int)0.9 == 0 – Python: int(float) – Python: int(0.9) == 0 – Python: int(-0.9) == 0 – Python: float.__trunc__() Other rounding modes (ex: Python decimal module): • ROUND_HALF_DOWN: Round to nearest with ties going towards zero. • ROUND_HALF_UP: Round to nearest with ties going away from zero. • ROUND_05UP: Round away from zero if last digit after rounding towards zero would have been 0 or 5; other- wise round towards zero. IEEE 754 defines 4 modes:
124 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
• ROUND_HALF_EVEN: default mode • ROUND_FLOOR • ROUND_CEILING • ROUND_DOWN Links: • https://vstinner.github.io/pytime.html • “double-rounding” https://bugs.python.org/issue24567 • https://bugs.python.org/issue32956 • double to float rounding on ppc64le: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88892
15.14 Linux: follow process execution
• execsnoop • linux process monitoring: NETLINK_CONNECTOR with CN_IDX_PROC and CN_VAL_PROC commands • exec-notify.c: PROC_EVENT_EXEC reading /proc/pid/cmdline
15.15 wget mirror
Download a “Index of” Apache listing and subdirectories, but not parents. wget –mirror –no-parent -e robots=off URL robots=off is needed to downloda OpenStack CI logs, since the robots.txt disallow everything.
15.16 dd
Write a raw image to a USB key: lsblk # check if the USB key is connected sudo dd if=bios.img of=/dev/disk/by-id/usb-LEXAR_JUMPDRIVE_0A4F1007191812160305-0\:0
˓→status=progress oflag=direct
15.17 ssh-agent
List keys of ssh-agent: ssh-add-l
Add a key: ssh-add~/.ssh/id_rsa
Remove all keys:
15.14. Linux: follow process execution 125 Victor Stinner’s Notes Documentation, Release 1.0
ssh-add-D
15.18 Status pages
• Python : https://status.python.org/ • GitHub : https://www.githubstatus.com/ and https://twitter.com/githubstatus • Travis CI : https://www.traviscistatus.com/ and https://twitter.com/traviscistatus
15.19 KDE Connect on Fedora
Commands: sudo dnf install kde-connect-nautilus sudo firewall-cmd--zone=public--permanent--add-port=1714-1764/tcp sudo firewall-cmd--zone=public--permanent--add-port=1714-1764/udp sudo systemctl restart firewalld.service
See also https://community.kde.org/KDEConnect
15.20 SELinux
Display SELinux alerts in Gnome: sealert. Dummy command to restore SELinux labels on the whole operating system: restorecon-Rv/
/etc/selinux/config config file:
SELINUX=enforcing SELINUXTYPE=targeted
Check current SELinux config:
$ getenforce Enforcing
15.21 posix_spawn
Python issues: • expose posix_spawn(p) • Support POSIX_SPAWN_USEVFORK flag in posix_spawn • subprocess uses os.posix_spawn in some cases vfork:
126 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
• https://ewontfix.com/7/ Performance: • https://github.com/rtomayko/posix-spawn
15.22 Valgrind
Search for memory leak: malloc() not followed by free(), limit the call stack to 20 frames:
PYTHONMALLOC=malloc valgrind--leak-check=full--num-callers=20./python x.py
Valgrind with gdb server to inspect a bug in gdb:
# First terminal valgrind--vgdb=yes--vgdb-error=0 program [arg1 arg2...]
# Second terminal gdb # then type in gdb: # (gdb) target remote | vgdb
Generate a suppression for a false alarm:
--gen-suppressions=yes
Python issues related to Valgrind: • https://bugs.python.org/issue38118 • https://bugs.python.org/issue37329
15.23 Floating point number
Binary IEEE 754: • http://fabiensanglard.net/floating_point_visually_explained/ • Python 3.9: math.ulp(), math.nextafter() • http://0.30000000000000004.com/ Other: • GMP: free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating- point numbers. • MPFR: multiple-precision floating-point computations with correct rounding. MPFR is based on the GMP multiple-precision library. • MPFI: multiple precision interval arithmetic library based on MPFR
15.24 Mplayer
Increase maxiumum volume:
15.22. Valgrind 127 Victor Stinner’s Notes Documentation, Release 1.0
mplayer-softvol-softvol-max 300 video.avi
15.25 Virtualization: run an AArch64 VM on x86-64
Before starting virt-manager, install (edk2-aarch64 is for UEFI):
sudo dnf install qemu-system-aarch64 edk2-aarch64
In virt-manager, pick “arch: AArch64” in the first dialog of the wizard. • https://fedoraproject.org/wiki/Architectures/AArch64/Install_with_QEMU
15.26 Coredump Linux
Default configuration:
$ cat /proc/sys/kernel/core_pattern |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
Create coredump file in the current directory:
sudo bash-c'echo" %e.%p.core" > /proc/sys/kernel/core_pattern'
Create coredump filename like python-123.core. Maximum core dump size:
$ ulimit -c unlimited
Test:
$ ./python -c 'import ctypes; ctypes.string_at(0)' Segmentation fault (core dumped) $ ls *.core python.347656.core
See also man core.
15.27 Contributions to open source
GCC bug reports: • https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93384 • https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88892 • https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47271
128 Chapter 15. Unsorted Notes Victor Stinner’s Notes Documentation, Release 1.0
15.28 Firefox about:config: • image.animation = once (default = “normal”) • mousewheel.with_alt.action = 1: – https://fedoraproject.org/wiki/Common_F32_bugs#Trying_to_scroll_with_mouse_wheel_in_inactive_ Firefox_window_results_in_back.2Fforward_instead – https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1650051 – https://gitlab.gnome.org/GNOME/gtk/issues/2112
15.29 Enter namespace filesystem of a Flatpak application or con- tainer
If a Flatpak application is the pid 76688, inspect the process with: • /proc/76688/root/ : Filesystem of the process. • /proc/76688/mountinfo : Mount informations • /proc/76688/ns/mnt : points to “mnt:[4026533594]” For example, in a Flatpak application, the first line of mountinfo is something like “(. . . ) /newroot / rw,nosuid,nodev,relatime - tmpfs tmpfs (. . . )” which means that the whole operating system is in memory, not on disk. Only following mounts can map to directories on the machine disk. See also the nsenter command, and ip netns help for network namespaces.
15.30 Debian
• List files contained in a package: dpkg --listfiles python3.9-dev. • Search which package contains a file: dpkg -S /path/to/file.
15.31 Gmail filters
• Google Support: Gmail filters • GitHub notifications • Manage GitHub notification messages in Gmail with Google Apps Scripts
15.32 Dev Cython
Run a single test of the Cython test suite:
~/python/master/python runtests.py'. *test_unicode.*'-vv
15.28. Firefox 129 Victor Stinner’s Notes Documentation, Release 1.0
15.33 Video for Linux (V4L): control your webcam
• GUI: gtk-v4l • CLI: v4l2-ctl --list-devices
130 Chapter 15. Unsorted Notes CHAPTER 16
Inspect an ELF binary file
On Linux, they are two main kinds of ELF binary files: programs (/usr/bin/bash) and dynamic libraries (/usr/ lib64/libc.so.6).
16.1 Symbols
List imported symbols:
objdump-T file
or:
# -D/--dynamic: display the dynamic symbols rather than the normal symbols nm-D file
List exported symbols:
objdump-T file
16.2 Misc
Disassemble all sections:
objdump-d file
131 Victor Stinner’s Notes Documentation, Release 1.0
132 Chapter 16. Inspect an ELF binary file CHAPTER 17
systemd
17.1 List services
Find the name of the systemd unit for MariaDB or RabbitMQ server. List all installed services, including disabled services, and search for “maria”: systemctl list-unit-files--type=service| grep maria
Alternative if you know the package name:
$ rpm -ql mariadb-server|grep service /usr/lib/systemd/system/mariadb.service
List enabled services: systemctl list-units
Note: it looks like “list-units” doesn’t show mariadb.service, probably because it is disabled (not started at boot).
17.2 System logs: journald and journalctl
• Binary logs are written into /var/log/journal/ • Show syslog from the most recent to the oldest logs: journalctl --reverse • Show all logs since the last boot: journalctl -b 0 • List boots: journalctl --list-boots • tail -f /var/log/syslog: journalctl -f • tail -f /var/log/syslog but only for apache: journalctl -u apache.service -f • Kernel logs of the current boot: journalctl -k (similar to dmesg but with better timestamp)
133 Victor Stinner’s Notes Documentation, Release 1.0
• Retain only journald logs of the past 30 days: journalctl --vacuum-time=30d • Filters: – journalctl _PID=7797 show logs of process pid 7797 – journalctl _COMM=chronyd -r shows latest logs of the program chronyd • Other fields: – _EXE: program full path – _CMDLINE: program command line with arguments – _UID: user identifier – _GID: group identifier – _BOOT_ID: boot UUID – _MACHINE_ID: machine UUID – _HOSTNAME: hostname – __REALTIME_TIMESTAMP: timestamp: number of microseconds (10**-6) since Unix epoch (January 1st, 1970 at 00:00), in the UTC timezone. – SYSLOG_IDENTIFIER • See all journalctl fields: journalctl -o export Example of a single log entry (in export mode):
__CURSOR=s=c9faccefcf184d67b8a1a1a8c9441d83;i=9417;b=b6852be278a647e3b1f1047604d828c8;
˓→m=3e8ae848;t=5a093d6b361fa;x=2c3f533b3fc622ed __REALTIME_TIMESTAMP=1583931706270202 __MONOTONIC_TIMESTAMP=1049290824 _BOOT_ID=b6852be278a647e3b1f1047604d828c8 PRIORITY=6 SYSLOG_FACILITY=3 _UID=1000 _GID=1000 _SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 _AUDIT_SESSION=3 _AUDIT_LOGINUID=1000 _SYSTEMD_OWNER_UID=1000 [email protected] _SYSTEMD_SLICE=user-1000.slice _SYSTEMD_USER_SLICE=-.slice _MACHINE_ID=1a6df4a3ce76477790264e5d3a9fa609 _HOSTNAME=apu _TRANSPORT=stdout SYSLOG_IDENTIFIER=gnome-shell _COMM=gnome-shell _EXE=/usr/bin/gnome-shell _CMDLINE=/usr/bin/gnome-shell _CAP_EFFECTIVE=800000 _SYSTEMD_CGROUP=/user.slice/user-1000.slice/[email protected]/gnome-shell-wayland.
˓→service _SYSTEMD_USER_UNIT=gnome-shell-wayland.service _PID=1702 _SYSTEMD_INVOCATION_ID=5afe30294e9c49d6b27ff3011323619a _STREAM_ID=e46f4fd89cd3445fbac494d0814d34a5 (continues on next page)
134 Chapter 17. systemd Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) MESSAGE <80>^@^@^@^@^@^@^@libinput error: client bug: timer event13 debounce short: scheduled
˓→expiry is in the past (-5ms), your system is too slow
Advantages over scattered text log files: • Timestamps seem to be more reliable, especially for kernel logs • Ability to display logs in the reverse order • Ability to filter logs by service, by user, by process pid, by boot, etc. • . . . in fact, I rarely use logs, so I don’t have strong expectations for logs :-)
17.3 Advantages of systemd to run services
• I like systemd global design to build “stateless” services, by isolating them from the system for example. • Security: systemd gives access to high level security protections like running a service with its own private temporary directory /tmp (option PrivateTmp). Options: – ProtectKernelModules=yes – MemoryDenyWriteExecute=yes – RestrictRealtime=yes – PrivateNetwork=yes – PrivateTmp=yes – ProtectSystem=yes – ProtectHome=yes – Read also Using systemd for more secure services in Fedora • systemd can even create a couple of temporary (user, group) to run a service and remove theme once the service stops. To be able to implement this feature, systemd has to cleanup all resources owner by the user. Running the service with a read-only filesystem except of a single writable directory helps to remove all files created by the service. Removing all IPC owned by a user is part of this cleanup (option RemoveIPC). • systemctl status service shows the last log lines. • Thanks to cgroups, systemd is able to list all processes of a service in a secure manner. systemctl status service lists all process identfiers of the service (main pid, but also pids of child processes). Moreover, when systemd stops a service, the usage of a cgroups makes sure that all processes are killed. Bye bye the legacy and annoying “pid file” causing so many troubles. • The simple .service file format makes it much easier to share these files between Linux distributions. Linux distributions can collaborate on more complex issues like handling properly NFS mounts: Systemd program- ming, 30 months later. Moreover, it’s easier to enable security protections for all Linux distributions. There is a similar trend to isolate desktop applications using sandboxes: see Flatpak. For security, but also to reduce dependencies to the system, and so run an old application on a newer system, or the opposite. Embedding libraries in Flatpak “containers” comes with its own set of issues, but that’s a different topic ;-)
17.3. Advantages of systemd to run services 135 Victor Stinner’s Notes Documentation, Release 1.0
17.4 coredumpctl
See also Fedora ABRT. Configuration: https://www.freedesktop.org/software/systemd/man/coredump.conf.html See also: https://wiki.archlinux.org/index.php/Core_dump coredumpctl checks for coredump in /var/lib/systemd/coredump/ directory:
$ coredumpctl list TIME PID UID GID SIG COREFILE EXE Wed 2020-03-11 13:46:38 CET 3350 1000 1000 11 present /usr/bin/python3.7 Wed 2020-03-11 13:48:28 CET 2211 1000 1000 11 present /usr/bin/abrt-applet
$ coredumpctl dump /usr/bin/abrt-applet > core PID: 2211 (abrt-applet) (...) Signal: 11 (SEGV) Timestamp: Wed 2020-03-11 13:48:27 CET (30min ago) Command Line: /usr/bin/abrt-applet --gapplication-service (...) Storage: /var/lib/systemd/coredump/core.abrt-applet.1000.
˓→b6852be278a647e3b1f1047604d828c8.2211.1583930907000000000000.lz4 Message: Process 2211 (abrt-applet) of user 1000 dumped core.
Stack trace of thread 2211: #0 0x00007f978e419fec problem_get_argv0 (libreport-gtk.so.0) #1 0x00005632d09a4d83 notify_problem_list (abrt-applet) #2 0x00005632d09a5291 show_problem_list_notification (abrt-applet) (...)
17.5 systemd trolls systemd features are not unique, it’s totally doable without sytemd. Right, but systemd comes with a simple configuration files (.service files) which gives an easy access to these features. systemd has bugs! Right, as any other software. And they are quickly fixed. systemd developers reject patches to support platforms other than Linux: Ok, this is a real issue. I have no answer for that one :-) Links: • Devuan: Debian fork without systemd • http://without-systemd.org/ • https://suckless.org/sucks/systemd/ BSD systems don’t use systemd but reimplemented the strict minimum systemd APIs required by Gnome.
136 Chapter 17. systemd Victor Stinner’s Notes Documentation, Release 1.0
17.6 Slow boot: systemd-udev-settle and hdaudioC1D0 systemd-analyze:
[root@apu vstinner]# systemd-analyze blame 2min 546ms systemd-udev-settle.service 47.157s dnf-makecache.service 5.935s NetworkManager-wait-online.service 3.331s plymouth-quit-wait.service 2.803s lvm2-monitor.service 2.695s fwupd.service (...)
System logs:
[root@apu vstinner]# journalctl -b mai 25 14:05:46 apu kernel: Linux version 5.6.12-300.fc32.x86_64 (...) (...) mai 25 14:06:53 apu systemd-udevd[626]: hdaudioC1D0: Worker [648] processing
˓→SEQNUM=3956 is taking a long time (...) mai 25 14:07:49 apu systemd[1]: systemd-udev-settle.service: Main process exited,
˓→code=exited, status=1/FAILURE (...) mai 25 14:07:49 apu systemd[1]: systemd-udev-settle.service: Failed with result'exit-
˓→code'. mai 25 14:07:49 apu systemd[1]: Failed to start udev Wait for Complete Device
˓→Initialization. (...) mai 25 14:08:53 apu systemd-udevd[626]: hdaudioC1D0: Worker [648] processing
˓→SEQNUM=3956 killed mai 25 14:08:53 apu systemd-udevd[626]: Worker [648] terminated by signal9 (KILL) mai 25 14:08:53 apu systemd-udevd[626]: hdaudioC1D0: Worker [648] failed
Blacklist i2c_nvidia_gpu kernel module: sudo bash-c'echo"blacklist i2c_nvidia_gpu" >/etc/modprobe.d/blacklist-i2c-nvidia-
˓→gpu.conf'
Kernel driver i2c-nvidia-gpu: “driver for I2C controller included in NVIDIA Turing and later GPUs and it is used to communicate with Type-C controller on GPUs”.
17.6. Slow boot: systemd-udev-settle and hdaudioC1D0 137 Victor Stinner’s Notes Documentation, Release 1.0
138 Chapter 17. systemd CHAPTER 18
Python in Fedora
• https://fedora-python.readthedocs.io/ • https://fedoraproject.org/wiki/SIGs/Python • Fedora Loves Python • Fedora Developer: Multiple Python interpreters • Fedora EPEL See also Fedora and Python in RHEL.
18.1 Python packages in Fedora
In April 2019, 3 Fedora versions are maintained: • f29: Fedora 29 (stable) • f39: Fedora 30 (beta) • master: Fedora Rawhide (unstable) There are the following packages for Python: • python2: Python 2.7 • python34: Python 3.4 • python35: Python 3.5 • python36: Python 3.6 • python3: Python 3.7 (in fc31) • python38: Python 3.8 • python39: Python 3.9 Old stuff:
139 Victor Stinner’s Notes Documentation, Release 1.0
• Old branches: f27 (Fedora 27), f26 (Fedora 26), . . . (python package has branches up to f10: Fedora 10) • The python package is now dead. It has been renamed to python2.
140 Chapter 18. Python in Fedora CHAPTER 19
Red Hat and Python
• Red Hat contributes to Python upstream. • Red Hat is a Diamond sponsor of the PSF See also: Python in Fedora.
19.1 Python shipped with RHEL
RHEL Python RHEL 6 Python 2.6 RHEL 7 Python 2.7 RHEL 7.7 Python 2.7.5 and Python 3.6.8 RHEL 8 Python 3.6.8, Python 3.8.0 (*), Python 2.7.15 (*)
In RHEL 8, Python 2.7 and Python 3.8 have shorter support than RHEL, they are shipped as app streams, not in the base operating system. (Latest table update: 2020-06-18.) How to install Python 3 on Red Hat Enterprise Linux 7 by Rob Terzi (August 2018). The Python shipped with RHEL is supported as long as RHEL: RHEL Life Cycle. April 2018, RHEL 7.5 Release Notes: Chapter 54. Deprecated Functionality: Python 2 has been deprecated: Python 2 will be replaced with Python 3 in the next Red Hat Enterprise Linux (RHEL) major release. How is Python 2 supported in RHEL after 2020?. • no new features will be added to Python 2 in RHEL 7 and earlier. • RHEL 8: Python 2.7 AppStream supported until June 2024.
141 Victor Stinner’s Notes Documentation, Release 1.0
By default on RHEL8, python3 is /usr/libexec/platform-python3.6 which dynamically linked to / lib64/libpython3.6m.so.1.0:
$ which python3 /usr/bin/python3
$ ls -l /usr/bin/python3 /usr/bin/python3 -> /etc/alternatives/python3
$ ls -l /etc/alternatives/python3 /etc/alternatives/python3 -> /usr/bin/python3.6
$ ls -l /usr/bin/python3.6 /usr/bin/python3.6 -> /usr/libexec/platform-python3.6
$ file /usr/libexec/platform-python3.6 /usr/libexec/platform-python3.6: ELF 64-bit LSB shared object, x86-64, version 1
˓→(SYSV), dynamically linked, ...
$ ldd /usr/libexec/platform-python3.6 libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007f9ad79f6000) ...
19.2 Python packages in RHEL
RHEL8 packages: • python3: Python 3.6 • python2 (Module): Python 2.7 See: • Python in RHEL 8 (November 2018) by Petr Viktorin • What, No Python in RHEL 8 Beta? (November 2018) by Langdon White
19.3 Software Collections
Currently supported (last update: 2018-04-23):
Python version Supported RHEL Python27 RHEL 7, RHEL 6 Python34 RHEL 7, RHEL 6 Python35 RHEL 7, RHEL 6 Python36 RHEL 7
No longer supported: • Python33 Software Collections support: https://access.redhat.com/support/policy/updates/rhscl Support shorter than RHEL support. Python27 will likely be supported at least until 2020.
142 Chapter 19. Red Hat and Python CHAPTER 20
Fedora
See also: Python in Fedora.
20.1 Debuginfo
To debug python3-cryptography, install debug symbols: sudo dnf debuginfo-install python3-cryptography
20.2 Download a source RPM sudo yum install yum-utils yumdownloader--source
You may have to enable “Source” repositories: see also “yumdownloader –enablerepo=xxx” option.
20.3 Search a package without updating yum cache yum search-C pattern
20.4 Which package provides the program route?
143 Victor Stinner’s Notes Documentation, Release 1.0
$ rpm -qf $(which route) net-tools-2.0-0.15.20131119git.fc20.x86_64
Or if the package is not installed:
$ yum whatprovides route ... net-tools-2.0-0.15.20131119git.fc20.x86_64 : Basic networking tools ... Nom de fichier: /usr/sbin/route
20.5 Listing the files in a package rpm-ql mongodb-server
20.6 Install dependencies to build the package digikam yum-builddep digikam
20.7 Packages
• https://apps.fedoraproject.org/packages/ • Example with gdb: https://apps.fedoraproject.org/packages/s/gdb • https://koji.fedoraproject.org/koji/ : server where packages are built Install manually a different version, package already installed: rpm -e gdb-8.2.50.20181010-5.fc30.x86_64.rpm
20.7.1 Rebuild a Fedora package
Rebuild a package: Fedora Source RPM. If you get a .src.rpm package, you can rebuild it with: rpmbuild--rebuild wrk-3.1.0-1.fc21.src.rpm
20.7.2 Unpack RPM
Unpack file.rpm in a new dir/ subdirectory: mkdir dir cp file.rpm dir/ cd dir rpm2cpio file.rpm| cpio-idmv
144 Chapter 20. Fedora Victor Stinner’s Notes Documentation, Release 1.0
20.8 sudo: add user to wheel group usermod-aG wheel vstinner
20.9 Python
• Multiple Python interpreters
20.10 Upgrade
Upgrade Fedora 29 to Fedora 30 in command line:
# sudo dnf upgrade --refresh sudo dnf install dnf-plugin-system-upgrade sudo dnf system-upgrade download--refresh--releasever=34--allowerasing # --skip-
˓→broken sudo dnf system-upgrade reboot # at first boot on the new Fedora, fix SELinux labels: sudo fixfiles-B onboot # and reboot https://fedoraproject.org/wiki/DNF_system_upgrade
20.11 ABRT
See also systemd coredumpctl.
20.11.1 ABRT components
• /usr/bin/abrt-applet • /usr/bin/abrt-dump-journal-core run by abrt-journal-core.service • /usr/sbin/abrtd run by abrtd.service • /usr/sbin/abrt-dbus run by DBus activation • /usr/bin/abrt-dump-journal-oops run by abrt-oops.service
20.11.2 Ignore crashes in $HOME
Edit BlackListedPaths in /etc/abrt/abrt-action-save-package-data.conf:
$ sudo vim /etc/abrt/abrt-action-save-package-data.conf BlackListedPaths = (...), /home/vstinner/* where (...) was the existing configuration. Full example:
20.8. sudo: add user to wheel group 145 Victor Stinner’s Notes Documentation, Release 1.0
BlackListedPaths=/usr/share/doc/ *, */example*,/usr/bin/nspluginviewer,/usr/lib */ ˓→firefox/plugin-container,/home/vstinner/ *
20.12 Rawhide and GPG keys
Rawhide: https://fedoraproject.org/wiki/Releases/Rawhide GPG:
$ rpm -qf /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-x86_64 fedora-gpg-keys-31-0.2.noarch
Import Fedora’s GPG key(s) (command comming from https://getfedora.org/security/): curl https://getfedora.org/static/fedora.gpg | gpg –import Last resort: disable gpgcheck in /etc/yum.repos.d/fedora-rawhide.repo (then reenable it).
20.13 List all packages installed on the system
Using dnf: dnf history userinstalled
Using rpm: rpm-qa
20.14 dnf: file . . . of xxx conflicts with file from package yyy
Attempt dnf remove xxx. If it’s really not possible: dnf download yyy sudo rpm-ihv--force
20.15 RPM specfile
• %bcond_without rpmwheels enables rpmwheels feature (true) • %bcond_with rpmwheels disables rpmwheels feature (false)
146 Chapter 20. Fedora CHAPTER 21
GNOME and Wayland
My articles: • https://vstinner.github.io/debug-hybrid-graphics-issues-linux.html • https://vstinner.github.io/graphics-bugs-firefox-gnome.html
21.1 Fedora IRC channel
Join #fedora-desktop on irc.gnome.org.
21.2 GNOME
GNOME desktop made of multiple components: • Mutter: compositor supportting Xorg and Wayland, use Clutter library • libinput: library that provides a full input stack for display servers and other applications that need to han- dle input devices provided by the kernel. Try sudo libinput list-devices and sudo libinput debug-events commands. • GJS: Javascript Bindings for Gnome, use Mozilla SpiderMonkey. Provide “gjs” program which can run Javascript in the command line. • Xwayland: X server using Wayland compositor • GNOME Shell is the GNOME desktop environment. It is written in C and JavaScript as a plugin for Mutter. It’s the gnome-shell program. Main features: – Handle inputs using libinput library – Wayland compositor using Mutter (as a library) – Run shell extensions written in Javascript using GJS (as a library)
147 Victor Stinner’s Notes Documentation, Release 1.0
Mutter spawns Xwayland and sets the DISPLAY environment variable which is inherited by child processes: all graphical applications started in GNOME.
21.3 Wayland
This section is unrelated to Hybrid Graphics, but useful to debug graphics issues.
21.3.1 Do I use Wayland?
Is “type wayland” found in the loginctl session status?
$ loginctl session-status|grep Service: Service: gdm-password; type wayland; class user
Is WAYLAND_DISPLAY environment variable set?
$ env|grep -E '^(XDG_SESSION_TYPE|WAYLAND_DISPLAY|DISPLAY)' XDG_SESSION_TYPE=wayland WAYLAND_DISPLAY=wayland-0 DISPLAY=:0
Is Xwayland running?
$ ps ax|grep Xwayland 1956 tty2 Sl+ 6:38 /usr/bin/Xwayland :0 ...
In GNOME Shell, Mutter spawns Xwayland and sets the DISPLAY environment variable which is inherited by child processes: all graphical applications started in GNOME.
21.3.2 Is this application using Wayland or Xorg? xprop or xwininfo program can be in Wayland to check if an application is using Xorg or Wayland: the mouse cursor becomes a cross only and only if the application is used Xorg (X11 API). xlsclients command lists programs using XWayland.
21.3.3 Opt-in for Wayland
To opt-in for Wayland support in Firefox and Thunderbird, set MOZ_ENABLE_WAYLAND=1 environment variable. For example, I put the following line into /etc/environment to run Firefox with Wayland:
MOZ_ENABLE_WAYLAND=1
When a Wayland compositor is running, Gtk applications prefer Wayland by default. In that case, you can opt-in for X11 by seting GDK_BACKEND=x11 environment variable.
21.4 Wayland and Xorg
Debug: https://fedoraproject.org/wiki/How_to_debug_Wayland_problems
148 Chapter 21. GNOME and Wayland Victor Stinner’s Notes Documentation, Release 1.0
See also: https://fedoraproject.org/wiki/How_to_debug_Firefox_problems Environment to opt-in for Wayland support: export GDK_BACKEND=wayland
To run Firefow with Firefox, edit /etc/environment to add the line:
MOZ_ENABLE_WAYLAND=1
Get GPUs:
$ lspci|grep VGA 00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GM107GLM [Quadro M1000M] (rev
˓→a2)
Get OpenGL GPU:
$ glxinfo|grep -E 'Device|rendering' direct rendering: Yes Device: Mesa DRI Intel(R) HD Graphics 530 (Skylake GT2) (0x191b)
Get screen resolution:
$ xrandr Screen 0: minimum 320 x 200, current 3840 x 1080, maximum 8192 x 8192 XWAYLAND0 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 510mm x
˓→290mm 1920x1080 59.96*+ XWAYLAND1 connected 1920x1080+1920+0 (normal left inverted right x axis y axis) 480mm
˓→x 270mm 1920x1080 59.96*+
Get screen DPI (96x96 in this example):
$ xdpyinfo | grep -B 2 resolution screen #0: dimensions: 3840x1080 pixels (1016x286 millimeters) resolution: 96x96 dots per inch
Check if an application is using Xorg or Wayland in Wayland: run xprop, the mouse cursor becomes a cross only for Xorg appplications.
21.4.1 Hybrid Graphics (2 GPUs)
Disable Nouveau driver: sudo grubby--update-kernel=ALL--args="modprobe.blacklist=nouveau"
Fedora 30, add an argument to all GRUB kernel configurations: sudo grubby--update-kernel=ALL--args="xdg.force_integrated=0"
Disable switcheroo-control (don’t run it anymore at startup):
21.4. Wayland and Xorg 149 Victor Stinner’s Notes Documentation, Release 1.0
sudo systemctl stop switcheroo-control.service sudo systemctl disable switcheroo-control.service
My Lenovo P50 has 2 GPU, one slow integrated Intel GPU and one fast Nvidia GPU. There is a switcheroo-control D-Bus service to check if the system has 2 GPUs. Linux kernel vgaswitcheroo:
$ sudo cat /sys/kernel/debug/vgaswitcheroo/switch 0:IGD:+:Pwr:0000:00:02.0 1:DIS: :DynPwr:0000:01:00.0
• IGD: Integrated Graphics Device • DIS: DIScrete graphics device • “+”: active card Links: • https://www.kernel.org/doc/html/latest/gpu/vga-switcheroo.html • https://help.ubuntu.com/community/HybridGraphics DBus: gdbus introspect--system--dest net.hadess.SwitcherooControl--object-path/net/
˓→hadess/SwitcherooControl ... interface net.hadess.SwitcherooControl { ... properties: readonly b HasDualGpu= true; };
See bumblebee. Launch an application with Nvidia GPU from a terminal:
DRI_PRIME=1 firefox
Firefox: • Go to about:support and search for the Graphics section • WebGL https://webglreport.com/
Unmasked Vendor: nouveau Unmasked Renderer: NV117
21.5 Xorg BadWindow issue
Set GDK_SYNCHRONIZE environment variable to debug such issue:
The program'gnome-shell' received an X Window System error. This probably reflects a bug in the program. The error was'BadWindow (invalid Window parameter)'. (Details: serial 352312 error_code3 request_code 18 (core protocol) minor_code0) (continues on next page)
150 Chapter 21. GNOME and Wayland Victor Stinner’s Notes Documentation, Release 1.0
(continued from previous page) (Note to programmers: normally, X errors are reported asynchronously; that is, you will receive the error a while after causing it. To debug your program, run it with the GDK_SYNCHRONIZE environment variable to change this behavior. You can then get a meaningful backtrace from your debugger if you break on the gdk_x_error() function.)
• https://gitlab.gnome.org/GNOME/gnome-shell/issues/760 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/1230 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/661 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/627 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/496 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/375 • https://gitlab.gnome.org/GNOME/gnome-shell/issues/213 • Ubuntu: https://bugs.launchpad.net/ubuntu/+source/gnome-shell/+bug/1821427 • Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=712612
21.6 My GPU bugs on Fedora
My bugs: • 2020-01-28, Intel IGP: i915 0000:00:02.0: GPU HANG: ecode 9:1:0x00000000, hang on rcs0 • 2020-01-23: d_alloc: list_add corruption. next->prev should be prev (ffff930b5d4b6ca0), but was 0000000000000000. (next=ffff930beff5b690) (Intel IGP?) My laptop Lenovo P50 has two GPUs: • Integrated Graphics Device: Intel IGP (Intel HD Graphics 530) • Discrete Graphics Device: NVIDIA GPU (NVIDIA Quadro M1000M) See Debug Hybrid Graphics issues on Linux.
21.7 Wayland copy/paste in command line
Commands:
$ wl-copy bla $ wl-paste bla
Fedora: dnf install wl-clipboard.
21.6. My GPU bugs on Fedora 151 Victor Stinner’s Notes Documentation, Release 1.0
152 Chapter 21. GNOME and Wayland CHAPTER 22
vim for developer
22.1 vim command line
• vim file.c +10: Open file.c directly at line 10 (doesn’t work with vi, only vim) • git diff | vim -: View the output of a command in vim (git diff command in this example)
22.2 Navigate
• gg: go to file start • GG: go to file end • 0: go to line start • $: go to line end • :10: go to line number 10 • [[, ]]: jump to previous/next of the class or function • gf: Go To File, open the file with the file name under the cursor
22.3 Buffers
• :vs: Vertical Split • :split: Horizontal Split • :%bd: close all buffers
153 Victor Stinner’s Notes Documentation, Release 1.0
22.4 Programming
• Go to a function definition using ctags: – Run “:ctags -R” to create an index of symbols (classes, functions, variables) – Put the cursor on a symbol: CTRL+[ goes to the symbol defintion
22.5 Misc
• Open a file from encoding cp850: :e ++enc=cp850 document.txt
22.6 Windows
• gvim configuration files for user vstinner: – C:\Users\vstinner\_vimrc: vim configuration, same than ~/.vimrc on Unix – C:\Users\vstinner\_gvimrc: gvim configuration, same than ~/.gvimrc on Unix Create a symbol link for configuration files using python:
C:\Users\vstinner>python Python 3.8.0 >>> import os >>> os.symlink(r'\vstinner\misc\conf\vimrc','_vimrc') >>> os.symlink(r'\vstinner\misc\conf\gvimrc','_gvimrc')
Create a symbol links: • from C:\vstinner\misc\conf\vimrc to C:\Users\vstinner\_vimrc • from C:\vstinner\misc\conf\gvimrc to C:\Users\vstinner\_gvimrc
154 Chapter 22. vim for developer CHAPTER 23
Assembly Intel x86
23.1 Instructions
• CDQE: RAX ← sign-extend of EAX. • MOVSXD rcx,ecx: Move doubleword to quadword with sign-extension. • MOVABS rcx,
23.2 Registers
64b register Lower 32b Lower 16b Lower 8b rax eax ax al r9 r9d r9w r9b
23.3 x86-64 ABI
System V AMD64 ABI (Linux): • First 6 arguments placed onto registers: RDI, RSI, RDX, RCX, R8, R9. Microsoft x64 calling convention: • First 4 arguments placed onto registers: RCX, RDX, R8, R9.
155 Victor Stinner’s Notes Documentation, Release 1.0
156 Chapter 23. Assembly Intel x86 CHAPTER 24
See also
• Python Development Documentation • Victor Stinner blog 3 • Tutorial to contribute to the CPython project’s documentation!
157 Victor Stinner’s Notes Documentation, Release 1.0
158 Chapter 24. See also CHAPTER 25
Indices and tables
• genindex • modindex • search “All problems in computer science can be solved by another level of indirection. . . except for the problem of too many layers of indirection.” – joke based on a famous aphorism of David Wheeler “There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors.”
159 Victor Stinner’s Notes Documentation, Release 1.0
160 Chapter 25. Indices and tables Index
P Python Enhancement Proposals PEP 400, 22 PEP 410, 22 PEP 416, 22 PEP 418, 21 PEP 433, 22 PEP 445, 21 PEP 446, 21 PEP 454, 21 PEP 475, 21 PEP 490, 22 PEP 509, 21 PEP 510, 22 PEP 511, 22 PEP 524, 21 PEP 540, 21 PEP 545, 21 PEP 546, 22 PEP 564, 21 PEP 587, 21 PEP 606, 22 PEP 608, 22 PEP 620, 21 PEP 8015, 22 Python Enhancement Proposals PEP 460, 22 PEP 461, 22 PEP 471, 22
161