Squeezing Water from Stone:

KornShell in 2019

Siteshwar Vashisht @SiteshwarV main() { printf(&["\021%six\012\0"],(unix)["have"]+"fun"-0x60);}

● Winner of IOCCC (The International Obfuscated Code Contest) one line entry in 1987. ● Termed as best one liner ever received. ● Written by David Korn. What this talk is about ?

● It’s about AT&T KornShell (ksh, not mksh or pdksh). ● Brief status update from new upstream maintainers (Kurtis Rader and Siteshwar Vashisht). ● No deep dive in the source code. ● It is slightly extended version of my talk from FOSDEM 2019 and All Systems Go! 2019. /whois situ

● Siteshwar Vashisht. ● Based in Brno, Czech Republic. ● Maintainer of and ksh at . ● Previously involved with and Sailfish OS. ● Current upstream maintainer of ksh. What is KornShell ?

● Direct descendant of . ● Initial development based on original bourne shell. ● Roots back to late 70’s and early 80’s. Spot the bug!

#!/bin/bash echo $'hello world' | while read foo do

bar="$foo" done echo "bar: " "$bar"

$bar is not available outside loop. Bash forks last command in a new process and $bar is lost. Advantages

● Superior language specification (Support for Discipline functions, namespaces, compound data , enums etc.). ● Better POSIX compliance. ● Fastest POSIX shell. Show me the numbers!

$ cat .sh

for ((i=0;i<10000;i++)) do

foo=$(echo "bar baz")

done

$ time bash test.sh $ time zsh test.sh $ time ksh test.sh real 0m2.977s real 0m2.433s real 0m0.078s user 0m2.032s user 0m1.570s user 0m0.062s sys 0m1.167s sys 0m1.233s sys 0m0.015s A 20 year old bug State of ksh in 2017

● Previous developers have left. ● Codebase has origins going back to the 80s. ● Old Tooling. ● No revision control history. ● Few meaningful comments in the source code. ● Spaghetti code. ● Bad test coverage. ● Lots of old bugs. ● And ... How do you maintain the unmaintainable ? Goals

● Create a modern but backward compatible version of ksh. ● Make it easier to contribute. ● Improve tooling (build system, CI etc.). ● Improve POSIX compliance. ● Better test coverage.

Bring this codebase into the 21’st century. One of my most productive days was throwing away 1,000 lines of code - Ken Thompson

We threw away more than 500,000 lines of code. Lines of code (ksh93v-)

Lines of code (current) What did we drop ?

● Support for non-POSIX operating systems. ● Reimplementation of POSIX functions. ● Entire subsystems like the AST vmalloc and locale. ● Lots of code that was not used by ksh. ● All external commands (e.g., grep) other than ksh were dropped.

The project is now focused solely on ksh rather than providing replacements for commonly used commands found in BSD, , and SysV. Build System

● Nmake and iffe (If Feature Exists). ● Build system was complicated. ● Builds were slow. ● We threw away old build system and switched to Meson. ● Build time reduced by a factor of 35.

Old Build System New Build System

real 5m15.922s real 0m9.239s user 4m12.355s user 0m11.290s sys 1m18.843s sys 0m4.812s Refactor Code Refactor Code Here Be Dragons

It checks if either NV_LJUST or NV_RJUST attribute is set, but not both. Here Be Dragons When ksh met gcc-9.0 Call for Developers

If you like to work on crazy problems, try refactoring ksh code.

https://github.com/att/ast “It was at first a small effort with about ten people, but it has grown gradually over time to 25 people. Most are researchers who create prototype Software.”

A quote from book “Practical Reusable Unix Software” Computer Scientist vs Computer Programmer

A computer scientist is someone who looks on one side of the street while crossing one way street.A programmer always looks on both sides. Better CI Better CI Test Coverage Test Coverage Test Coverage

All Red Hat internal tests are executed on every upstream build. Call for Testers

You can be a crazy tester. Coverity Defect Rate Graph It’s simple to try It’s simple to try Call for Packagers

You don’t have to be a crazy packager.

$ meson build && ninja -C build

ksh-2020.0.0-beta1 was released in August 2019. If you don’t see it in your favorite distro, please rebase to newest version. Guidelines for packagers can be found at https://github.com/att/ast/wiki/Guidelines-for-Packagers Package Maintainers

Boyuan Yang - Debian

Cy Schubert - FreeBSD

Eli Schwartz - Archlinux

Mike Gilbert - Gentoo

Ryan Schmidt - MacPorts

Stig-Ørjan Smelror - Mageia Is it too late to make a change ? Change is Good. Better Late Than Never!

Thank you!