DOCSLIB.ORG

Squeezing Water from Stone: Kornshell in 2019

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Squeezing Water from Stone: Kornshell in 2019 Squeezing Water from Stone: KornShell in 2019 Siteshwar Vashisht @SiteshwarV main() { printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);} ● Winner of IOCCC (The International Obfuscated C Code Contest) one line entry in 1987. ● Termed as best one liner ever received. ● Written by David Korn. What this talk is about ? ● It’s about AT&T KornShell (ksh, not mksh or pdksh). ● Brief status update from new upstream maintainers (Kurtis Rader and Siteshwar Vashisht). ● No deep dive in the source code. ● It is slightly extended version of my talk from FOSDEM 2019. /whois situ ● Siteshwar Vashisht. ● Based in Brno, Czech Republic. ● Maintainer of bash and ksh at Red Hat. ● Previously involved with fish shell and Sailfish OS. ● Current upstream maintainer of ksh. What is KornShell ? ● Direct descendant of Bourne shell. ● Initial development based on original bourne shell. ● Roots back to late 70’s and early 80’s. Spot the bug! #!/bin/bash echo $'hello world' | while read foo do bar="$foo" done echo "bar: " "$bar" $bar is not available outside loop. Bash forks last command in a new process and $bar is lost. Advantages ● Superior language specification (Support for Discipline functions, namespaces, compound data type, enums etc.). ● Better POSIX compliance. ● Fastest POSIX shell. Show me the numbers! $ cat test.sh for ((i=0;i<10000;i++)) do foo=$(echo "bar baz") done $ time bash test.sh $ time zsh test.sh $ time ksh test.sh real 0m2.977s real 0m2.433s real 0m0.078s user 0m2.032s user 0m1.570s user 0m0.062s sys 0m1.167s sys 0m1.233s sys 0m0.015s A 20 year old bug State of ksh in 2017 ● Previous developers have left. ● Codebase has origins going back to the 80s. ● Old Tooling. ● No revision control history. ● Few meaningful comments in the source code. ● Spaghetti code. ● Bad test coverage. ● Lots of old bugs. ● And ... How do you maintain the unmaintainable ? Goals ● Create a modern but backward compatible version of ksh. ● Make it easier to contribute. ● Improve tooling (build system, CI etc.). ● Improve POSIX compliance. ● Better test coverage. Bring this codebase into the 21’st century. One of my most productive days was throwing away 1,000 lines of code - Ken Thompson We threw away more than 500,000 lines of code. Lines of code (ksh93v-) Lines of code (current) What did we drop ? ● Support for non-POSIX operating systems. ● Reimplementation of POSIX functions. ● Entire subsystems like the AST vmalloc and locale. ● Lots of code that was not used by ksh. ● All external commands (e.g., grep) other than ksh were dropped. The project is now focused solely on ksh rather than providing replacements for commonly used commands found in BSD, Linux, and SysV. Build System ● Nmake and iffe (If Feature Exists). ● Build system was complicated. ● Builds were slow. ● We threw away old build system and switched to Meson. ● Build time reduced by a factor of 35. Old Build System New Build System real 5m15.922s real 0m9.239s user 4m12.355s user 0m11.290s sys 1m18.843s sys 0m4.812s Refactor Code Refactor Code Here Be Dragons It checks if either NV_LJUST or NV_RJUST attribute is set, but not both. Here Be Dragons When ksh met gcc-9.0 Call for Developers If you like to work on crazy problems, try refactoring ksh code. https://github.com/att/ast “It was at first a small effort with about ten people, but it has grown gradually over time to 25 people. Most are researchers who create prototype Software.” A quote from book “Practical Reusable Unix Software” Computer Scientist vs Computer Programmer A computer scientist is someone who looks on one side of the street while crossing one way street.A programmer always looks on both sides. Better CI Better CI Test Coverage Test Coverage Test Coverage All Red Hat internal tests are executed on every upstream build. Call for Testers You can be a crazy tester. Coverity Defect Rate Graph It’s simple to try It’s simple to try Call for Packagers You don’t have to be a crazy packager. $ meson build && ninja -C build ksh-2020.0.0-beta1 was released in August 2019. If you don’t see it in your favorite distro, please rebase to newest version. Guidelines for packagers can be found at https://github.com/att/ast/wiki/Guidelines-for-Packagers Package Maintainers Boyuan Yang - Debian Cy Schubert - FreeBSD Eli Schwartz - Archlinux Mike Gilbert - Gentoo Ryan Schmidt - MacPorts Stig-Ørjan Smelror - Mageia Is it too late to make a change ? Change is Good. Better Late Than Never! Thank you!.
Load more

Recommended publications
  • Shell Scripting with Bash
    Introduction to Shell Scripting with Bash Charles Jahnke Research Computing Services Information Services & Technology Topics for Today ● Introductions ● Basic Terminology ● How to get help ● Command-line vs. Scripting ● Variables ● Handling Arguments ● Standard I/O, Pipes, and Redirection ● Control Structures (loops and If statements) ● SCC Job Submission Example Research Computing Services Research Computing Services (RCS) A group within Information Services & Technology at Boston University provides computing, storage, and visualization resources and services to support research that has specialized or highly intensive computation, storage, bandwidth, or graphics requirements. Three Primary Services: ● Research Computation ● Research Visualization ● Research Consulting and Training Breadth of Research on the Shared Computing Cluster (SCC) Me ● Research Facilitator and Administrator ● Background in biomedical engineering, bioinformatics, and IT systems ● Offices on both CRC and BUMC ○ Most of our staff on the Charles River Campus, some dedicated to BUMC ● Contact: [email protected] You ● Who has experience programming? ● Using Linux? ● Using the Shared Computing Cluster (SCC)? Basic Terminology The Command-line The line on which commands are typed and passed to the shell. Username Hostname Current Directory [username@scc1 ~]$ Prompt Command Line (input) The Shell ● The interface between the user and the operating system ● Program that interprets and executes input ● Provides: ○ Built-in commands ○ Programming control structures ○ Environment
    [Show full text]
  • Bash Guide for Beginners
    Bash Guide for Beginners Machtelt Garrels Garrels BVBA <tille wants no spam _at_ garrels dot be> Version 1.11 Last updated 20081227 Edition Bash Guide for Beginners Table of Contents Introduction.........................................................................................................................................................1 1. Why this guide?...................................................................................................................................1 2. Who should read this book?.................................................................................................................1 3. New versions, translations and availability.........................................................................................2 4. Revision History..................................................................................................................................2 5. Contributions.......................................................................................................................................3 6. Feedback..............................................................................................................................................3 7. Copyright information.........................................................................................................................3 8. What do you need?...............................................................................................................................4 9. Conventions used in this
    [Show full text]
  • Bash Tutorial
    Bash Shell Lecturer: Prof. Andrzej (AJ) Bieszczad Email: [email protected] Phone: 818-677-4954 Bash Shell The shell of Linux • Linux has a variety of different shells: – Bourne shell (sh), C shell (csh), Korn shell (ksh), TC shell (tcsh), Bour ne Again shell (bash). • Certainly the most popular shell is “bash”. Bash is an sh- compatible shell that incorporates useful features from the Korn shell (ksh) and C shell (csh). • It is intended to conform to the IEEE POSIX P1003.2/ISO 9945.2 Shell and Tools standard. • It offers functional improvements over sh for both programming and interactive use. Bash Shell Programming or Scripting ? • bash is not only an excellent command line shell, but a scripting language in itself. Shell scripting allows us to use the shell's abilities and to automate a lot of tasks that would otherwise require a lot of commands. • Difference between programming and scripting languages: – Programming languages are generally a lot more powerful and a lot faster than scriptin g languages. Programming languages generally start from source code and are compil ed into an executable. This executable is not easily ported into different operating syste ms. – A scripting language also starts from source code, but is not compiled into an executabl e. Rather, an interpreter reads the instructions in the source file and executes each inst ruction. Interpreted programs are generally slower than compiled programs. The main a dvantage is that you can easily port the source file to any operating system. bash is a s cripting language. Other examples of scripting languages are Perl, Lisp, and Tcl.
    [Show full text]
  • Contextual Shell History Student: Bc
    ASSIGNMENT OF MASTER’S THESIS Title: Contextual Shell History Student: Bc. Šimon Let Supervisor: Ing. Lukáš Bařinka Study Programme: Informatics Study Branch: System Programming Department: Department of Theoretical Computer Science Validity: Until the end of summer semester 2020/21 Instructions Analyze shell history features and existing shell history tools. Examine how users interact with shell and shell history. Design a shell history solution based on known and documented problems. Leverage the context of commands entered by the user. Use principles of user-centered design. Implement a shell history tool that is capable of recording shell history with command context, offers history entries to the user based on the current context and provides means to search the recorded history. Suggest meaningful metrics (e.g. based on distance in history or number of characters needed to get searched command from history). Based on chosen metrics evaluate the usefulness of the recommended history entries. Perform user testing or testing based on real user data to find out the user experience of the solution. References Will be provided by the supervisor. doc. Ing. Jan Janoušek, Ph.D. doc. RNDr. Ing. Marcel Jiřina, Ph.D. Head of Department Dean Prague January 17, 2020 Master’s thesis Contextual Shell History Bc. Simonˇ Let Department of theoretical computer science Supervisor: Ing. Luk´aˇsBaˇrinka May 28, 2020 Acknowledgements I would like to thank my supervisor Ing. Luk´aˇsBaˇrinkafor his guidance and advice. His insights and expertise shaped this work significantly. Next, my thanks go to my colleagues and friends for their valuable feed- back. They helped me spot my personal biases and assumptions rooted in my personal shell workflows.
    [Show full text]
  • Shell Scripting, Globs, and Ranges
    SHELL SCRIPTING AND CUSTOMIZATION GPI F20 · Emmanuel & Sayan PET TAX Carole Bashkin! Ran her program, bashed it Can’t convince her that bash isn’t awesome Pipin’, globbin’, multitaskin’ What’s happen? Carole Bashkin! WHAT IS A SHELL? ● A shell is the outermost layer that allows a user to interact with the operating system. ● For a programmer, it is the program that powers your terminal. ● We will focus on Unix shells, command-line interpreters for Unix-like operating systems. ● They come in many flavors. ○ zsh, bash, fish, etc. COMMAND CENTER ● Show contents of a file cat path ● Show contents of a directory ls ● Change directories cd path ● Remove a file rm src ● Move/rename a file mv src dst ● Copy a file cp src new_name ● Execute a binary ./name ● Print something echo “Hello, world!” OVER, OVER, AND OVER AGAIN ● Sometimes, you want to run the same set of commands multiple times. ○ Compiling and testing your code ○ Renaming a bunch of files ○ Initializing your git repos with a .gitignore ○ Archiving your new favorite xkcd comics ○ Setting up your dev environment on a new machine ● Shell scripts allow you to run a sequence of shell commands from a file. WRITE YOUR OWN SHELL SCRIPT ● Each script begins with a shebang followed by the commands you want to execute. ○ #!/bin/zsh ○ #!/bin/bash ○ #!/usr/bin/env python3 ● Add executable permissions to your file. ○ chmod +x script.sh ○ This changes the mode of the file to an executable. ● Run the script! ○ ./script.sh FUNCTIONS ARE SHELL SCRIPTS ● Apart from interfacing with the operating system, they support other programming constructs as well.
    [Show full text]
  • Lmod Documentation Release 8.5.16
    Lmod Documentation Release 8.5.16 Robert McLay Sep 24, 2021 Contents 1 PURPOSE 1 2 OVERVIEW 3 3 Lmod Web Sites 5 4 Introduction to Lmod 7 5 Installing Lmod 27 6 Advanced Topics 55 7 Topics yet to be written 119 8 Indices and tables 121 i ii CHAPTER 1 PURPOSE Lmod is a Lua based module system that easily handles the MODULEPATH Hierarchical problem. Environment Modules provide a convenient way to dynamically change the users’ environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable. Modulefiles for Library packages provide environment variables that specify where the library and header files can be found. 1 Lmod Documentation, Release 8.5.16 2 Chapter 1. PURPOSE CHAPTER 2 OVERVIEW This guide is written to explain what Environment Modules are and why they are very useful for both users and system administrators. Lmod is an implementation of Environment Modules, much of what is said here is true for any environment modules system but there are many features which are unique to Lmod. Environment Modules provide a convenient way to dynamically change the users’ environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable. A modulefile contains the necessary information to allow a user to run a particular application or provide access to a particular library. All of this can be done dynamically without logging out and back in. Modulefiles for applications modify the user’s path to make access easy. Modulefiles for Library packages provide environment variables that specify where the library and header files can be found.
    [Show full text]
  • Shell Scripting
    Running Applications on HPC Systems (Using the Shell) Michael Griffiths and Norbert Gyenge Corporate Information and Computing Services The University of Sheffield www.sheffield.ac.uk/cics/research SESSION OUTLINE! 1. Software on the HPC Systems 2. Why use shell programs 3. Basics of shell programming 4. Using variables and parameters 5. User Input during shell script execution 6. Aliases SOFTWARE ON THE HPC SYSTEMS For information on software check the link http://www.sheffield.ac.uk/cics/research/hpc/ Ganglia Portland, GNU,Intel CentOS 64bit Sun of Grid Linux Engine Intel Haswell OpenMPI E5-2630-v3 RUNNING PROGRAMS ON HPC • The login node is the gateway to the cluster of worker nodes and the only one where direct logging in is allowed. • Login node’s main purpose is to allow access to the worker nodes but NOT to run cpu intensive programs. • All cpu-intensive computations must be performed on the worker nodes. This is achieved by the srun --pty bash -i command for the interactive jobs and sbatch command for the batch jobs.! • Once you log into the HPC system, taking advantage of the power of a worker node for interactive work is done simply by typing srun --pty bash -i and working in the new shell window that is opened. This what appears to be a trivial task has would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster. • The next set of slides assume that you are already working on one of the worker nodes.
    [Show full text]
  • La Kornshell: Lenguaje De Programación Y Comando
    La KornShell: Lenguaje de Programación y Comando Una guía para programadores de Shell Scripts de Unix Jesús Alberto Vidal Cortés http://inicia.es/de/chube [email protected] www.kornshell.com Madrid, Agosto de 2001 Índice PARTE I: INTRODUCCIÓN......................................................................................................... 11 1. SOBRE EL LENGUAJE KORNSHELL ........................................................................................... 12 ¿Qué es una Shell?.............................................................................................................. 12 Beneficios del uso de ksh..................................................................................................... 13 Mejoras como Lenguaje de Comandos ................................................................................................................ 13 Mejoras como Lenguaje de Programación........................................................................................................... 14 Versiones a las que se aplica este manual............................................................................. 15 ¿Cómo obtener ksh?............................................................................................................ 16 2. NOTACIÓN UTILIZADA EN ESTE MANUAL..................................................................................... 18 General ............................................................................................................................... 18 Teclas sobre
    [Show full text]
  • Squeezing Water from Stone: Kornshell in 2019
    Squeezing Water from Stone: KornShell in 2019 Siteshwar Vashisht @SiteshwarV main() { printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);} ● Winner of IOCCC (The International Obfuscated C Code Contest) one line entry in 1987. ● Termed as best one liner ever received. ● Written by David Korn. What this talk is about ? ● It’s about AT&T KornShell (ksh, not mksh or pdksh). ● Brief status update from new upstream maintainers (Kurtis Rader and Siteshwar Vashisht). ● No deep dive in the source code. ● It is slightly extended version of my talk from FOSDEM 2019 and All Systems Go! 2019. /whois situ ● Siteshwar Vashisht. ● Based in Brno, Czech Republic. ● Maintainer of bash and ksh at Red Hat. ● Previously involved with fish shell and Sailfish OS. ● Current upstream maintainer of ksh. What is KornShell ? ● Direct descendant of Bourne shell. ● Initial development based on original bourne shell. ● Roots back to late 70’s and early 80’s. Spot the bug! #!/bin/bash echo $'hello world' | while read foo do bar="$foo" done echo "bar: " "$bar" $bar is not available outside loop. Bash forks last command in a new process and $bar is lost. Advantages ● Superior language specification (Support for Discipline functions, namespaces, compound data type, enums etc.). ● Better POSIX compliance. ● Fastest POSIX shell. Show me the numbers! $ cat test.sh for ((i=0;i<10000;i++)) do foo=$(echo "bar baz") done $ time bash test.sh $ time zsh test.sh $ time ksh test.sh real 0m2.977s real 0m2.433s real 0m0.078s user 0m2.032s user 0m1.570s user 0m0.062s sys 0m1.167s sys 0m1.233s sys 0m0.015s A 20 year old bug State of ksh in 2017 ● Previous developers have left.
    [Show full text]
  • Common Desktop Environment: Desktop Kornshell User's Guide
    Common Desktop Environment: Desktop KornShell User’s Guide Sun Microsystems, Inc. 901 San Antonio Road Palo Alto, CA 94303 U.S.A. Part No: 805-3917–10 October 1998 Copyright 1998 Sun Microsystems, Inc. 901 San Antonio Road, Palo Alto, California 94303-4900 U.S.A. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, SunDocs, Java, the Java Coffee Cup logo, and Solaris are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and SunTM Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements.
    [Show full text]
  • Squeezing Water from Stone: Kornshell in 2019
    Squeezing Water from Stone: KornShell in 2019 main() { printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);} ●Winner of IOCCC (The International Obfuscated C Code Contest) one line entry in 1987. ●Termed as best one liner ever received. ●Written by David Korn. What this talk is about ? ●It’s about AT&T KornShell (and not mksh/pdksh). ●Brief status update from new upstream maintainers (Kurtis Rader and Siteshwar Vashisht). ●No deep dive in the source code. /whois situ ●Siteshwar Vashisht. ●Based in Brno, Czech Republic. ●Maintainer of bash and ksh at Red Hat. ●Previously involved with fish shell and Sailfish OS. ●Current upstream maintainer of ksh. What is KornShell ? ●Direct descendant of Bourne shell. ●Initial development based on original bourne shell. ●Roots back to late 70s/early 80s. Advantages ●Superior language specification (Support for Discipline functions, namespaces, compound data type, enums etc.). ●Better POSIX compliance. ●Fastest POSIX shell. A 20 year old bug State of ksh in 2017 ●Previous developers have left. ●Codebase has origins going back to the 80s. ●Old Tooling. ●No revision control history. ●Almost no comments in the source code. ●Spaghetti code. ●Bad test coverage. ●Lots of old bugs. ●And ... How to maintain the unmaintainable ? One of my most productive days was throwing away 1,000 lines of code - Ken Thompson We threw away more than 500,000 lines of code. Lines of code (ksh93v-) Lines of code (current) What did we drop ? ●Support for non-POSIX operating systems. ●Reimplementation of POSIX functions. ●Entire subsystems like the AST vmalloc and locale. ●Lots of code that was not used by ksh.
    [Show full text]
  • Case Statement in Shell Script
    Case Statement In Shell Script cercarianHeated Ivan and repossess unglues excursively very assertively while whilecariogenic Vite remains Roddie rippingdouble-parks and freehold. and immaterialized. Flitting or lascivious, Keefe never sleepwalk any geomagnetists! Abram is Advanced Bash Scripting Guide devconnected. In law pass, the code is actually executed. Thats it matches any nesting level other programming language keywords are very important to case statement. Pimp My Bash Script. Then yes possible pattern was provided, in quotes if it contains whitespace characters, followed by living close parenthesis. However, customs are some limitations to case statements. You really clean to specify these. If you word matches any cost then instance block of code belonging to that hello is executed. The beard form our control statement that allows Shell or execute statements in more controlled way besides making decision between two choices. Linux Bash Case Statement with Examples 1206201 by smail Baydan Bash provides an interactive shell and programming environment after its users. Less code one variable and bash is used for and csh do not go to the previous patterns and end them in without any test most of. So what the Bash functions? Case with shell Scripting case in conditional statement how to. Write Script using case statement to perform basic math. Remove the shells just echo the statements in general purpose, so if file at a software developer documentation for and referenced the quoted to. If statement in shell script that the shells are cases, the brackets are used for the beginning of. But what shirt you needed to wicked the value chart the variable as testament of a larger string? 'case' Statement The case statement executes any one troop of commands based on primary outcome however a dental match we have a variable that.
    [Show full text]