IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Going Barefoot with UNIX for the DB2 DBA Robert Mala, Database Administrator Session Code: 1089 Date and time: 13th September 2012, 10.30am to 11.30am Platform: DB2 for LUW Abstract: Using UNIX can be bewildering until you have grasped the basic concepts, capabilities and features. DBAs with a z/OS background find UNIX to be arcane, esoteric, and pernickety. Getting the UNIX “mindset” takes effort, but once obtained UNIX becomes a capable, powerful and productive environment in which to work. Stepping through tailored examples using DB2 LUW, this presentation is for the DB2 DBA wanting to go barefoot with UNIX. Objectives: - Introduce UNIX concepts, capabilities and features. - Understand and cultivate a UNIX "mindset". - Provide tailored examples using DB2 LUW with UNIX. - Getting the most from the interactive Korn Shell. Speaker Biography: Robert has worked with DB2 and SQL for over 20 years and has worked in a variety of database centric roles for organisations in Australia and Europe. Robert is IBM Certified for DB2 UDB Database Administration for LUW & z/OS, and DB2 UDB Family Application Development. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 1 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style Agenda • UNIX Software Tools Philosophy • Simple Commands • Compound Commands • awk • Secure Shell • Interactive Korn Shell • GNU screen Regardless of the platform or product, the activities of a Database Administrator are broadly the same. From checking house keeping has completed successfully to promoting changes, deploying stored procedures to tuning SQL. The working practices and methods may vary but the basic activities of a Database Administrator remain the same. GUIs can give a large productivity boost for certain types of activities. They can also be very confining. GUIs restrict you to what the developers of the GUI think is needed. This can result in having to do activities in a less than optimal way. The tooling can also leave large gaps in functionality leaving you to find alternatives. GUIs shield you from the underlying commands and insulate you from server side information which at times is critical to doing the job effectively. If you are responsible for looking after many instances of db2 over many servers, GUIs can be unwieldy when trying to apply changes globally. CLIs give the freedom to do just about anything. That freedom comes at a cost of a steep learning curve. Scripts can be developed for common activities so that you need not reinvent the wheel each time. However the command line can be a harsh and unforgiving place. It requires a good level of typing skills and time to build the knowledge and skills to be able to drive the CLI proficiently. Driving the CLI via UNIX is a potent mix. It gives the freedom and flexibility of the CLI coupled with an environment that can super charge your productivity. Going barefoot with Unix for the DB2 DBA requires study and practice to establish practices and finger habits. Which once established will enable you to develop command line scripts that help you do you job quickly and efficiently. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 2 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style UNIX Software Tools Philosophy • It's a philosophy, not standardised methodology • Based on pragmatic and practical experience • Programs designed to do one thing well • Create new programs rather than extend • Text stream as the universal interface • Output of one program can be used as input to another • Sum is greater than the parts There are a number of guiding principles used when developing UNIX commands and programs in general. Collectively they are referred to as the UNIX Software Tools Philosophy. The philosophy is not a standard or methodology developed by a committee or from academic ponderings. Instead, it is based on the pragmatic and practical experience of a number of individuals who contributed to the development of UNIX. There are several interpretations of the philosophy but the points of interest for this presentation are: •Design programs to do one thing well. •Create new programs rather than extend existing ones. •Use text as the universal interface •Make the output of one program usable as input to another. Creating new programs rather than extend existing ones goes some way in explaining the overlap of functionality between programs. The idea is programs become overly complex when they are extended. As additional options and features are added the program becomes more difficult to use. This philosophy is complimentary to the "do one thing well" guideline. The down side to creating new programs is that UNIX appears to have a number of programs that have overlapping uses. In part this has given UNIX a reputation as being disorderly collection of tools. The benefit of creating new programs is that tools are small light weight programs can be coupled together in various ways. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 3 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style Simple Commands Name Description awk Pattern scanning and processing language cat Concatenate and print files cut Cut out selected fields of each line of a file date Display the date and time df Report free disk space find Find files grep Search text for a pattern nl Line numbering filter print / printf Write output / formatted output sed Stream editor sort Sort, merge, or sequence check text tr Translate characters uniq Report or filter out repeated lines wc Line, word, byte or character count xargs Constructs parameter lists and runs commands While there are many unix utilities there are some that are used very frequently. Of all the utilities, grep, sed, and awk are the “work horses”. The list of commands above is by no means extensive or exhaustive. Though my experience has been that the above utilities will cover 95% of my activities. I haven’t included vi, ed or ex editors, even though these can be driven from the command line. What makes unix so powerful is when commands are combined. This requires that programs are developed using the unix software tools philosophy. Programs can then be combined in many different ways. This leads to the whole becoming greater than the some of the parts. Complex results can be achieved from simple steps. There is a similar mindset in combining unix utilities as to writing complex sql statements. In unix, the biggest hurdle is to not think in terms of iterative processing of records within files. Doing so will lead to overly complicated solutions that go against the unix way. Instead you need to think of commands processing files and data streams. Once you have master this way of thinking the utilities provided by unix will start to work for you rather than against you. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 4 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style Compound Commands • Pipeline Command • List Commands • And , Or , Sequential , Compound List , Background Process • Command Grouping • Subshell , Brace • Iteration Commands • for , while , until Compound commands are used combine commands. There are various ways to combine commands including pipelines, list commands, command grouping and iteration commands. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 5 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style Pipeline Command • Pipelines are the easiest way to work with UNIX • The | ( pipeline ) operator is used to combine commands command [ | [ newline… ] command ]… • pipeline is a sequence of one or more simple or compound commands. • Stdin for command comes from the keyboard or previous command • Stdout for command goes to the terminal or next command The pipe is arguably the most important construct in unix. Pipes are used to flow data from one command to another. It's named as an analogy to a physical pipe. Several unix philosophies make pipes very effective. Namely text as the universal interface, output of a command should reflect the format of the input, doing one thing well. When programs follow these philosophies, pipes allows commands to work together. The shell does not run the commands in a pipe directly. Child processes are created for each command. While you can think of the sequence of execution going left to right, in reality all commands are running concurrently. Though they might not be doing anything if waiting on input from the previous command. To improve the efficiency and performance of a pipeline, previous commands should filter as much of the data as quickly as possible. Commands that must process the entire data-stream before passing data to the next command, such as sorting, are costly. This should all sound very familiar if you tune sql statements. One or more newlines can be entered after the | character. From the command line you will be given the secondary prompt ( PS2 is normally set to a > character) which indicates that ksh is waiting for the rest of the pipeline to be entered. Adding newlines to a pipeline is useful as it makes reading and understanding the pipeline easier. "Going Barefoot with UNIX for the DB2 DBA" by Robert Mala Page 6 IDUG DB2 Tech Conference Sydney, Australia, September 12 - 14, 2012 Click to edit Master title style Pipeline Command Unix tools solution to show word frequency $ man grep | > tr -cs '[:alnum:]' '[\n*]' | > tr '[:upper:]' '[:lower:]' | > sort | > uniq -c | > sort -k1,1nr -k2 | > head -3 108 the 47 a 25 to The pipeline in the slide demonstrates the power of pipelines ( examples using db2 coming up shortly).