Becoming an efficient DB2 LUW DBA by leveraging and in

Pavan Kristipati Huntington Bank Session Code: D11 05/25/2016 1:00 PM | Platform: DB2 for Linux, Unix and Windows

Photo by Steve from Austin, TX, USA Pavan Kristipati • IBM Champion 2015, 2016 • IDUG North America Conference Planning Committee • DB2 LUW DBA since 2005 • Presented IDUG in 2014 and 2015 • IBM Certified Advanced Database Administrator • Technical blogging • www.db2talk.com – Owner • www.db2commerce.com – Occasional guest blogger • https://www.linkedin.com/in/pavankristipati • @pkristipati @db2talk

2 Agenda • Objectives • Motivation and Introduction • Learning how to use awk and sed in regular DBA activities • Usage examples • Questions

3 Objectives • Introduction to awk and sed • If you haven’t considered using awk and sed until now, my goal is to motivate you to do so • Share 4 things about awk and 2 things about sed that a DB2 DBA efficient • Show that awk and sed are not that clumsy! • Show how sed and awk together could be powerful tools

4 Setting the stage • No heavy attention to syntax • Multiple ways to get the same thing done. Only one approach is discussed • Focus is on breath. Not depth

5 Motivation to look into awk and sed • Reduce (boring) typing • Do, , do, wait, do.. doesn’t scale • Increased confidence in scripting/automating over  self sufficient DBA • UNIX scripts to make your job simpler • Automation using scripts • Up to the challenge of learning usage of seemingly arcane utilities

6 So, what are awk and sed? • Two of the most under-appreciated utilities in UNIX • Invaluable if you ever have to: • Make repetitive changes to large pieces of code or text • Analyze some text • Part of most UNIX offerings

7 Introduction to awk • Originally designed/implemented in 1977 by Al Aho, Peter Weinberger, and Brian Kernigan - language takes it name from authors’ initials • Several implementations (that rhyme with awk) • General purpose programming language in UNIX that handles text () as easily as numbers • This makes awk one of the most powerful of the Unix utilities • Easy manipulation of structured data • Process fields while sed process lines

8 awk basics and terminology

9 awk - Input and Output • Input • Content in a - $ filename | awk ‘command’ • Output from a command - $db2 list db directory | awk ‘command’ • Output • Fields in a file • Single value • Functions or other actions on content • Open to imagination

10 awk - Input and Output • What can we do with output? • Send to a file • print "expression" > "file name“ (Think SQL files) • Send through a pipe for further actin • print "expression" | "command” (One – liners)

11 Lets get basics right - Fields and Delimiters in awk • Think of field as a column • Default delimiter: ‘space’ or ‘tab’

• $1 = 1st field • $2 = 2nd field • $0 = the entire line

12 Lets get basics right - Fields and Delimiters in awk • awk’s most basic function – Printing text (don’t underestimate) • Print 1st field (print only schema names) $ cat file1 | awk '{print $1}‘

• Notice the difference: • cat file1 | awk ‘{print “$1”}’ – prints $1 instead of value of 1st field • Text in double quotes is printed literally

13 Unleashing the power of awk

14 awk Tip #1

15 In tip #1, we will look at: • Generate DB2 commands • Combine SQL and awk • Perform a set of actions on a: • static file of list of tables • dynamic list of tables

16 Power of awk – Generating DB2 Commands Scenario: Runstats on multiple tables • Table list in a file

• Do-Wait-Do-Wait.. is not for efficient DBAs  • Use awk to be efficient • 2 steps • Generate command (we will use UNIX’s ‘cat’ in this step) • Run command (and take a coffee break)

17 Power of awk – Generating DB2 Commands Template: $ cat tabslist | awk '{print “runstats command” variables}’

Step1: $cat tabslist| awk ‘{print “runstats on table “$1”.”$2”;”}’ > stats.sql

Step2: $db2 –tvf stats.sql runstats on table SCHEMA1.TABLE1 DB20000I The RUNSTATS command completed successfully. runstats on table SCHEMA2.TABLE2 DB20000I The RUNSTATS command completed successfully. …………………… 18 awk one liners • Could this be done in 1 step? Yes

• Just send awk’s output to DB2 prompt • $cat tabslist | awk ‘{print “runstats on table “$1”.”$2}’ | db2 –v db2 => runstats on table SCHEMA1.TABLE1 DB20000I The RUNSTATS command completed successfully. db2 => runstats on table SCHEMA2.TABLE2 DB20000I The RUNSTATS command completed successfully.

19 Power of awk – Generating Commands from a dynamic list Two questions: • Does this scale to large number of tables? Yes • Does the table list has to be in a file? No 3 actions in one line: . Generate a list . Pass the list to awk as input and . Pass the entire output to DB2 prompt $db2 -x "select char(tabschema,20), char(tabname, 50) from syscat.tables where = 'T' and tabschema = 'SYSIBM'" | awk '{print "runstats on table "$1"."$2}‘ | db2 -v Runstats on all tables in SYSIBM schema (144 tables in DB2 10.5)

20 awk Tip #2

21 Combining SQL and awk In tip #2, we will look at: • Pass awk fields into SQL • Generating SQL statements using awk • Handling single quotes in awk in SQLs ‘where’ clause

22 Combining SQL and awk • Example: Print column names • Where clause has table and schema names in single quotes SQL to print column names: • select char(tabschema, 20), char(tabname, 60), char(colnames,200) from syscat.indexes where tabschema='SCHEMA1' and tabname='TABLE1’

23 Handling quotes in awk – What doesn’t work Quotes needs extra attention and handling Goal: Enclose SCHEMA and TABLE in quotes None of these work $cat list | awk '{print "select char(tabschema, 20), char(tabname, 60), char(colnames, 200) from syscat.indexes where clause Enclose fields in single quotes: × where tabschema = ‘$1’ and tabname = ‘$2’;"}’ Using combination of double and single quotes: × where tabschema = "'$1'" and tabname = "'$2'";"}‘ Escaping single quotes: × where tabschema = "\'$1\'" and tabname = "\'$2\'";"}‘

24 Handling quotes in awk - – What works Solution – awk’s Assignment Operator ‘-v’: • $cat list | awk -v x="'" '{print "select char(tabschema, 20), char(tabname, 60), char(colnames, 200) from syscat.indexes where tabschema="x$1x" and tabname ="x$2x}‘ • Looks complex? Actually it is not! • Just have assignment operator before and after fields ($1 $2) • http://db2talk.com/2015/07/31/dba-tip-easy-way-to- handle-single-quotes-in-awk/

25 Assignment Operator – Add date/time to vmstat data • By default, AIX’s vmstat output does not have a good way to record date/time values. • vmstat in RHEL has ‘-t’ option for date/timestamp • How do we add date/time values? • Use awk’s assignment operator.

26 Adding date/time to vmstat data using awk $vmstat 15 | parse_vmstat.sh ## Pass vmstat output to

##parse_vmstat.sh – Snippet ## Step 1 – Capture current date and time in a shell variable $cur_time=$(date '+%m/%d/%Y %H:%M:%S') ## Step 2 – Pass this variable to awk and print it along with vmstats output $line | – swpd | awk –v cTime="$cur_time" '{print cTime, $0}‘ # output 03/05/2016 15:31:12 1 0 0 3311268 327304 3757292 0 0 12367 58 13 24 17 2 80 0 0 03/05/2016 15:31:27 1 0 0 3311268 327304 3757292 0 0 12367 58 13 24 17 2 80 0 0

27 awk Tip #3

28 In tip #3, we will look at: • Printing output in a better format • Separate fields in output for clarity

29 Applying make up to output – Tip #3

• Print output in a formatted way – Use (fancy print) • Format specifiers let us separate fields in output for clarity

30 Applying make up to output – Tip #3

31 awk Tip #4

32 In tip #4, we will look at: • How to handle non-default field separators • Other special variables in awk

33 Custom Field Separator in awk • Until now, we worked with files that have fields separated by space or tab. • What if fields are separated by other characters?

Request from a developer: Hey DBA, could do you drop these tables? 34 Custom Field Separator in awk

• Use awk’s ‘F’ or ‘FS’ for field separator.

35 Other Special Variables in awk

• FS -- The input field separator. The default value is a blank. • OFS - The output field separator (default is a space).

• RS -- Input record separator (default is a new-line character). • ORS - Output record separator (default is a new-line character).

• NF -- The number of fields in the current record.

36 sed

37 Introduction to sed • stream editor (sed) • Perfect for applying a series of edits to a number of files or to output of a command • Processes lines while awk processes fields

How it works? Input text flows through the (sed) program, is modified, and is directed to output

Output from a Sed command Command prompt / command /script file /text file

38 sed – Input and Output • By default, ‘sed’ operates on each line. • By default, all ‘sed’ commands are applied on the pattern buffer, hence the input file remains unchanged. • GNU ‘sed’ provides a way to modify the input file in-a-place.

39 sed tip #1 The most popular substitute command

40 sed’s most popular substitute command – #1

Goal: To delete double quotes from a DDL file

$cat ddlfile | sed 's/"//g‘ > newddlfile More variations of this command are in notes

41 sed’s most popular substitute command - #2

Goal: To replace EDWDV with EDWQA  Think QA migration

$cat ddlfile | sed ‘s/EDWDV/EDWQA/g‘ > newddlfile

What if we wanted quotes to go away as well?

42 sed tip #2 handling large number of substitutions

43 Handling multiple substitutions – the efficient way • If you have a large number of sed commands, you can put them into a file and use sed -f sedscript new

44 Multiple substitutions using sed Delete quotes, lines with CONNECT, COMMIT, RESET and TERMINATE

How did this happen? 45 Multiple substitutions using sed dv.ddl

sed –f sed.file dv.ddl > qa.ddl

Simpler example: Case in-sensitive replace s/[Aa]/A/g s/[Ee]/E/g s/[Ii]/I/g s/[Oo]/O/g s/[Uu]/U/g 46 awk and sed together

47 Example #1

48 Handling $# =1 or $# =2

Scenario: There could be 1 or 2 arguments. How to handle?

$#=1

$grantdev SCHEMA.TABLE (arguments could be copied and pasted) Vs. $grantdev SCHEMA TABLE (more effort to copy and arguments)

$#=2 Goal: Come up with a way for the script to handle $#=1 or $#=2

49 Handling $# =1 or $# =2 $grantdev DB2INST1.EMPLOYEE $grantdev DB2INST1 EMPLOYEE Goal: Both should work ## Code Snippet for grantdev period='.‘ ## assign ‘.’ to a shell variable delim=' ‘ ## default delimiter is ‘space’ schema=$(echo $1 $2) if [[ "$schema" == *"$period"* ]] ; then ## checks if $schema has ‘.’ in it delim=$period ## if yes, assigns delimiter to ‘.’ fi SCHEMA=$(echo $schema | awk -v delim1=$delim -F"$delim" '{print toupper($1)}' | sed 's/^ *//;s/ *$//') TBNAM=$(echo $schema | awk -v delim1=$delim -F"$delim" '{print toupper($2)}' | sed 's/^ *//;s/ *$//' ) P.S.: Refer notes for comments on the above 2 lines

50 Example #2

51 Granting privileges - multiple tables – one liner Goal: Grant privileges with ease for large number of tables

$cat $ddl | grep -i "create table" | sed 's/(.*//g' | awk '{print $3}' | awk -F '.' '{print "grantdev "$1" "$2}' | sh | -a grant.log sed – get rid of all chars after tablename example: CREATE TABLE DB2INST1.TAB1(col1 integer….) awk – • print SCHEMA.TABLE • set delimiter to ‘.’ • print “grantdev SCHEMA TABLE”

52 Example #3

53 Simplifying reorg operation after altering tables • Some of the ‘alter table’ statements require reorg • Imagine you just altered bunch of tables – How do you know which ones to reorg? • What’s the most efficient way? Use awk and sed

## First get a list of all (unique) tables $cat ddlfile | grep -i "alter table" | awk '{print $3}' | -u | awk - F '.' '{print $1" "$2}' > $$.tmp

54 Simplifying reorg operation after altering tables while read line do SCHEMA=$(echo $line | awk '{print $1}'| a-z A-Z) • Less typing TABLE=$(echo $line | awk '{print $2}' | tr a-z A-Z) • No guess work checkx=$(db2 -x "select distinct(reorg_pending) from on which table table(sysproc.admin_get_tab_info('$SCHEMA','$TABLE')) as t") is in reorg- echo $checkx | grep -q Y ## with zero status if ‘Y’ is found returncode=$(echo $?) pending

if [ "$returncode" -eq "0" ]; then echo "Table $SCHEMA.$TABLE is in reorg-pending mode" db2 -v "reorg table $SCHEMA.$TABLE" else echo "reorg not needed for table $SCHEMA.$TABLE" fi done < $$.tmp 55 Cool things we did with awk and sed

56 Some of the cool things we do with awk and sed

• Alert on invalid objects • Alert on reorg pending objects • Alert if backups run too long • Keep track of table size growth, database growth etc. • Keep track of log utilization and generate alerts • Mix those with tails, top, and a job… and we can get a great reporting system • Simplify DDL changes in a DPF environment and lots of other things that do not tie a DBA to his screen 

57 58 Pavan Kristipati Huntington Bank Please fill out your session [email protected] evaluation before leaving!

D11: Becoming an efficient DB2 LUW DBA by leveraging awk and sed in UNIX

Photo by Steve from Austin, TX, USA