Download in a Separate Pdf File

Note: Appendices are available for download in a separate pdf file.

But we NEED to BE developers!

DBAs frequently (constantly?) get called on to develop “quick” database scripts and programs, never mind that we don‟t have the time, the tools, or the training.

From a quick, little runstats script in ksh, to a full blown application to load data or “fix a few rows”, it seems that if the data is in the database, it‟s my job to fix it!

This presentation will present examples of manipulating DB2 data in several programming languages commonly used with DB2 LUW ( ksh, perl/dbi, SQL/PL, java/jdbc ) and discuss the plusses and minuses of each. Heavy emphasis will be placed on error handling, as this is one of areas that is most often overlooked or under developed.

My favorite utility will be explained and sample code in perl and java will be discussed.

1 Next, I will share some of my favorite utilities, tools, tips and tricks to help the DBA get things done.

1 Show similar common DBA tasks in different languages, and give you some “tips and tricks” for each.

Number 1 Depends on your DBA job description. But, for some of us… It is a novel idea that needs to be evangelized.

2 As a DBA, we are constantly tasked with fixing data, loading tables, and running utilities.

It seems like, “If it lives in a database table, It‟s our responsibility to load, correct, and maintain.”

We need to have and use the proper tools to do this.

Many developers with LOTS of tools. One DBA, with…. Ksh and CLP.

3 This is the proper attitude for us to have. As long as we have a great set of tools to work with!

However, UNLIKE Jeff Spicoli, we need to actually HAVE good tools and know how to use them.

4 Show similar common DBA task in three different languages, and give you some “tips and tricks” for each.

Favorite bumper sticker – Think Geek used to sell them.

Woodshop analogy: • If I want to help you get into woodworking: • Show you some cool tools: • Miter saw • Band saw • Planer • Jointer • All “cut” wood. All very cool. All useful. • Good in different ways and in different cases • Show you a cool “Chest of drawers” that you can build with it • That is what I am going to do with software tools

5 Show similar common DBA task in different languages, and give you some “tips and tricks” for each.

Many times we are going into a fight with no weapons. (Or at least the wrong ones. )

Others? Python XML

6 Some good Manuals from the DB2 documents. http://www-01.ibm.com/support/docview.wss?uid=swg27038855

7 Note: Typeset –u – uppercase dbname ${1-sample} – default value for a parameter Backticks ` ` to run a command and capture it‟s output

We are running db2 commands via the clp

Five GLARING errors or Issues with this script: 1) No error checking What if you don‟t get something good back from db2? What if connect fails? 2) No way to check return codes from “underneath” tee command 3) No return code on the exit 4) “race condition” exists on select vs delete. The time between them is >0, so they may get different row count as rows may get old enough to be purged. 5) NO Intermediate COMMITS!!!!! Will this blow up your logs? What if it doesn‟t run for a while, then gets run? This is hard to fix in ksh / clp

Note:

Trying to handle the “race” condition by using start_ts (which is, The time at which I start purging rows) Use count as the Number of rows to delete

Using an error handling function: sql_error_handler Part of my standard library simply a case statement on rc 0 – ok 1 – no rows found 4 – warning 8 - error

Note: Looping until we delete enough rows. Committing every “chunk_sz” rows Using variable output to capture sqlca info from “db2 –a” purge_row is: number of rows from sqlca sqlcode: from sqlca

Issues: Must rerun query to determine deletable rows over and over – no cursor. This could be really expensive!

10 This is an example of what “db2 –a” output looks like, so that you can see what I am “cut”ting out.

This is somewhat fragile, should DB2 change this output.

Not prod ready IMHO…

Not sufficiently advanced technology - I can tell it from Magic 

11 DB2 CLP is limited by the SQLCA - 32K buffer size “Panel” at idug in philly had a suggestion (that I haven‟t gotten to work, yet) WLM_SET_CLIENT_INFO procedure

Have been forced to do some work in “O-rackle” lately. I MISS the CLP…

12 Need: A library of tools / functions.

13 See Appendix A

Use of function in ksh

Comments! They are GOOD!

Check to see if the log file exists. If not, see if I can touch the log file and if so, can I write to it.? If not, write out reasonable errors and die with a RC 1.

14 Note:

If both stdout (1) and stderr (2) are a tty, then I must be on a terminal.

Basically, I am sending stdout and stderr to the log file, or if I‟m a terminal, I send it to tee and push it to the tty and the log file.

use of print – more POSIX compliant vs echo.

15 See Appendix A

This just shuts down the file descriptors using exec and makes sure to wait for the children to actually return.

Maybe it should check to see if the logging is actually started, before it tries to stop it…

16 Note:

“Source” in our .profile – big deal if running from cron Cron environment VERY different from your login environment. None of YOUR “stuff” .profile, .login pulled in. Cron environment is a “crippled” environment.

Pull in our function library.

Use of basename to get program name

Fully qualified program names – good practice especially if running from cron ($PATH may not be your $PATH)

Function call: Call to start_logging – passes in a log file name (This is verified in the function) stop_logging – no parameters

Reading input from a file on a specific file descriptor – not commonly done in ksh. In this program, I also read from the terminal, so I need to use a separate “fd” for this input file.

Continue statement { } – use – delimiter – reduces ambiguity – BEST practice!

LOGFILE=${RECIPEFILE%.*}.log - strips anything to the right of the first „.‟

17 Note:

Good to have a heading with common stuff in it.

Options for ksh: xtrace (-x) - debug nounset – will die if you don‟t have a param set errexit - can make error handling easier

18 Note:

Co-worker who did this trick all the time. I had never seen it.

Concern: What if you don‟t get exactly what you expect, when you read? Can you catch all possible weirdness??? - Buffer overflows, and SQL injections in this kind of thing, are very possible.

19 Note:

Co-worker who did this trick all the time. I had never seen it.

Concern: What if you don‟t get exactly what you expect, when you read? Can you catch all possible weirdness??? - Buffer overflows, and SQL injections in this kind of thing, are very possible.

20 From my backup script: All this just to subtract one from the log file name!

Could do MUCH more easily in perl.

21 See Appendix B

So, let‟s move on to Stored Procedures.

See Appendix B

23 Documentation implies that you have to do ENABLE to get the Output buffer to work. I Never have before. However, this time, I wrote more than the default 20,000 bytes and the proc would die with a SQL20511N. Yikes!

This ENABLE Proc makes it “work”, but IMHO, it is still very fragile, if I log a bit too much, my proc will break.

He contends that this is much more efficient than the previous cursor based Stored proc. Serge knows more about SQL and the optimizer than I. I believe that it IS faster than the cursor version. But I notice that it does do the select multiple times. Which can be bad if the select is expensive. There are many variables, such as how expensive the select is, how the cursor is handled, i.e.: how many rows are buffered on each fetch, etc.

29 30 Here is a way to solve the “wrap a variable in single quotes but still let the shell inside the quotes to evaluate the variable” issue.

I used a here document. (which adds a gratuitous newline). I used tr to chop the new line off.

31 When I started this presentation, I thought that I could create this little wrapper script quickly. I spent hours getting this „single quote‟ issue resolved. Still doesn‟t work in ksh/Cygwin on windows. ??? Anyone?? - superceded…

Note: Solved!!!! TRIMMED_TS=`echo ${QUOTED_TS} | tr -d '\n‘ | tr –d ‘\r’` windows newline.

I have never, BTW, gotten the perl/dbi working against db2 on Cygwin. Anyone?

32 The DBI is a library that provides database routines that are agnostic about the underlying database. The various DBDs provide routines to support dbi calls for a given database manager.

33 The DBI is a library that provides database routines that are agnostic about the underlying database. The various DBDs provide routines to support dbi calls for a given database manager.

May need to recompile perl for AIX to get performance enhancements, Multi- threading, etc.

A few Gotchas under AIX 6.1!

35 This is what a “typical” dbi program looks like. Use DBI Connect Prepare Execute Fetchrow-array or hashref in a loop

Commit (or rollback)

Similar example but showing INSERT vs. SELECT.

37 Wanted a generic tool to do this VERY common task.

Non perl-capable dbas can use it to do data updates.

DBI – must think “dbi way” Don‟t open a db2 cursor.

As we saw in the ksh example, We only want to do the select once, as it may be very costly. But, we must have intermediate commits (as log size is finite!), so we are forced to do CURSOR SELECT and then updates in a loop.

39 See Appendix C1

First part of the .ini file.

Second half of the .ini file.

42 See Appendix C2

43 Error handling with issues in the .ini file could use some improvement. DB_connection functions could be improved.

Rollback: I tried (and really wanted) to be able too, actually run the “update” statement and DO the work in the database. Then, if UPDATE_DATA flag is set to „0‟ (false), just issue a rollback vs a commit each time. This would have provided a much better idea (worst case scenario, maybe) of the time the updates would actually take to run. However, rollback, closes down the “cursor” in the dbi, and ends the loop, etc.

From the “DECLARE CURSOR” page in knowledge center: • For units of work ending with ROLLBACK: • All open cursors are closed. • All locks acquired during the unit of work are released. • LOB locators are freed.

So, I just log what I “would have updated”. This can still be very useful for testing and debugging, as it prints out the values of the columns in debug log level.

48 See dbi performance documentation for details on Fetchrow_arraryref vs hashref, etc.

Perl has Autovivification – java does not 

49 50

Dumps out the settings in the ini file for documentation. Select can be devilishly complex! - join, merge, etc.

51 Shows the Update statement as prepared.

You can see the Parameter Markers and the values that each was assigned for each row.

If we had committed, during the run, it would have logged the rate in rows / second. This is VERY useful on long performance test/prod runs. “Did it slow down?”

If you have DEBUG logging set in the ini file, you get param values for each “update” statement. Might be too much for prod Enormous log file Time to write log records INFO would just show each commit / rate info.

52 See Appendix D

53 Why Java

Slide says it all. Most development projects have at least SOME java.

54 Look in the SQLLIB/samples/jdbc directory

55 Simple JDBC example Connect Execute statement Loop thru Result Set Close

56 See Appendix E1

Properties file – similar to .ini file in perl. Not all functionality has been duplicated, yet.

57 See Appendix E2

64 Rollback closes cursor and shuts down all the fun, here too.

66 An attempt to solve this issue of not knowing the type of each column ahead of time.

Didn‟t work.

67 Java, Pascal, Ada and C require all variables to have a declared type, and support the use of explicit casts of arithmetic values to other arithmetic types. Java, C#, Ada and Pascal are sometimes said to be more strongly typed than C, a claim that is probably based on the fact that C supports more kinds of implicit conversions, and C also allows pointer values to be explicitly cast while Java and Pascal do not. Java itself may be considered more strongly typed than Pascal as manners of evading the static type system in Java are controlled by the Java Virtual Machine's type system. C# is similar to Java in that respect, though it allows disabling dynamic type checking by explicitly putting code segments in an "unsafe context". Pascal's type system has been described as "too strong", because the size of an array or string is part of its type, making some programming tasks very difficult.[7][8] Smalltalk, Perl, JavaScript, Ruby, Python, and Self are all "strongly typed" in the sense that typing errors are prevented at runtime and they do little implicit type conversion, but these languages make no use of static type checking: the compiler does not check or enforce type constraint rules. The term duck typing is now used to describe the dynamic typing paradigm used by the languages in this group.

68 What would a Star Trek Themed presentation be without Lt. Uhura?

69 Ken has been a DB2 Consultant for fifteen years, specializing in LUW systems. He has worked on a variety of large databases for large corporate and government clients. Most of these projects have involved deploying and supporting mission critical applications where constant availability and high performance are required. Ken is an IBM Champion and finished second in the 2014 DB2‟s Got Talent Competition. He worked as a DB2 DBA on the mainframe somewhere around the v2.2 timeframe. However, the details are a bit fuzzy… In between, Ken was a COBOL / SQL developer and a C / SQL developer. He then spent over ten years as a UNIX systems administrator, supporting almost every flavor of UNIX except AIX. (His LUW experience has remedied that).