AD210 Application Development and Deployment for DB2-Enabled IBM Lotus Domino Server

John Curtis Tom McGary IBM Software Group Agenda

ƒ DEMO – “The OmniLink View” ƒ Basic IBM Domino/DB2 Enablement, with DB2 Access Views ƒ System Configuration Planning ƒ Application/Data Planning ƒ Creating DB2 Access Views and Query Views ƒ Query View Complexity and Design ƒ Filtering ƒ Function Library Overview ƒ Opening remote documents ƒ DEMO – “The Ultimate Inbox” DEMO

The OmniLink View Content Disclaimers

ƒNOT a substitute for really learning DB2 SQL ƒ IBM DB2 SQL Workshop and DB2 SQL for Experienced Users http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_descripti on&courseCode=CF120 http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_description&c ourseCode=CF131 ƒ Publications: SQL Reference Volumes 1 and 2 (SC09-4844-00 and SC09-4845-00) Content Disclaimers (continued)

ƒ NOT a substitute for really learning DB2 Administration/Tuning ƒ IBM DB2 Universal Database Administration Workshop for Windows/UNIX http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_description&co urseCode=CF231 ƒ DB2 UDB Performance Tuning and Monitoring Workshop

http://www- 304.ibm.com/jct03001c/services/learning/ites.wss?pageType=course_description&courseC ode=CF412&country=us&language=en ƒ Publications: Administration Guide: Implementation (SC09- 4820-00), Administration Guide: Performance (SC09-4821-00) and System Monitor Guide and Reference (SC09-4847-00) ƒOnline documentation: http://publib.boulder.ibm.com/infocenter/db2help/index.jsp Terminology/Acronyms

ƒ DAVs or DB2 Access Views - Mechanism (DB2 View) to expose Domino data to the outside, relational world ƒ QVs or Query Views - Mechanism (Domino View) for rendering relational data using Domino views ƒ DAVPOP or DB2 Access View populator – Domino process to initialize DAVs with Domino data ƒ Federation – incorporating data from multiple sources into a central composite ƒ 2 types ƒ Domino database federation – JOINs/UNIONs across different NSFs (well, DB2NSFs) ƒ DB2 federation – pulling in data from foreign relational stores (other DB2 instances or Oracle, etc) Domino/DB2 and DB2 Access View Enablement

ƒ Attend Session ID105 for more detail ƒ Setting up IBM Domino Server for DB2 ƒEssentially, follow the directions ƒFor DB2 Access, remember to create the id file first ƒRead documentation carefully, follow to the letter

ƒ DB2 Access View Setup ƒUse default DB2UserID for QVs if possible ƒKeeps DB2 superuser credentials from normal QV use Application Planning

ƒDiscovering Hybrid Applications - Questions ƒWhat portions of corporate relational data are the most valuable to participate in Domino applications? ƒReduce redundancy ƒ(Re)organize presentation ƒEnrich content, etc. ƒEase of implementation important (e.g. ETL is costly) ƒWhat Domino databases easily “join” to present data never seen together before? ƒWhat portions of Domino data are the most valuable to participate in relational applications (not covered this session) ƒWhich applications would benefit most from having Messaging/discussion/workflow data in relational form? Application Planning (continued)

ƒMore Questions: ƒ What new applications are implied by the Domino/relational melding? ƒ Messaging reports, document access from outside Domino ƒ What transformation/design modification would best enable hybrid applications? ƒ Join keys, name field standardization, data uniformity

ƒ HOW WELL DO YOU KNOW YOUR DATA? ƒ From Domino - Items, Forms, Views ƒ From Relational – databases, tables, columns, applications Configuration Planning ƒServer Configuration ƒ Will DB2 run on the same (local) or different (remote) server? ƒ Advantages of DB2 on a dedicated server ƒ 64-bit memory support ƒ Disk configuration ƒ Separate tuning and administration ƒ Possible consolidation of multiple Domino servers onto a single DB2 server ƒ Will non-Domino relational data be part of the application? ƒ If yes, where will it reside and how will it be accessed? Remote access (that is, to another RDBMS) needs planning and configuration Materialized Query Tables (MQTs) – temporary local copies of relational data kept up to date automatically Configuration Planning

ƒ Application Data planning ƒHow much processing will be done on the DB2/Domino Server? ƒFull blown databases – allow for complete processing on the Domino/DB2 server ƒSkeleton replicas (one way – in) ƒKeep only the fields necessary to render Query View(s) ƒSince remote documents can be opened, this is all you may need ƒSavings are at least 95% in disk ƒEnables consolidation of many database for highly federated QVs ƒInterface is laborious (working on it) Configuration Planning (continued)

ƒ How much processing will be done on the DB2/Domino Server (continued)? ƒQuery View – only databases ƒGood for developing Query Views apart from production data ƒAllow for independent (from data) access control ƒDomain issues ƒRepetitive work – exporting DAVs and QVs across the enterprise Creating the Design Objects ƒDB2 Access Views ƒWhat they are: ƒMechanism to expose Domino data to the outside, relational world ƒUpdatable DB2 Views (functionally equivalent in SQL to tables), with one row per document where rows contain Copies of selected Domino document fields, available in relational form Security information (e.g. $Readers) Domino system/navigational data (#noteid, #unid, #modified) Creating the Design Objects (continued)

ƒDB2 Access Views (continued) ƒ How they work ƒ User Defined Function (UDF) executes for every SQL statement Checks Domino privileges at database and document level Passes through SQL to DB2 ƒ Populated in DAVPOP – batch process making copies ƒ Maintained under normal NSF core updating API calls ƒ Overhead ƒ Disk footprint – even with indexes, 3% is the most measured ƒ Update costs – 8-12% extra time (wallclock) Creating the Design Objects (continued)

ƒ DB2 Access Views (continued) ƒForm-based vs. All forms – updating consideration ƒSmall cost to make form a DB2 column to add to WHERE clause ƒFewer DAVs = less updating/fewer objects ƒIf only one form is of interest, save space and map its documents only ƒIncluding extra fields ƒAlways included #noteid, $Readers ƒUnless you have a reason not to, include these ƒGMT Conversion – makes date math very straightforward Creating the Design Objects (continued)

ƒ DB2 Access Views (continued) ƒChoosing columns – be thorough but judicious ƒKnow your data or you will learn about it as you go ƒPlan for federation in advance (JOINs, UNIONs, etc.) ƒError avoidance/data mapping ƒTruncation ƒLists and ranges vs. single occurrences ƒStrong data typing – clean up multi-typed Domino fields Creating the Design Objects (continued)

ƒ DB2 Access Views (continued) ƒDAVPOP ƒDebugging problems – need mechanism to get documents by NoteID ƒCheck lengths, types and occurrences ƒOngoing criteria enforcement – if you had problems populating, problematic data may get replicated in again Creating the Design Objects (continued)

ƒQuery Views ƒ What they are ƒ Mechanism for rendering relational data using Domino views ƒ Temporary SQL query results No persistent footprint in Domino Solution for Domino view explosion Document-backed Aggregate, report data Combination ƒ How they work ƒ Map SQL query column values to Domino view columns ƒ Every time the QV is opened, it rebuilds, firing the SQL query ƒ The SQL query can be different each time Creating the Design Objects (continued)

ƒQuery Views (continued) ƒ Choosing view columns – ƒ As usual in view design, but consider multiple sources and shapes of data E.g. a federated “subject” or “author” column Less is better (use SQL to your advantage) ƒ Categories and responses supported, multi-value permutation works via Lotusscript Explode ƒ Selection criteria (WHERE clause) ƒ Can be specified at runtime ƒ More restrictions are better than fewer ƒ Common restrictions (e.g. form = ‘Memo’) can be supplied via additional DB2 views ƒ ORDER BY ƒ Overridden by any Domino View column collation ƒ Useful in UNION (ALL) selects Creating the Design Objects (continued)

ƒQuery Views (continued) ƒ Implicit columns (mentioned under DAVs, part of why you want them:) ƒ #noteid – for opening federated documents ƒ #oid – to support @ functions like @Created ƒ #unid – to support app calls using UniversalID ƒ Non-Domino data will not have these, but via JOINs these can be supplied

e.g. – to open a Domino-resident description document from a view with relational-based manufacturing parts:

SELECT a.cost, a.partno, a.partdesc, a.supplier, B.#noteid FROM corporate.parts a LEFT OUTER JOIN dom.description b ON a.partno = b.partno Creating the Design Objects (continued) ƒQuery Views (continued) ƒSQL selection ƒSQL columns are mapped to view columns based upon column value (type formula) = SQL column name ƒThe SQL “AS” clause projects any name to be mapped across any SQL table (e.g. SELECT rtrim(doctype) AS type FROM table)

ƒColumn Construction can get ornate: SELECT lastname || ‘, ‘ || firstname || ‘ phone: ‘ || phone AS contact Query View Complexity and Design

ƒ Simple Query Views ƒSelf contained, where document data in this database (i.e. using its DAV(s) only) can be: ƒFiltered ƒReformatted ƒRe-categorized ƒMost common uses: ƒReplace/augment existing views Replaced views create savings by eliminating update/updall processing ƒNew reports Query View Complexity and Design (continued) ƒComplex Query Views ƒ Query references data in this database (using its DAV(s)) and at least one other source (either another DAV or relational table) ƒ Use JOIN or UNION ƒ Simplest – adding column(s) to the SELECT list:

SELECT a.#noteid, a.from AS FROM, b.phone AS PHONE FROM mail a LEFT OUTER JOIN phone b ON a.id = b.id

ƒ Most complex – presenting data from several sources in a single view:

SELECT ‘server_a’ AS server, ‘db_a’ AS database, from AS name, #noteid FROM joe.mail_dav UNION ALL SELECT ‘server_b’ AS server, ‘db_b’ AS database, author AS name, #noteid FROM corporate.discussion_dav Query View Complexity and Design (continued) ƒ Complex Query Views (continued) ƒ Regular table expression – allows for highly variable internal SELECT statements (maps well to categorized view)

WITH VIEWTAB (TYPE, NAME, SUBJECT, #NOTEID) AS (SELECT ‘E-MAIL RECEIVED’, principal, subject, #noteid FROM joe.mail_dav WHERE FORM = ‘Memo’ AND principal <> ‘Joe’) UNION ALL (SELECT ‘E-MAIL SENT’, principal, subject, #noteid FROM joe.mail_dav WHERE FORM = ‘Reply’ ORDER BY DELIVEREDDATE DESC FETCH FIRST 50 ROWS ONLY) UNION ALL (SELECT ‘DISCUSSION’, author, topic, #noteid FROM corporate.discussion_dav) ) SELECT * from viewtab Query View Complexity and Design (continued) ƒ Categorized QVs - ƒ DB2 column values work as categories ƒ QueryOpenDocument NotesUIView CaretCategory does not contain the categorized column value so, if you it: ƒ Create a second, invisible, collated column under the category including the categorized value (and perhap noteid ??) ƒ If the categorized value corresponds to the database housing the document to be opened, create another view to lookup server and DBname from the categorized value

WITH VIEWTAB (TYPE, NAME, SUBJECT, #NOTEID) AS (SELECT ‘E-MAIL RECEIVED’, principal, subject, #noteid FROM joe.mail_dav WHERE FORM = ‘Memo’ AND principal <> ‘Joe’) UNION ALL (SELECT ‘E-MAIL SENT’, principal, subject, #noteid FROM joe.mail_dav WHERE FORM = ‘Reply’) UNION ALL (SELECT ‘DISCUSSION’, author, topic, #noteid FROM corporate.discussion_dav) ) SELECT * from viewtab Query View Filtering Techniques

ƒ Fight Infoglut – Providing access to exactly the right data at the right time – SQL at its best

ƒ Filtering – limiting the data presented in a given context ƒ Table/Domino database level – limit contributing sources of data ƒ FROM clause or SELECTs within a UNION query ƒ Document/row level – within a source, limit the documents ƒ WHERE clause, ORDER BY (or Domino View collation)

ƒ Where and how to obtain filtering data ƒ User entry – prompts and pull-down lists ƒ From the data ƒ Author, title, form, other fields/columns ƒ Action (reading e-mail, responding to discussion document, etc.) QV Filtering Techniques (continued) ƒAdvanced/hidden context ƒ Embedded words (e.g. “ACTION REQUIRED”) ƒ Not scientific ƒ Must use LIKE operator in WHERE clause – LIKE ‘%ACTION REQUIRED%’ ƒ Domino contextual lookup during Query construction ƒ Combined context ƒ WHERE clause (e.g. form = ‘Memo’ AND from = ‘The boss’ AND subject LIKE ‘%Your raise%’) ƒ Join with another DAV (or table) to limit the rows returned ƒ New context – ƒ Design Question: What new fields/columns can be included in application data to help DB2/SQL determine context better? Opening Remote Documents

ƒWithout capturing control, a double-click will open a NoteID in the current database, so… ƒ QueryOpenDocument method must supply logic to open the right document in the right database on the right server ƒ NotesUIView properties that help: CaretCategory – value in first sortable column in the view entry For categorized view, the first non-categorized sortable column CaretNoteid – the #noteid value in the view entry View – the NotesView corresponding to the NotesUIView But.. need to find current entry (usually by unique CaretCategory) ƒ Needed to open a remote document Server name, Database name or ReplicaID, NoteID or UNID (#noteid supplied via CaretNoteID) ƒ Once database is opened, use GetDocumentById or GetDocumentByUnid DB2 Related @ functions

ƒ@IsDB2 – T/F return, is the Domino database DB2-enabled

ƒ@DB2Schema – will return the schema name for any DB2-enabled NSF – ƒ must prepend table name (schemaname.tablename) in FROM clause

MySchema:=@DB2Schema(@DbName); MyTable:=MySchema + ".MAIL"; DB2 Function Library (not an exhaustive list) ƒ Scalar Functions ƒ Datatype Casting – CHAR, VARCHAR, DECIMAL, INTEGER, DOUBLE, BIGINT, TO_DATE functions ƒ String manipulation – RTRIM, LTRIM, SUBSTR, LOCATE, UCASE, LCASE, CONCAT (or ||) ƒ Dates and date math – DAYS, DAY, DAYOFWEEK, DAYOFYEAR, WEEK, WEEK_ISO, YEAR, QUARTER ƒ Times and time math – HOUR, MINUTE, SECOND, MIDNIGHT_SECONDS, MICROSECOND ƒ Aggregate (or column) Functions ƒ MAX, MIN, SUM, AVG, COUNT, COUNT_BIG, STDDEV, CORRELATION, VARIANCE … ƒ User Defined Functions (UDFs) ƒ Row functions – return 1 row per invocation ƒ Table functions – return a table of rows per invocation Writing/Debugging QV Queries

ƒ Use DB2 tools to produce working SQL ƒ CLP/Query Editor or other ƒ Use formula language ‘+’ operator to keep SQL readable in selection criteria

“SELECT #noteid, from AS person, delivereddate AS date “ + “ FROM “ + @DB2Schema(@DbName) + “.MAIL “ + “ WHERE form = ‘Reply’ AND “ + “ delivereddate > date(days(CURRENT DATE) – 7) “ + “ ORDER BY deliverreddate DESCENDING “ + “ FETCH FIRST 40 ROWS ONLY”

ƒ Virtualize EVERYTHING YOU CAN (@functions, variables, etc.) Tuning ƒ Indexes ƒ DAV creation allows for single-column indexes to be created ƒ If a column appears in WHERE clause an index becomes more valuable the more data in the DAV ƒ These will not help in ORDER BY processing ƒ Multi-column indexes help with WHERE clause AND ORDER BY processing

CREATE INDEX myidx ON mail_dav_t (delivereddate DESCENDING, form, from, #noteid) ALLOW REVERSE SCANS

SELECT #noteid, delivereddate AS date, form, from AS PERSON FROM mail_dav WHERE deliverreddate > date(days(CURRENT DATE) – 7) ORDER BY delivereddate descending, form, from FETCH FIRST 50 ROWS ONLY

ƒ However, indexes are not free - Updating cost + Disk space Tuning (continued)

ƒCritical DB2 Parameter Settings ƒ Consult nsfdb2cfg16k.bat (or .sh) - the basics ƒ Should be automatically set when Domino/DB2 initializes ƒ Increase bufferpool sizes to use more memory (AIX – 64 bit) ƒ Autonomics – autoconfigure (learns your environment from system load and recommends parameter settings) ƒ See http://publib.boulder.ibm.com/infocenter/db2help/index.jsp - search for autoconfigure Tuning (continued)

ƒ Diagnosis/Measurement – Snapshots ƒ Bring down Domino and DB2 (db2stop) ƒ Bring up DB2 ƒ From CLP: db2 update monitor switches using sort on lock on table on bufferpool on uow on statement on db2 get monitor switches ƒ Run Domino with QV traffic ƒ From CLP: db2 get snapshot for dbm >c:\db2mon.out (or some other filename) db2 get snapshot for all on >>c:\db2mon.out ƒ Look at Dynamic SQL section ƒ Across all SQL statements, compare Total Execution Time and Total Execution Time/Number of Executions Tuning (continued)

ƒAccess plan analysis ƒFrom DB2 CLP ƒrun db2expln –d -q “” –t ƒOr, db2 explain all for , then db2exfmt ƒFrom Command Editor ƒConnect, copy in SQL statement, choose SELECTED- >ACCESS PLAN ƒLook for diamonds - IXSCANS (index scans) generally much cheaper than TABSCANS DEMO

The Ultimate Inbox Related Sessions

ƒ AD211 - Using IBM Lotus Domino 7 with DB2: A Real-World Example ƒ BP307 - IBM Lotus Domino 7 and DB2: Effective Use of Access and Query Views ƒ ID105 - DB2 Setup and Administration for IBM Lotus Domino Server ƒ HND202 - IBM Lotus Domino NSFDB2: Development and Integration Hands-On Q&A