Adm07: The Health Check Extravaganza for Social and Collaboration Environments
Kim Greene, Kim Greene Consulting, Inc. Luis Guirigay, IBM
#engageug 1 Kim Greene - Introduction • Owner of Kim Greene Consulting, Inc. • 15+ years experience with Domino and Sametime and 20+ years of experience with IBM i (AS/400, iSeries) • Services include System & Application performance optimization, Administration, upgrades, health, performance, security etc. checks, migrations, custom development, enterprise integration • IBM Champion • Blog: www.dominodiva.com
@iSeriesDomino http://www.kimgreene.com
#engageug 2 Luis Guirigay - Introduction • WW Executive IT Specialist • Global Technical Ambassador at IBM • Published Author. IBM Redbooks and developerWorks (Domino, DB2, iSeries, Connections, Sametime) • IBM Certification Exams for ICS Products (writer and reviewer) • WW Program Manager for Project Hawthorn (mail support for MS Outlook) • SME – Social, Collaboration, Cloud, Verse and Messaging
Follow me @lguiriga or http://about.me/lguiriga
#engageug 3 Agenda • Proper maintenance • Configuration & best practices • Keeping current a.k.a patching • Monitoring • Security • SmartCloud Notes Hybrid • IBM Mail Support for MS Outlook
#engageug 4 PROPER MAINTENANCE
#engageug 5 Many Moving Parts – What to do?
• Modern Collaborative Systems have many moving parts but which ones are most in need of maintenance and how?
• Let’s look at some of the Systems and all their Moving Parts
#engageug 6 The Bits and Pieces of IBM Domino
These are the individual moving parts that make up your IBM Domino environment:
Main Components: Possible Additions: ● Servers (the OS) ● Transaction logs ● Server function ● .NLOs (application, mail, traveler, ● DB2 etc.) ● Third party products / ● Domino NSFs applications
#engageug 7 .NSF Maintenance
Updall • Updates view indexes • Runs at 2AM by default
Fixup • Check integrity of Domino databases • Resolve corruption problems • Especially important if not using transaction logging
#engageug 8 DBCapture Tool
Automatic identification and collection (i.e. taking them off-line) of corrupt databases without bringing down Domino server • Files renamed to .cor and moved to IBM_Technical_Support folder • Can still run fixup / compact / updall on them!
Enable using server notes.ini: • DATABASE_CAPTURE_ENABLED=1
#engageug 9 DBCapture Tool
Tips: Can invoke manually; ignores Status but respects Capture and Size limits • load dbcapture dbnames.nsf DATABASE_CAPTURE_SIZE_LIMIT (in mbs) sets size of all collected databases • Default: 100 / set to 0 for no limit DATABASE_CAPTURE_LIMIT sets maximum # corrupt DBs to be collected • Default: 10 / set to 0 for no limit Gotcha • DATABASE_CAPTURE_ENABLED value resets every time capture is done, and is enabled again when server is restarted! (i.e. does not run continuously)
#engageug 10 .NSF Maintenance
• Compact • Equivalent to a “Defragment” for a Domino database • Rearrange database or reduce file size • Run with multiple threads via notes.ini • debug_enable_compact_8_5=1
OR
• Use DBMT
• Recent customer example Compacted databases after upgrade, recovered 418 GB of disk space, a 42% reduction!!
#engageug 11 Compact Tips
compact_filter =dbname.nsf • Prevents compact running on specific databases • Ex: compact_filter=log.nsf, names.nsf, admin4.nsf
> load compact -c mail/ladmin.nsf Database 'mail/ladmin.nsf' is not present in the ini parameter 'COMPACT_FILTER'. Proceeding with compact.
• Compact –ODS • Copy style compact if current ODS is less than desired level
• 95% Space Utilization is a good thing
#engageug 12 Compact Replication • Use –REPLICA switch on Compact command • Creates replica of database under the covers while source database remains accessible • Use to remedy “ Insufficient memory ” or “ Unable to extend an ID table – insufficient memory ” errors caused by frequent additions and deletions in a database
• Internally reorganizes IDs in new replica • Avoids ID table fragmentation leading to above errors • Preventative maintenance to avoid fragmentation causing database to become inaccessible
• Maintains Views and Unread Marks between old and new replica
#engageug 13 Derby Database Maintenance • Over time Traveler performance can deteriorate, defrag to restore performance • Steps to start a Defrag manually: • Tell traveler shutdown • Tell http quit • Load traveler -degrag
• Notes.ini variables • NTS_DEFRAG_INTERVAL_DAYS=<# of days> • NTS_LAST_DEFRAG=
#engageug 14 Traveler Database Maintenance
Defragging changes in 9.0.1.8 and later versions of Traveler, use DBMAINT now • Tell traveler dbmaint set interval 7 ** • 11/19/2015 09:37:02 Traveler: DB maintenance will be performed every 7 days. • Tell traveler dbmaint set time 23:00 • 11/19/2015 09:39:58 Traveler: Time of day for DB maintenance has been set to 23:00 • Tell traveler dbmaint set day Sunday • 11/19/2015 09:51:40 Traveler: Day is now configured to Sunday. • Tell traveler dbmaint set auto on ** • 11/19/2015 10:12:27 Traveler: Automatic maintenance of your database has been set. • 11/19/2015 10:12:27 Traveler: The next maintenance is scheduled for 2015-11-22 23:00. • 11/19/2015 10:12:27 Traveler: Maintenance will be performed every 7 days at 23:00. • ** Only options available for Derby database
#engageug 15 The Bits and Pieces of IBM Connections These are the individual moving parts that make up your IBM Connections environment:
Main Components: Possible Additions: ● Servers (the OS) ● Cognos ● WebSphere ● IBM Docs / IBM FileViewer ● DB2 ● IBM Forms/Surveys ● LDAP ● Third Party Products ● IHS ● ICMail ● TDI ● Shared File Space (NAS/NFS, etc.)
#engageug 16 The Bits and Pieces of IBM Sametime
These are the individual moving parts that make up your IBM Sametime environment:
Main Components: Possible Additions: ● Servers (the OS) ● Proxy Servers ● WebSphere ● Integrations with Voice/Video ● DB2 devices ● LDAP ● Integrate Sametime with other ● Domino (Community systems (awareness, meetings, etc.) Server only) ● Third Party Products – IM Queue Managers, etc.
#engageug 17 VPUserInfo.nsf - A Contact List Tune-Up
• Vpuserinfo.nsf can grow very large and make Sametime very slow to login and respond to searches.
• Use a custom agent to look for users no longer registered in the Domino Directory and remove all contact lists for those users.
• If users are seeing partial empty lists: • load fixup vpuserinfo.nsf • load updall -r vpuserinfo.nsf • load compact vpuserinfo.nsf
#engageug 18 DB2 - Three “Rs” rule
• Reorganisation • Recommended after large amounts of data get added
• Runstats • Run often to make sure queries are being executed optimally
• Rebind • Recommended after applying a fix pack or similar
#engageug 19 CONFIGURATION & BEST PRACTICES
#engageug 20 The Tyranny of the “Default”
• Everyone gets an “average” server if they do nothing at all
• It will run, but will it run well?
• Is this acceptable to you?
#engageug 21 Connection Documents
• Key for properly controlling replication • What to replicate Replication type and files / directories to replicate, and avoid • Tip: Are you replicating names.nsf, admin4.nsf, events4.nsf and dir assist db throughout the domain??
• Replication time limit • Tip: Set to less than the repeat interval #engageug 22 Connection Document Settings • Critical to watch connection document settings • Customer example • 09/26/2014 11:52:33 AM {User ABC/ACME} DBStore::GetDB: Unable to open CN=DomMail/O=ACME!!mail\uabc.nsf (Connection denied. The server you connected to has a different name from the … • Connection document was culprit
IP of Source server, not Target!
#engageug 23 Notes.ini Files
• Is there lurking debug still enabled? • Did you really check?? • Consumes valuable resources
• Make sure your notes.ini doesn’t look like this • Debug_threadid=1 • Log_AgentManager=1 • Debug_sem_timeout=10000 • Log_update=2 • NSF_DocCache_Thread=1 • debug_nif=0 • Debug_nif_update=1 • FT_LIMIT_HIGHLIGHT_FILTER=1 • LDAPDEBUG=1 • SMTPDebug=3
#engageug 24 Notes.ini Files • Don’t forget about the configuration document!!
#engageug 25 Notes.ini Files
• Recommended to be enabled at all times:
• CONSOLE_LOG_ENABLED=1 • Captures server console data and logs to console.log file • CONSOLE_LOG_MAX_KBYTES=204800 • Restricts the console Log size to 200MB and then overwrites oldest entries • DEBUG_THREADID=1 • Stamps server threads and logs to console.log file • DEBUG_CAPTURE_TIMEOUT=1 • Captures semaphore time stamp and logs to semdebug.txt • DEBUG_SHOW_TIMEOUT=1 • Captures semaphore information and logs to semdebug.txt
#engageug 26 Transactional Logging • Has been around for years! Remove unread mark • Sequential writing into a log file Remove unread mark
1 2 3 4 5 6 7 8 9 … vs.
• Allows Incremental Backup/Restores • TXN Logs stored on a separate disk controller for best performance (depending on your platform)
#engageug 27 Traveler HTTP Threads and Sizing
• Tell Traveler stat show push.devices.total
Push.Devices.Total = 225 • This indicates that 225 devices are registered for synchronization with the Notes Traveler server and that at least 270 HTTP threads are needed (1.2 x 225 = 270).
Tip: The number of active HTTP threads needed for Traveler is calculated this way: 1.2 x Number of registered devices = Number of needed active HTTP threads
#engageug 28 Sametime MUX • Geographic • Go from 20,000 users to 100,000 per Community
#engageug 29 IBM Connections
• DB2: 64 Bits, 8GB -128GB • Dedicated Storage or high performance disk
• Use a Caching Proxy Server https://ibm.biz/BdHCUh
• DB2 Pool Size
• Content Compression
• More Tuning Tips https://ibm.biz/BdHC5j
#engageug 30 KEEPING CURRENT A.K.A PATCHING
#engageug 31 On Disk Structure
• Don’t forget to upgrade databases to latest ODS level when upgrading servers • What is the ODS about? • Newest internal structure enables database to benefit from newest features • Examples of benefits • R5.0 (ODS41) = participate in transaction logging • R6.0 (ODS43) = LZ1 compression and shared templates • R8.0 (ODS48) = design and document compression • R8.5 (ODS51) = DAOS • R9.0.1 (ODS52) = Performance improvements, better handling of huge (2GB+) attachments
#engageug 32 How to Upgrade On Disk Structure • For server • Copy style compact (compact –c) • Remember compact -ODS • For client • Use policies to update local ODS levels • Push to clients via dynamic policies / or organizational policies • Desktop Settings policy document: Mail tab > “ Enable upgrade for all local NSFs to latest ODS version ” • Gotcha: requires the 8.5.2 Domino Directory on server • CREATE_R(85/R9)_DATABASES=1 • Even better: NSF_UpdateODS=1 (Will keep updating ODS levels as new versions are released) • Tip: Although it’s said to be both server & client side, it only works on the client side!
#engageug 33 Preventing ServerTasksAt Updates
• Tired of losing your ServerTasksAt customizations when upgrading?
• SetupLeaveServerTasks to the rescue • Add SetupLeaveServerTasks=1 to server’s notes.ini • Disables automatic updating of ServerTasksAt#= lines during a Domino Server upgrade
#engageug 34 MONITORING
#engageug 35 Key Items To Keep In Mind When Monitoring All systems require you to cover the basics for all servers involved: CPU, Memory, Disk, Network
When monitoring: • Make it actionable • Know your baseline • Know what your results mean • Investigate!
#engageug 36 Monitoring for Domino
• Pay attention to console messages, don’t ignore them! • admin4.nsf has not replicated (PUSH) with ANY server since MM/DD/YYYY HH:MM:SS ( 1681 hours ago ) • Error validating execution rights for agent 'Notify' in database ‘subdir/dbname.nsf'. Agent signer ‘XXX01/YYY', effective user ‘XXX01/YYY'. Agent signer. • RnRMgr: The design of Rooms.nsf is not one supportable by RnRMgr. Autoprocessing is being disabled for this DB. • Directory Cataloger finished processing DirectoryCatalog.nsf: File does not exist • Agent Manager: Full text operations on database ‘mail/myfile.nsf’ which is not full text indexed. This is extremely inefficient.
#engageug 37 Monitoring for Domino • Health Monitor • Easy to use and provides 24/7 monitoring • Enabled via Administration Preferences
#engageug 38 Monitoring for Domino • Health Monitor • Watch servers on single screen • Monitor servers and/or tasks needing attention
#engageug 39 Monitoring for Domino • tell traveler status Example Yellow status
• Example Green status
#engageug 40 IBM Connections
• CPU Utilization on WAS • If > 70% for 5 minutes or longer = too high
• CPU Utilization on DB2 • If >50% for 5 minutes or longer = too high
• Look for these words in SystemOut log • “Hung” • “Starvation”
#engageug 41 SECURITY
#engageug 42 Security and Collaboration Systems
IBM Connections, Sametime and Domino are made up of individual components that all have separate security concerns and (potential) vulnerabilities.
No system will be 100% secure. If Your Domino/Connections/Sametime environment were your home, what you would look for: 1. Every door of your house has a lock and a deadbolt and every window can be closed and locked. 2. You would not leave a key under the front mat or in the flower pot next to the door. 3. No Notes sticking on the front door detailing which flowerpot to look under for the key. 4. You would have a security light or two and maybe a warning sign of the dangerous attack Chihuahua dog that lives in your house . . .
#engageug 43 Security: Common Sense Questions to Ponder
1. Do you really want to use the same system/generic account for each function? 2. Do you really need the “One Admin Account to Rule Them All”? 3. Do you have so many admins that creating individual admin accounts for them is a great administrative overhead? 4. When assigning rights, are you thinking of “person” or of “job function”? 5. Do you have more than one “person” or “admin type” for each function so you have continuity? 6. Is your brilliant administration scheme actually documented someplace? 7. If you use hierarchical directories (LDAP …, it’s hierarchical) are you taking advantage of it?
#engageug 44 Domino – Protected Groups
• Prevents accidental deletion of designated “critical” groups
• Configured in Directory Profile of the Domino Directory • Tip: You must edit and save once to become operational
• Requires Domino directory to have 9 design
• Defaults to LocalDomainAdmins, LocalDomainServers, and OtherDomainServers
#engageug 45 Domino – Protected Groups
• Open Domino Directory→Actions →Edit Directory Profile
#engageug 46 Domino – Protected Groups • Prevent deletion of these groups
#engageug 47 Internet Access to Domino • Oldie but goodie.....PASSTHRU SERVERS!!! • Separate Domino Domain • Configuration Only Names.nsf
#engageug 48 Lock Down Ports • Lock down ports not using • Number one step for outside attacks • Nmap is great tool for testing open ports
#engageug 49 Lock Down Ports • Ports commonly seen open Port Function 252525 SMTP 808080 HTTP 858585 110110110 POP3 113113113 Authentication service 143143143 IMAP 179179179 Border gateway protocol 389389389 LDAP 443443443 HTTP SSL 465465465 SMTP SSL 541541541 uucp-rlogin Fortimanager and Fortigate server 587587587 Alternate outgoing SMTP 993993993 IMAP SSL 995995995 POP3 SSL 1352 Notes remote procedure call 2050 Java server console 1503 Sametime meeting server listen 1533 Sametime community server listen 8081 Alternate HTTP port 60000 DIIOP #engageug 63148 Remote debug manager 50 Lock Down Ports • Lock down at firewall level • To prevent getting to server • Lock down at server level • In case firewall is not secured properly • Is LDAP, POP3, IMAP, DIIOP, etc. in use? • Enabled by default
#engageug 51 ID Vault • It’s a vault with a secured/encrypted copy of all user ids • You can have multiple vaults • Important: Do not use standard replication for ID Vault replicas • Some of the benefits are: • Lost or forgotten user passwords can be recovered or reset easily • User renames and key rollovers are automated • User IDs are synchronized across machines • No need to carry ID files for new installs • Corrupted IDs are replaced automatically
#engageug 52 IBM MAIL SUPPORT FOR MS OUTLOOK (HAWTHORN)
#engageug 53 Functionality Today • Primary Domino communication via HTTP • Exchange ActiveSync synchronizes all data • Mail, calendar, contacts, folders • REST services: • Out of office • Encryption • Room finder • Quota management • Delegate management • Address book search via LDAP • Native Outlook capability • Any LDAP will work (not just Domino)
#engageug 54 Architecture Guidelines • Outlook users must have a replica on the IMSMO servers • Second IMSMO Server required for HA via Outlook • Load Balancer is also required. Outlook is dumb! • You can build a cluster with IMSMO and non-IMSMO servers • You can use the same DB2 server to host multiple DB2 instances • Think one DB2 server for multiple IMSMO clusters • You must use a proper SSL certificate
#engageug 55 SMARTCLOUD NOTES HYBRID
#engageug 56 Architecture
#engageug 57 Questions
Thank You!!
#engageug 58