Adminp Advanced Topics

AdminP Advanced Topics

Susan Bulloch - IBM ISV Technical Enablement Engineer

IBM Software Group | Lotus software

Agenda …

AdminP history AdminP processing and operations Tuning AdminP Monitoring AdminP Defining best practices Implementing tips and tricks What’s coming in Domino 8 Wrap-up

1 IBM Software Group | Lotus software

What Was AdminP Designed to Do?

AdminP is a server task (adminp) that automates many administrative tasks You initiate the tasks, and the Administration Process completes them for you It was introduced in R4 to manage name changes There were 19 requests when introduced in R4 And now? AdminP is a required server task and an integral part of the Domino system It’s taking on more work with each release 180+ requests in Domino 8 With each release, it’s becoming more efficient, too!

IBM Software Group | Lotus software

What Does AdminP Do?

AdminP automates things that can be done over time Moving files Deleting files Changing names Creating replicas on remote servers It automates things that need to be exact Server build numbers Server port names Client builds If the data needs to be exact, AdminP can often do it

2 IBM Software Group | Lotus software

What Else Can AdminP Do?

Help manage user mail access Allows the user to be set down to Editor Performs various functions that formerly required manager access Sets Out-of-Office status, mail & calendar delegation Manage registration and recertification using the CA process Allows Web-based user registration Allows ID management with no user actions Allows more secure administration Integrates with many 3rd party tools

IBM Software Group | Lotus software

AdminP is Self-Configuring!

If you leave it alone, AdminP configures itself! A database replica stub is created on each new server The ADMIN4.NSF database is created upon startup of each new server The replica ID of Admin4.nsf is based on the Directory Replica ID – So is unique to each environment Replication must be allowed from admin hub to spokes – Either directly or through other hubs If you interfere, it can cause problems Attempts to change Replica ID will usually fail! The replica ID needs to be set as designed

3 IBM Software Group | Lotus software

AdminP Requirements

The AdminP server task must be running on the server Load adminp at startup using servertasks= in the ini file It’s there by default Best practice is to leave it this way There must be an Administration Requests database (ADMIN4.NSF) on each server Users and Admins need appropriate access to this database The databases must be well maintained and replicating properly More details later …

IBM Software Group | Lotus software

AdminP Requirements (cont.)

An Administration server must be set for the Domino Directory as well as ADMIN4.NSF The setting “Do not modify Names fields” for Domino Directory and ADMIN4.NSF is required A copy of CERTLOG.NSF must exist on your administration server You can have copies elsewhere too if you wish An administration server set in the ACL of databases Any database that you want AdminP to maintain You probably want them all maintained There’s a command to know if all databases are set properly – Details in a few minutes

4 IBM Software Group | Lotus software

Where AdminP Works

On the administration server of a database Changes are made on this copy of a database This minimizes chance of replication conflicts On the administration server of the Directory Often the “main” server of a system “All Servers” or * Every server in the domain For example, name changes are processed by all servers “Named” server A specific server to perform a request For example, the move replica request works on the “target” server for the move, but no others

IBM Software Group | Lotus software

Processing Requests

Most processes are timed Sequential actions trigger the next action Process continues until all steps are complete The shortest interval is one minute (immediate requests) Something starts the process Usually an Administrator Examples of admin-initiated processes are user renames, deletions, replica creation A response is expected by Domino Example: User authenticates with home server, replica stub created on target server The next step is started Example: Unread marks change, group entries change

5 IBM Software Group | Lotus software

Processing Requests (cont.)

The processes continue Some can continue for a week But you can speed this up – There’s even more control in Domino 8 Some processes can stay active for more than a week Mail moves Name change requests – These are monitored and controlled in Person Documents – You do NOT need to keep documents 21 days in ADMIN4.NSF – Any processes that need to continue will re-start based on the person documents

IBM Software Group | Lotus software

Automated Processing — Almost

Interim steps sometimes require human touch Anything affecting Directory documents or database files Also name change reversions! Anything that must be approved along the way by someone with rights to the document or file In other words, an Administrator: – This allows delegation to less experienced employees – Protects Directory data from employees in groups who are not Notes Admins – Security teams often perform renames – They often have limited Domino training – This tiered approval process protects your system

6 IBM Software Group | Lotus software

Examples of AdminP Tasks

Delegate mail files Set end-user agents to run Manage CA administration Manage roaming users Create and rename rooms and resources Find users Manage policies Change HTTP password Create new mail files in the background

IBM Software Group | Lotus software

Examples of AdminP Tasks (cont.)

Add/remove servers in cluster Change user password in Domino Directory Add Internet Certificate to Person Record Configure Domain Catalog Enable server’s SSL ports in Domino Directory Move mail files Rename groups But this isn’t all …

7 IBM Software Group | Lotus software

AdminP Operations

Every hour, by default, AdminP checks for work This is a tuneable parameter Only requests that are “new” are processed on a server On AdminP start-up, task requests with no response document (log) or entry in a hidden ID table are flagged to be processed When AdminP is already running, new entries (based on time/date stamp) to the ADMIN4 database are flagged This can cause problems if “old” data is replicated back into newer databases – This must be prevented – We’ll tell you how

IBM Software Group | Lotus software

AdminP Operations (cont.)

Immediate requests are performed within a minute of posting to the ADMIN4 database No option exists to change the immediate request interval (1 minute) Typically these requests should be processed quickly: Create replica Change user password in Domino Directory Update client information in Person Record Change HTTP password in Domino Directory Immediate requests are denoted in ADMIN4 with a “bolt” icon

8 IBM Software Group | Lotus software

AdminP Batched Requests

These were introduced in 6.0 to increase efficiency They perform certain modifications for many users at once The database is accessed once Several user changes can be made Example: Four user names can be changed in the Access Control List (ACL) Pre-batch methods caused 4 accesses Currently 18 requests are batched Interval times should be long enough to accumulate multiple batch types Interval should be short on the admin hub, longer on spokes If the interval is too long, the server won’t have time to accumulate similar requests.

IBM Software Group | Lotus software

List of Batched Requests

Rename in ACL Delete in Person Documents Delete in ACL Delete in Reader/Author fields Rename in Person Documents Rename in Reader/Author fields Rename Group in ACL Rename Group in Reader/Author fields Rename Person in Unread List

9 IBM Software Group | Lotus software

List of Batched Requests (cont.)

Rename Web User in ACL Rename Web User in Person Documents Rename Web User in Reader/Author fields Rename Web User in Unread List Delete Person in Unread List Rename in Design Elements Delete in Design Elements Rename Web User in Design Elements Rename Group in Design Elements

IBM Software Group | Lotus software

AdminP — The Database (ADMIN4.NSF)

Contains processing action requests AdminP requests Contains processing action results Known as AdminP response (log) Administration approval requests are there also Examples: Confirm database deletion Certification requests for change hierarchy Provides views to help with troubleshooting Use Domino Domain Monitoring (DDM) to monitor the database in Domino 7! Finds stalled requests

10 IBM Software Group | Lotus software

AdminP – The Database (ADMIN4.NSF) (cont.)

AdminP is designed to be managed Workflow requires attention/approval The database will grow in size if ignored Sufficient access is needed Default is Author with Create for users Can be No Access in ND6 and later – Requests from users are mailed to the database – Default mail-in database is called Administration Requests Can use wildcard if Default needs to be No Access Administrators need Author, minimum Editor access to approve requests

IBM Software Group | Lotus software

AdminP – The Database (Admin4.nsf) (cont.)

Proper replication is required Admin4.nsf should replicate as often as Directory The size can grow unacceptably if it doesn’t Replication retention should be standardized The default is 7 days 10 is acceptable, as is 14 or 21 Anything longer is unnecessary and dangerous! Improper replication causes old requests to “come back” Causes server slowdowns – Replication “storms” can occur This is the number one cause of AdminP meltdowns! Easily controllable, preventable

11 IBM Software Group | Lotus software

Tuning the AdminP System

Default settings will work in small companies AdminP default interval is 60 minutes Every hour, AdminP checks for work to be done Daily processes run at midnight Delayed processes run on Sunday at midnight Because they are processor intensive Large organizations need to tune the AdminP system Virtually everything is configurable Start in the Server Document

IBM Software Group | Lotus software

Deep Dive into Tuning: Server Document Settings

12 IBM Software Group | Lotus software

Deep Dive into Tuning: Server Document Settings (cont.) Interval Default is 60 minutes (blank in Server doc) You can reduce this as needed 15 minutes on administrative server is acceptable Be sure to increase replication interval also Store Admin Process log entries when status of no change is recorded Change from “Yes” to “No” This will reduce the admin4.nsf database size By as much as 20%! “No” is the default beginning in Domino 6.5.5, 7.0

IBM Software Group | Lotus software

Deep Dive into Tuning: Server Document Settings (cont.)

Delayed Request Settings The default is Sunday Consider running these requests more often This is the Reader/Author name change You can run this every night Delayed requests generate messages in the server log 18-10-2002 19:57:04 Begin MIME to CD Conversion (Process: ? (000004C4:00000002), Database: D:\data\mail\xxx.nsf, Note: 0000766E) Set converter_log_level=10 in server ini file to shut off these messages It’s AdminP preparing data to work on It was always there but not always logged

13 IBM Software Group | Lotus software

Deep Dive into Tuning: Server Document Settings (cont.)

Maximum number of threads Multiple threads are supported Default is 3, maximum is 10 One thread is used to dispatch requests Three threads to process the requests Threads are only activated when required to process request Test incrementally if you increase Notes 8 offers more thread options

IBM Software Group | Lotus software

Tips for Tuning

Speed up replication Especially if you reduce interval timing Requests will replicate out faster, be processed quicker Skipping databases Reader/Author name renames take a long time — they’re resource intensive Skip databases using $Adminp hidden view Use a selection formula to show only documents with Reader/Author fields – All others are skipped If view is blank, the entire database is skipped You can see a sample in PERNAMES.NTF – Modify to suit your needs

14 IBM Software Group | Lotus software

Tuning Tools: Server Console Commands

You may need to use Server Console command when troubleshooting Use with caution unless you’re sure of the impact Tell AdminP Process New Processes all new requests Use to jump-start a process Use this one instead of almost any other you want to use Tell AdminP Process People Processes Person Document changes Tell AdminP Process Time Used for shared mail systems Used for load balancing mail moves

IBM Software Group | Lotus software

Tuning Tools: Server Console Commands (cont.)

Tell AdminP Process All Processes all new and modified requests Includes immediate, interval, delayed, and daily requests This is probably not what you want to do when using this command Causes requests to back up until “ALL” are finished Use with extreme caution Never use during production hours Tell AdminP Process Daily Processes all new and modified daily requests to Person Documents

15 IBM Software Group | Lotus software

Tuning Tools: Server Console Commands (cont.)

Tell AdminP Process Delayed Processes all new and modified delayed requests Based on start executing on/at setting This is a “Sunday morning process” because it is processor intensive But it doesn’t delay new requests Like Tell AdminP Process All does Tell AdminP Process Interval Processes all immediate and interval requests

IBM Software Group | Lotus software

Tuning Tools: Server Console Commands (cont.)

Tell AdminP Show Databases Lists databases with and without a designated admin server See your server log for the list You can ensure all databases are protected this way Tell AdminP Process MA Validates whether mail policies were updated Not a new request type, but a new AdminP thread (Domino 7 only) Tell AdminP Quit Stops AdminP task Load AdminP Starts AdminP task

16 IBM Software Group | Lotus software

Bonus Trick: How You Can Use AdminP

Tell AdminP Process Daily example You change a user’s name using AdminP The process rolls along The user calls you saying, “My unread marks are all messed up! You broke my Lotus Notes!” You tell the user “I can fix this. I need you to log out of Notes for 10 minutes” I’m thinking we should tell them to turn off the PC just to be sure You type “tell adminp process daily” at the Server Console When the user logs back in, the unread marks are fixed

IBM Software Group | Lotus software

Monitoring AdminP

AdminP is designed to be managed Some database views offer you information Administrative attention required These are informational, there’s a button to remove them from view Some end-user notifications can be automated – Select Action – Enable/Disable end-user notification Other views require an action Individual approval required File deletions require approval Name change reversions – No more “21-day” issue Pending by age/server will show older requests that may need attention today

17 IBM Software Group | Lotus software

Monitoring AdminP (cont.)

Documents that need attention or action will stay in the database until: You look at them or You process them or You delete them They are protected by a $NoPurge Field Your database will grow and grow Assign rotating responsibility for ADMIN4 monitoring Or let the new admins do it all!

IBM Software Group | Lotus software

New Feature for AdminP: DDM

DDM (Domino Domain Monitoring) can monitor the progress of requests Monitors 11 different types of AdminP requests See me later for how to add more New in Domino 7 The default server probe is the “Administration” type Any error in AdminP processing will create a notification in DDM Stalled rename requests will notify DDM You don’t have to monitor the database as closely But you have to start using DDM

18 IBM Software Group | Lotus software

New Feature in AdminP: DDM (cont.)

AdminP requests monitored by default in DDM

IBM Software Group | Lotus software

Best Practices

Learn from the mistakes of others The ADMIN4.NSF database must replicate throughout your system It must have the Replica ID assigned by Domino Old or test servers should not exist in production domains ADMIN4.NSF exists on all servers When old servers are turned back on, databases replicate In addition to ruining NAMES.NSF, you ruin ADMIN4

19 IBM Software Group | Lotus software

Best Practices (cont.)

Never restart a server that has been out of service for more than the purge interval of ADMIN4 Old documents replicate back in Old requests are read by AdminP Servers send error messages stating that the requests are too old Customers have clogged their systems this way Never run test servers in your production domain They, too, have a copy of ADMIN4.NSF

IBM Software Group | Lotus software

Best Practices (cont.)

Keep the database size down Do it for your server Process the requests that require your touch regularly Monitor replication Rules of thumb All copies should have the same Replica ID and ACL All copies should be nearly the same size Number of documents should be nearly the same Exceptions: – Admin server can store more information – If you use a selective replication formula, sizes will differ

20 IBM Software Group | Lotus software

Selective Replication

Selective replication formulas can help in large systems They work best when created and maintained on the spoke servers You’ll need a process to add these when the database is replaced – Customers who use them, love them This limits the size of the spoke databases Also limits the amount of data replicated Especially useful over slow links Admin hub receives all requests, so can do the processing needed Designed to allow the spoke server to receive only what it needs Anything it or cluster mate needs to process Spoke will send anything it originates to the admin hub

IBM Software Group | Lotus software

Selective Replication (cont.)

Sample code All disclaimers apply with this code TEST, TEST, TEST

SELECT @Contains(@UpperCase(ProxyServer) ; ″server":″clustermate" ) | @Contains(@UpperCase(ProxyServerName) ; ″server":″clustermate" ) | @Contains(@UpperCase(ProxyActionRequestor) ; ″server":″clustermate" ) | @Contains(@UpperCase(InboundReplicaServers) ; ″server":″clustermate" ) | @Contains(@UpperCase(ProxyServer) ; "*" ) | @Contains(@UpperCase(ProxyServerName) ; "*" )

21 IBM Software Group | Lotus software

Tweaking Name Changes

Increasing the time a user can accept name changes Necessary in Europe Change the default Allowable values are 14 to 60 Allows the user to go on holiday

IBM Software Group | Lotus software

Names Fields

Use caution when implementing feature: All Names fields Using the “Modify All Names Fields” in ACLs may have unexpected effects If used in mail files, AdminP will remove users from “Sent” fields when you delete users Do NOT change the default AdminP settings in mail database (or in the Domino Directory) – Everything is coded to work as set by Domino/Notes If used in other databases, the Creator name is removed This could be a compliance issue One more thing If the last person in any Reader/Author field is removed, the document becomes public Use this feature with care!

22 IBM Software Group | Lotus software

Programmability

Custom AdminP code can be written in LotusScript Notes Administration Process Class Introduced in Domino 6.0 There are 6 properties and 39 methods Useful you want to automate certain things Like user-generated rename processes Use with caution and test your code Problems have occurred with third-party tools that weren’t thoroughly tested

IBM Software Group | Lotus software

Things to Watch Out For

Renames can take a long time Semaphore gets locked doing ACL changes Other changes cannot be processed Fixed in 6.5.4 with code and ini setting TN 1174405 ADMINP_ENABLE_CASCADE_DESIGN_ELEMENTS=1 Mail file moves to a large, empty SAN using AIX can fail AdminP reports insufficient disk space Fixed in 7.0 Had problems with scientific notation

23 IBM Software Group | Lotus software

Things to Watch Out For (cont.)

Notes has problems with short names in Location documents Both AdminP and Dynamic Client Configuration have failed if the server name is short Example: Notes1 instead of Notes1/Acme CA-Process registered users have certificates in ADMIN4.NSF Not in certlog This can create a lot of documents IBM/Lotus is researching this

IBM Software Group

AdminP Improvements in Domino 8

24 IBM Software Group | Lotus software

Direct Deposit of AdminP Requests

Works for the “Named Server” requests Mail file moves, etc. Replication of ADMIN4 is skipped If a connection is available Reduces replication and time lag Speedy If a direct connection is not available Regular process occurs You can disable it ADMINP_DONT_ATTEMPT_DIRECT_DEPOSIT=1

IBM Software Group | Lotus software

Special Purpose Threads

Remember the maximum number of threads for AdminP? It’s 10, with a default of 3 In Domino 8, you can specify some of those 10 threads to certain process types ADMINP_IMMEDIATE_THREAD=X ADMINP_INTERVAL_THREAD=X Works like an overflow valve Only used when needed Only used for those 2 types of requests Other types are processed normally

25 IBM Software Group | Lotus software

Override Default Run Intervals

Use this with care Can cause problems if done wrong If you want to change how certain items run, you can: ADMINP_IMMEDIATE_OVERRIDE = x, x, x ADMINP_INTERVAL_OVERRIDE = x, x, x ADMINP_DAILY_OVERRIDE = x, x, x ADMINP_DELAYED_OVERRIDE = x, x, x Domino 8 Admin Help has the list of numbers

IBM Software Group | Lotus software

Override Default Run Intervals (cont.)

Why would you do this? Use to change actions like “Rename in Unread List” to Interval instead of Daily ADMINP_INTERVAL_OVERRIDE = 68.00 If you’re doing a lot of name changes Change Rename in Person Documents to Immediate instead of Interval – ADMINP_IMMEDIATE_OVERRIDE=16.00 You’ll fly through the changes!

26 IBM Software Group | Lotus software

Improved Rename Processing

A new, per database names list If a name being processed is not in this list, the database is skipped Limited to 4K per database No support for “Modify All Names Fields” choice in ACL Requires optional new ODS ODS change is not automatic or required You have to enable it with an ini setting Create_R8_Databases=1 Then run copy-style compact

IBM Software Group | Lotus software

Synchronize Unread Marks

Inconsistencies are caused by AdminP replica creation methods Manual per-user synchronization via Notes Client is not practical Create and move replica In 7.02, ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1 In Domino Admin 8.0, “Exchange Unread Marks” is a UI option Move mail file In 7.02, ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1 In 8.0, automatic synchronization Synchronization may impact overhead Mail files With limited users, synchronization should have limited impact Applications With numerous users, this may significantly change creation time

27 IBM Software Group | Lotus software

Database Redirect

Domino 8.0 introduced “Database Redirect File” (.NRF) Placeholder file directs client to the new database Clean up stale bookmarks and open alternate replica Found in the Admin Client “Move Database” Tool Optionally create the redirect to new replica Admin Client “Delete Database” Tool

IBM Software Group | Lotus software

Database Redirect (cont.) New Admin Client Processes also Create “Database Redirect File” Update “Database Redirect File”

28 IBM Software Group | Lotus software

Automatic Inbox Maintenance

There is a significant decrease in server I/O with small inboxes For information about the impact of large inboxes: http://www.ibm.com/developerworks/lotus/library/ notes-mail-files/ You beg and plead for users to file mail in folders They never do We give you a new tool AdminP will move the mail for them Age-based document trimming via mail policies or Server document WARNING: Get management permission first!

IBM Software Group | Lotus software

Automatic Inbox Maintenance (cont.)

AdminP poll thread executes LotusInboxCleanup mail file agent Tell adminp process mb This task does not remove documents from the mail file They will still be available in All Documents view Your users will still call you It may take a while to get permission But you now have a tool to use

29 IBM Software Group | Lotus software

Improved Server Commands

tell adminp process all Changed in 8.0 Requeue all new and modified requests – No waiting for requests to finish tell adminp process restart Waits for all requests to finish, rebuilds all queues Formerly, tell adminp process all did this Use with care, not during prime hours

IBM Software Group | Lotus software

For More Information about AdminP

Technotes Knowledge Collection — the Administration Process in Domino 6.0x and 6.5x http://www.ibm.com/support/docview.wss?uid=swg21213224 Frequently Asked Questions — the AdminP Process http://www.ibm.com/support/docview.wss?rs=899&uid=swg21212760 developerWorks articles “All About AdminP,” Parts 1 and 2 http://www.ibm.com/developerworks/lotus/library/ls-AllAboutAdminP_1/ http://www.ibm.com/developerworks/lotus/library/ls-AllAboutAdminP_2/index.html LotusScript: The NotesAdministrationProcess Class in Notes/Domino 6 http://www.ibm.com/developerworks/lotus/library/ls-LS_AdminProcess/ Creating a Custom Administration Process Request Handler http://www.ibm.com/developerworks/lotus/library/ls-Custom_AdminP_Handler/

30 IBM Software Group | Lotus software

Questions?

How to contact me: Susan Bulloch [email protected] http://notesgoddess.net