DominoDefrag v3.0 Release Notes Sep. 12 Overview
DominoDefrag is an open source C API Lotus Domino server task I developed using Visual Studio C++. It uses "contig.exe" (v1.55) to defrag Domino databases on all 32bit and 64bit Windows server versions and will also defrag a full-text index associated with a Notes database and the Domino server’s transaction log and DAOS files. The output is recorded to the Domino server console and miscellaneous events log.
DominoDefrag has the added functionality of being able to compact a database prior to defragging and supports multi-processing (can load multiple times to run concurrently) and use of an indirect file (.ind) to have the same compact batch functionality.
Performance checks can also be tested using generated document collections to determine the before and after defrag millisecond read performance of databases and their associated full-text indexes.
DominoDefrag now incorporates “jkdefragcmd.exe” (v3.36) for Domino data volume free space consolidation and “fast sort” file placement optimization. In conjunction with “contig.exe” and the compact task (ncompact.exe) they provide the functionality outlined below:
Order Utility Description 1 jkdefragcmd.exe Consolidates free space (-a5), alphabetically sorts all file placements (-a7) and performs a complete defrag and fast sort optimization (-a3) of the Domino data volume.
It runs as a separate function and can take some time to perform. Runs when DominoDefrag_OptimizeDataVolume=1
2 ncompact.exe Compacts a database when DominoDefrag_Compact=1. If DominoDefrag_FreeSpace=x also used only compacts the database that has > x% unused space
3 contig.exe Optimizes a database to have x% unused space (free space) = 100%-x used space when DominoDefrag_FreeSpace=x
4 contig.exe Defrags a database. When DominoDefrag_FreeSpace=x ensures future Domino writes will be made into the free space residing on one contiguous fragment. It will defrag a database’s full-text index when DominoDefrag_FTIndex=1, the Domino server’s transaction log when DominoDefrag_TransactionLog=1 and the Domino server’s DAOS files when DominoDefrag_DAOS=1
5 jkdefragcmd.exe Performs a “fast sort” file placement optimization of database, full-text, transaction log or DAOS files when DominoDefrag_OptimizeFiles=1
1 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Usage and Copyright Disclaimer
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Please read more about the details of the GNU Affero General Public License @ http://www.gnu.org/licenses/ and in the distribution’s “GNU AFFERO GENERAL PUBLIC LICENSE.txt” file.
Copyright © 12 September 2010
Andrew Luder ([email protected]) http://notestools.com and http://www.linkedin.com/in/andrewluder)
DominoDefrag Enterprise Edition
The "DominoDefrag Enterprise Edition" product comprises a mail-in database with file name "DominoDefragEE.nsf" and business support for it and the current "DominoDefrag.exe" (provided on OpenNTF) under an Enterprise Support Agreement (ESA) upon full fee payment.
A fully functioning “DominoDefragEE.nsf” mail-in database can be downloaded from ftp://notestools.com/DominoDefragEE.zip , but has limited support and can only process one Server Defrag Profile and run on one Domino server within your domain. A new “DominoDefrag.nsf” mail- in database will be provided configured with owner license details upon full fee payment.
The new DominoDefrag_EnterpriseSupport INI parameter specifies that the “DD_ServerLog.csv” and “DD_DatabaseLog.csv” files should be created and attached to the summary email for use by the DominoDefrag Enterprise Edition mail-in database. Only works when the mail-in database e-mail address is specified using the DominoDefrag_SummaryEmailAddress parameter.
If you would like to get an obligation free quote to purchase the DominoDefrag Enterprise Edition (price includes one year business support and software maintenance), please complete the Contact form (also providing the number of Domino server licenses you require) at https://notestools.com/index.php?p=1_16 .
Note: Other Information such as your Domino Domain name and number of “DominoDefragEE.nsf” instances will be requested if you decide to purchase the "DominoDefrag Enterprise Edition" product.
2 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Domino Data Volume Free Space Consolidation and Optimization
A question that’s been posed to me several times is why does “contig.exe” only do a partial defrag of some large Domino databases? The answer is "contig.exe" simply looks to move the fragmented database to the first contiguous gap (defragmented free space) on the disk volume that is big enough. If there is no gap big enough then "contig.exe" will reduce the number of fragments in the file by moving as much of the file as possible into the largest gaps available.
The DominoDefrag server task now uses the "JkDefragcmd.exe” utility to consolidate and optimize the free space on the Domino data volume in preparation, so that these partial database defrags will be complete database defrags next time “contig.exe” processes them. It has been implemented to operate as a separate function and I’d recommend something you would perform once a month.
Figure 1 below shows a Windows 2000 Domino data volume which “contig.exe” has been running on for a while.
Figure 1 - Windows 2000 Domino Data Volume using “contig.exe”
Figure 2 below shows the result of using the DominoDefrag_OptimizeDataVolume=1 parameter to tell DominoDefrag to use "JkDefragcmd.exe” to consolidate free space (-a5), alphabetically sort all file placements (-a7) and perform a complete defrag and “fast sort” optimization (-a3) of the Windows 2000 Domino data volume. A similar strategy is recommended on the http://kessels.com/jkdefrag/ page.
3 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Figure 2- Windows 2000 Domino Data Volume using “JkDefragcmd.exe”
The remaining issue is that some of the free space can be fragmented and I am looking at other free command line tools like Defraggler (http://www.piriform.com/defraggler) that have the functionality to defragment this free space available in a future DominoDefrag version.
Optimizing Database Free Space
A Notes database is proportioned into used and unused space (also called free space). Free space is only recovered when it’s compacted; otherwise Domino will continue writing data into the free space until it’s full and has to allocate more. Defragging a database after optimizing the free space (unused space) ensures future writes will be made into the free space that resides within the database on one contiguous fragment.
The new DominoDefrag_FreeSpace Notes INI parameter attempts to optimize the free space (unused space) to the percentage specified and will only allow a compact to occur if there's a greater percentage free space.
The new DominoDefrag_FreeSpaceMaxMB ensures that the amount of unused space optimized (new free space that can be pre-allocated) is limited to the specified megabyte amount (when not specified it defaults to 1MB). For example, if DominoDefrag_FreeSpace=10 and set to DominoDefrag_FreeSpaceMaxMB=5, only 5MB of unused space is optimized for a 1GB database and not 100MB. This is vital because it allows you to estimate increased disk usage with certainty, because in the worst case, 100% used space would mean at most the database size would only increase by 5MB.
4 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Optimizing File Placement
Performs a “fast sort” file placement optimization of database, full-text, transaction log or DAOS files when DominoDefrag_OptimizeFiles=1 after defragmentation has occurred. The “fast sort” file placement optimization attempts to keep Domino files in the same zone – that is together, closer to the beginning of the hard disk and therefore means the hard disk head doesn’t have to travel as far to get the Domino data. Therefore, faster I/O!
Note: Optimizing file placement is discussed more on the http://kessels.com/jkdefrag/ page.
Donations
Please click here to make donation if you feel DominoDefrag is worth supporting. No amount is too small - it all helps :-)
5 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Version History
Version Changes 3.0 DominoDefrag_OptimizeDataVolume Notes INI parameter specifies that “jkdefragcmd.exe” is used to consolidate free space (-a5), alphabetically sort all file placements (-a7) and perform a complete defrag and fast sort optimization (- a3) of the Domino data volume. It runs as a separate function and can take some time to perform. Does not work on NT4. DominoDefrag_OptimizeFiles Notes INI parameter specifies that “jkdefragcmd.exe” is used after defragmentation to "fast" sort file placement on the Domino data volume. Does not work on NT4. DominoDefrag_EnterpriseSupport INI parameter specifies that the “DD_ServerLog.csv” and “DD_DatabaseLog.csv” files should be created and attached to the summary email for use by the DominoDefrag Enterprise Edition mail-in database Fixed bug NOSS-87ED49 to check database quota size limit and skip adding more free space if exceeds Added summary information counters to report both before and after fragments averages for databases and full-text indexes 2.5 DominoDefrag_FreeSpace Notes INI parameter specifies a percentage value between 1 and 100 used to optimize free space within a database to reduce future fragmentation and that a compact can only occur if there's % greater free space DominoDefrag_FreeSpaceMaxMB Notes INI parameter specifies a maximum megabyte value that is used to limit the amount of new free space that can be pre-allocated. Only works if DominoDefrag_FreeSpace is used. Defaults to 1MB if not specified Fixed bug NOSS-86EQHL on database limit issue as identified limitation with Notes API List functions and replaced with C++ native List functions Fixed bug NOSS-862NYH on terminated abnormally error by using the _pclose() function in return code Added command line argument information into the summary email
6 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 File Information
The C API DominoDefrag 32bit server task requires the following files installed in the Domino program directory on a Windows NT4/2000/2003/2008 32 bit server:
File Source contig.exe http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx (must use “contig.exe” version < 1.55 when running on Windows NT4) DominoDefrag.exe In 32bit folder in the “DominoDefrag v3.0.zip” distribution file JkDefragCmd.exe In the “JkDefrag-3.36.zip” distribution file at http://kessels.com/jkdefrag/ (does not work on Windows NT4)
The C API DominoDefrag 64bit server task requires the following files installed in the Domino program directory on a Windows 2003/2008 64bit server:
File Source contig.exe http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx DominoDefrag.exe In 64bit folder in the “DominoDefrag v3.0.zip” distribution file JkDefragCmd.exe In the “JkDefrag-3.36.zip” distribution file at http://kessels.com/jkdefrag/
7 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Notes INI Setting Information
Domino server Notes INI parameters are:
Notes INI Parameter What it does DominoDefrag_Directory Defaults to root data directory if not specified DominoDefrag_Directory = mail Same as “c:\lotus\domino\data\mail”. Exits if not found
DominoDefrag_Directory = names.nsf Same as “c:\lotus\domino\data\names.nsf”. Exits if not found DominoDefrag_MinuteInterval = 60 Runs the server task continuously to the specified minute interval. Will default to 0 (off) if not specified DominoDefrag_Runtime = 30 Server task stops safely after the specified amount of minutes has elapsed and when the current database defrag has completed. Will default to 0 (off) if not specified DominoDefrag_FTIndex = 1 Will defrag associated full text index folder's file(s). Will default to 0 (off) if not specified DominoDefrag_StartPoint = 10 Specifies the average number of fragments required before a database or full-text index is defragged. Will default to 2 if not specified DominoDefrag_VerboseLogging = 1 Controls amount of server console information. Will default to 0 (off) if not specified DominoDefrag_Compact = 1 Allows in-place compact before defrag. Will default to 0 (off) if not specified DominoDefrag_CompactOptions = -C Allows use of "ncompact.exe" options and overrides the default behaviour. Will only work when DominoDefrag_Compact=1 and 2nd command line parameter not specified DominoDefrag_SummaryEmailAddress = Sends summary e-mail for server to specified e-mail World Administrator address. Will not send email if not specified DominoDefrag_SummaryEmailFrom = Sends summary e-mail for server from specified e-mail [email protected] address. Will default to “DominoDefrag” if not specified DominoDefrag_TransactionLog = 1 Will defrag transaction log files under the Notes INI "TRANSLOG_Path" directory value. Will default to 0 (off) if not specified DominoDefrag_DAOS = 1 Will defrag DAOS files under the Notes INI "DAOSBasePath" directory value. Will default to 0 (off) if not specified DominoDefrag_PerformanceCheck = 1 DominoDefrag_PerformanceCheck Notes INI tests before and after defrag millisecond performance of databases and their associated full-text indexes using generated document collections. Will default to 0 (off) if not specified DominoDefrag_FreeSpace=1 Specifies a percentage value between 1 and 100 used to optimize free space within a database to reduce future fragmentation and that a compact can only occur if there's greater % free space. Will default to 0 (off) if not specified DominoDefrag_FreeSpaceMaxMB Specifies a maximum megabyte value that is used to limit the amount of new free space that can be pre-
8 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Notes INI Parameter What it does allocated. Only works if DominoDefrag_FreeSpace is used. Defaults to 1MB if not specified DominoDefrag_OptimizeDataVolume=1 Specifies that “jkdefragcmd.exe” is used to consolidate free space (-a5), alphabetically sort all file placements (- a7) and perform a complete defrag and fast sort optimization (-a3) of the Domino data volume. It runs as a separate function and can take some time to perform. Does not work on NT4. Will default to 0 (off) if not specified DominoDefrag_OptimizeFiles=1 Specifies that “jkdefragcmd.exe” is used after defragmentation to "fast" sort file placement on the Domino data volume. Does not work on NT4. Will default to 0 (off) if not specified DominoDefrag_EnterpriseSupport=1 Specifies that the “DD_ServerLog.csv” and “DD_DatabaseLog.csv” files should be created and attached to the summary email for use by the DominoDefrag Enterprise Edition mail-in database. Will only work if DominoDefrag_SummaryEmailAddress is specified. Will default to 0 (off) if not specified
9 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Run Information
The DominoDefrag server task can be run from the Domino server console, a program document, or the Notes INI servertask line. Server task parameters are (override Notes INI equivalents):
DominoDefrag.exe
Defrag/Compact Requirement Load DominoDefrag statement example Defrag all directories Load Dominodefrag Defrag all directories with compact Load Dominodefrag “ “ –c Defrag all directories with compact Load Dominodefrag “ “ “–B –S 10” containing space(s) in options
Defrag subdirectory Load Dominodefrag mail Defrag subdirectory with wildcard Load Dominodefrag mail\a*.nsf Defrag subdirectory with compact Load Dominodefrag mail -c Defrag subdirectory with compact Load Dominodefrag mail “–B –S 10” containing space(s) in options Defrag subdirectory containing space(s) Load Dominodefrag “mail\company x” Defrag subdirectory containing space(s) with Load Dominodefrag “mail\company x” -c compact Defrag subdirectory containing space(s) with Load Dominodefrag “mail\company x” “–B –S 10” compact containing space(s) in options
Defrag nsf/ntf/dir/box file Load Dominodefrag names.nsf Defrag nsf/ntf/dir/box file with wildcard Load Dominodefrag mail*.box Defrag nsf/ntf/dir/box file with compact Load Dominodefrag mail85.ntf -c Defrag nsf/ntf/dir/box file with compact Load Dominodefrag mail.box “–B –S 10” containing space(s) in options Defrag nsf/ntf/dir/box file containing Load Dominodefrag “tdrive link.dir” space(s) Defrag nsf/ntf/dir/box file containing Load Dominodefrag “tdrive link.dir” -c space(s) with compact Defrag nsf/ntf/dir/box file containing Load Dominodefrag “tdrive link.dir” “–B –S 10” space(s) with compact containing space(s) in options
Defrag indirect file Load DominoDefrag files1.ind Defrag indirect file with compact Load DominoDefrag files1.ind -c Defrag indirect file containing space(s) with Load DominoDefrag “files 1.ind” “–B –S 10” compact containing space(s) in options
Figure 3 - Program document example
10 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Developer Workstation Setup Information
Server Requirement OpenNTF Published Document Choice(s) Compile to work with “DominoDefrag Visual Studio 2005 32bit Workstation Setup.pdf” Windows NT4 “DominoDefrag C API Visual Studio 2005 C++ 32bit Workstation Setup.pdf” Compile to work with “DominoDefrag Visual Studio 2005 32bit Workstation Setup.pdf” Windows “DominoDefrag C API Visual Studio C++ 2010 Express Edition 32bit 2000/2003/2008 Workstation Setup.pdf” 32bit versions “DominoDefrag Visual Studio 2010 32bit Workstation Setup.pdf” Compile to work with “DominoDefrag C API Visual Studio 2010 C++ 64bit Workstation Windows 2003/2008 Setup.pdf” 64bit versions “DominoDefrag Visual Studio 2010 64bit Workstation Setup.pdf”
Independent Review
Over a 7 month period, Spanish Lotus User Group (ESLUG) blogger Albert Buendia road tested DominoDefrag (versions 1.4-2.1) and presents his comprehensive findings in the PDF files below @
English version -> http://www.slug.es/blogslug.nsf/dx/defragging-domino-servers- v12.pdf/$file/defragging-domino-servers-v12.pdf Spanish version -> http://slugmail.slug.es/BlogSlug.nsf/dx/desfragmentando-servidores- domino-v2.pdf/$file/desfragmentando-servidores-domino-v2.pdf
Other Related information
Contig.exe (v1.55) By Mark Russinovich (Sysinternals) - http://technet.microsoft.com/en- us/sysinternals/bb897428.aspx JkDefragCmd.exe (v3.36) By Jeroen Kessels - http://kessels.com/jkdefrag/ Lotus C++ API toolkits - http://www.ibm.com/developerworks/lotus/downloads/toolkits.html IBM Technote 1229817 - "Slow response from a Domino server when attempting to open a Lotus Notes database or switch views in the database" - http://www- 01.ibm.com/support/docview.wss?rs=899&uid=swg21229817 Lotus Notes and Domino wiki article - "IBM Lotus Domino Server performance: Hard disk fragmentation" - http://www- 10.lotus.com/ldd/dominowiki.nsf/dx/01152009062114PMWEBVDT.htm Greyhawk's Meanderings Blog - "The great defrag debate. Defrag Domino or No" - http://www.greyhawk68.com/greyhawk68/home.nsf/d6plinks/JROG-7F8TZS The Weird World of Eknori - "Database Defragmentation - Poor Man’s Solution" - http://www.eknori.de/2008-11-29/database-defragmentation-poor-mans-solution/ Visual Studio 2008/2010 C++ Express Editions - http://www.microsoft.com/express/download/ IBM Technote 1208136 on indirect file usage for batch processing - http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg21208136
11 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Annex A – DominoDefrag Enterprise Edition Features and Setup
Features
Create a Server Defrag Profile per Domino Server to control “DominoDefrag.exe” execution and report on activity Record all Server and Defrag defragmentation activities. Only need to implement in one place in your Domino domain as a mail-in database to capture all Domino server defragmentation activities. Control all Domino server DominDefrag Notes INI settings Generate Excel charts on Domino server total and average information. Symphony support coming soon… Register the DominoDefrag task in te domadmin.nsf so you can see it when monitoring server tasks in the Domino adminstrator Generate IND files and much more…..
DominoDefrag Enterprise Edition Setup Steps
Identify
Identify the R6x-R8x Domino server where the mail-in database (MIDB) will reside in your Domino Domain Identify the infrastructure template signing ID you’re going to use to sign it and what the ID's home mail server is. Identify the Domino server(s) running the DominoDefrag server task that you want to send a summary email from to the MIDB for processing.
Access
The infrastructure template signing ID will require Sign or run unrestricted methods and operations specified in the mail-in database's home Domino Server document (see Figure 4)
Figure 4 - Server Document Restrictions
12 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Sign the DominoDefragEE.nsf database with the infrastructure template signing ID (see Figure 5).
Figure 5 - Signing DominoDefragEE.nsf database
Create Mail-in Database Document
Create a Mail-In database document called DominoDefrag MIDB (or other choice) and point it to the Domino server where the DominoDefragEE.nsf database is located (see Figure 6)
Figure 6 - Mail-in database document
If the mail-in database server is not the signing ID's home mail server, the Process Summary Messages agent will not run. Fix this by adding AMGR_DISABLEMAILLOOKUP=1 to the Domino server's Notes INI file and restarting the Domino server.
13 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Create a Server Defrag Profile
Go to the Server Defrag Profiles view, click Create Profile, select a Domino Server and click OK (see Figure 7).
Figure 7 - Create Server Defrag Profile
Accept the default values and click Save & Close (see Figure 8).
Figure 8 - Save Server Defrag Profile
Turn on DominoDefrag.exe Enterprise Support
In the DominoDefrag.exe Domino server's Notes INI file add DominoDefrag_EnterpriseSupport=1 and DominoDefrag_SummaryEmailAddress= DominoDefrag MIDB (or other choice) and restart the Domino server.
14 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12 Annex B – Examples of latest functionality
Figure 9 - Example 1 showing the new DominoDefrag_OptimizeDataVolume=1 feature. See the consolidate free space, alphabetical sort and fast sort phases
15 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Figure 10 - Example 2 showing the new DominoDefrag_OptimizeFiles=1 feature after defragging the mail\tuser.nsf database
16 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Figure 11 - Example 3 showing the new summary email average counters and DominoDefrag_EnterpriseSupport=1 feature adding the CSV files to the summary email, which the DominoDefrag Enterprise Edition mail-in database DominoDefragEE.nsf processes
Figure 12 - Example 4 shows an Excel chart that can be generated from within the DominoDefrag Enterprise Edition mail- in database DominoDefragEE.nsf. The information was used from Example 3’s CSV files.
17 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Figure 13 – Example 5 showing the new C API DominoDefrag server task using the DominoDefrag_FreeSpace=10 and DominoDefrag_FreeSpaceMaxDB=5 settings. DominoDefrag_VerboseLogging=1 will show the before and after free space information.
18 | P a g e
DominoDefrag v3.0 Release Notes Sep. 12
Figure 14 - Example 5 Summary Email sent to World Administrator
19 | P a g e