Christoph Adler - Senior Consultant at panagenda
19+ years of Lotus / IBM / HCL Digital Solutions experience
Focused on • Notes Client Management (incl. HCL Nomad) • Analysis and Optimization of Notes/Domino infrastructures • Application Modernization • Cloud Migrations
Good to know • Obviously German (accent) • Bass player & geek • Likes music, football (soccer) and beer https://www.slideshare.net/christophadler50 Agenda
1. Introduction
2. Roaming
3. HCL Notes 11.0.1 FP2 Installation
4. Live Demo - Workspace Improvement
5. Q & A Introduction
Make Your Data Work For You What are virtual environments (in this case)
Examples • VDI – Virtual Desktop Infrastructure – VMWare Horizon View OR Citrix XenDesktop • TS – Terminal Services/Server – Citrix XenApp What we typically see at customers
Aged Notes installations in virtual environments that are working “somehow” – mostly Notes data directories on network drives which cause • long startup times of HCL Notes (up to 60 sec) • hangs and wait times due to interruptions and poor network latency • high network traffic as HCL Notes is constantly reading / writing data • frequently corrupted local applications • permanently occupied disk space on file servers / SAN / NAS – at least 120MB per user – often because of old and outdated files, even templates What we typically do at customers
• Sparking their enthusiasm for HCL Notes – especially in virtual environments, HCL Notes can be optimized to start up fast, run smooth and contribute to a positive end user IT experience by • switching to local Multi-user installations • frequently maintaining contents of HCL Notes data folders • relieving network, file servers and/or SAN/NAS systems
• Encouraging administrators to spend a little more time on troubleshooting and analyzing HCL Notes – Solve root causes, not symptoms! Roaming
Make Your Data Work For You To roam or not to roam: that is the question
• Given the previous reasons why HCL Notes in virtual environments should not have any files in the network, the following logical questions arise – How can personal HCL Notes data folders of end users be maintained locally on the virtual environment servers? • Think of having N servers in the farm where personal end user data potentially needs to be on every single server
• How do end users get their most recent HCL Notes configuration no matter on which server in the farm they log on?
• How to ensure that disk space used by HCL Notes data folders is cleared after log off? To roam or not to roam: that is the question (cont.)
• How to avoid transfers of large HCL Notes data folders per user during log on and log off which increase wait times and cause network traffic?
• The answer is: Roaming! – To provide end users with their latest HCL Notes configuration, their data needs to be roamed onto the server they log on to start their session – Roaming can be achieved through own procedures (e. g. logon script), Domino Roaming or third-party solutions To roam or not to roam: that is the question (cont.)
Log on to virtual Start Log on to Work with session HCL Notes HCL Notes HCL Notes
Roam data Roam data Roam data
(happens here: (or here: MarvelClient (or here: logon script, Roaming) Domino Roaming) 3rd-party solutions) Log off from Shutdown Clear data virtual session HCL Notes
Backup Backup data data
(or here) (happens here: Domino Roaming or MarvelClient Roaming) HCL Notes 11.0.1 FP2 Installation
Make Your Data Work For You Basic stuff
• Use HCL’s default paths – Program directory C:\Program Files(x86)\HCL\Notes – Shared data directory C:\ProgramData\HCL\Notes\Data\shared – Data directory C:\Users\
• MultiUser install only à please!
• Use Install parameter “CITRIX=1” (at least if TS involved)
• No need to make changes to MSI package (scripts can help here) Command line(s)
• Uninstall & cleanup first – HCL NICE (Notes Install Cleanup Executable): https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0040768 "C:\Install\NotesV11\nice.exe" -rp –wipe –qn – rd /S /Q "C:\Program Files (x86)\IBM\Notes" – rd /S /Q "C:\Program Files (x86)\IBM\Lotus" – rd /S /Q "C:\ProgramData\Lotus" – rd /S /Q "C:\ProgramData\IBM\Notes" Command line(s)
• Install – Notes Client 11.0.1 “C:\Install\Notes_1101_Win_English.exe" -s -a /s /v"SETMULTIUSER=1 CITRIX=1 /qn /lime “C:\Install\Notes_1101_Win_English.log“
– Multilingual User Interface Group 2 “C:\Install\Notes_1101_MUI_G2A.exe" -s -a /s /v"ADDLOCAL=Russian CITRIX=1 /qn /lime “C:\Install\Notes_1101_MUI_G2A.log“
– Fix Pack 2 “C:\Install\HCL_Notes_11.0.1FP2_Win.exe" -s -a /s /v“METHOD=INSTALL CITRIX=1 /qn /lime “C:\Install\HCL_Notes_11.0.1FP2_Win.log“ ODS53
ODS = On Disk Structure • ODS 16 = Notes 2 • ODS 17 = Notes 3 • ODS 20 = Notes 4 • ODS 41 = Notes 5 • ODS 43 = Notes 6 & 7 • ODS 48 = Notes 8 • ODS 51 = Notes 8.5 & 9 • ODS 52 = Notes 9.0.1 • ODS 53 = Notes 10 & 11
The difference between ODS 43 and 53 = up to 80% LESS FILE I/O ODS53 (cont.)
To upgrade the ODS of all shared templates in a virtual HCL Notes Multi-User installation you can follow these steps (includes ODS 53 for new databases created by the user) • Add the following line to the shared notes.ini in %ProgramData%\HCL\Notes\Data – CREATE_R10_DATABASES=1
• Make sure that the Administrator account used to log on to the virtual server does not have a personal HCL Notes Data directory yet (in %LocalAppData%)
• From a command prompt type the following commands – cd “%ProgramData%\HCL\Notes\Data\Shared” – “%ProgramFiles(x86)%\HCL\Notes\ncompact.exe” -C -ODS -* Notes.ini
• A properly configured shared notes.ini can be essential for smooth running virtual HCL Notes environments
• The shared notes.ini is used by every HCL Notes client session for which there’s no personal HCL Notes data folder yet
• It’s located in %ProgramData%\HCL\Notes\Data
• It is used as a template/stub notes.ini out of which all parameters are transferred into a users personal notes.ini upon the very first HCL Notes start
• Find some useful shared notes.ini parameters on the next slide (additionally to the standard entries) Notes.ini (cont.)
• Ports=TCPIP
• TCPIP=TCP,0,15,0,,12288, – Note that this sets port compression to OFF, which is appropriate in virtual environments; Port compression adds 5% CPU overhead and with 60 concurrent sessions this would result in needing 3 dedicated CPUs for just port compression
• DISABLE_CLIENTRECORD=1 – This disables logging of client information back into the public address book as all virtual environment users share the same servers in the farm anyway Notes.ini (cont.)
• CREATE_R10_DATABASES=1 – Enables HCL Notes V11 Beta-1 to use the latest ODS version for newly created local applications. Make sure to not use CREATE_R85_DATABASES (IBM Lotus Notes 8.5) or CREATE_R9_DATABASES (IBM Notes 9.0.1) to avoid the use of an older ODS in HCL Notes 11.0.1
• Log=,1,0,7,40000 – This disables logging, which is optional but gains a slight performance improvement – This could cause issues when applications needs an existing log.nsf à please check before!
• ConfigFile=C:\ProgramData\HCL\Notes\Data\config.txt – For ConfigFile details see next topic Notes.ini (cont.)
All parameters for copy & paste
Ports=TCPIP TCPIP=TCP,0,15,0 DISABLE_CLIENTRECORD=1 CREATE_R10_DATABASES=1 Log=,1,0,7,40000 ConfigFile=C:\ProgramData\HCL\Notes\Data\config.txt
[always keep a blank line at the bottom!] Config File
• HCL Notes can be set up unattended by using a so called ConfigFile – often referred to as config.txt or setup.txt
• The ConfigFile needs to be referenced in the shared notes.ini using the parameter ConfigFile=Drive: or UNC\folder\filename.txt (see previous topic)
• For more details, the following HCL Knowledge Article provides some very good examples on how to create a ConfigFile as generic as possible - https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0036482 Config File (cont.)
Example ConfigFile for copy & paste
Username=%USERNAME% Domino.Name=Hubserver/Acme Domino.Address=servername.acme.com Domino.Port=TCPIP Domino.Server=1 AdditionalServices=-1 ShareClasses Improvement
• In HCL Notes the jvm.shareclasses file (*_xpdplat_.jvm_G41L00) is stored in each users Data\workspace\.config\org.eclipse.osgi folder and has a fixed size of 64MB
• To optimize file I/O, speed up the HCL Notes Client start and save storage (960MB per server == 15users), this file can be shared across all users of the virtual environment server 1. Create the following folder “%ProgramData%\HCL\Notes\Data\ShareClasses“ and set folder rights for “everyone” to read / write access 2. Change the value of this parameter “jvm.shareclasses.loc=“ in %ProgramFiles(x86)%\HCL\Notes\framework\rcp\deploy\jvm.properties • from “${rcp.data}/.config/org.eclipse.osgi“ • to “C:/ProgramData/HCL/Notes/Data/ShareClasses“ à it’s important to use “/” instead of “\” HCL Notes 11.0.1 FP2 - jvm.properties file
To speed up the HCL Notes JVM it’s useful to optimize memory settings
Open %ProgramFiles(x86)%\HCL\Notes\framework\rcp\deploy\jvm.properties – Modify vmarg.Xmx from “-Xmx256m” to “-Xmx1024m” – Modify vmarg.Xms from “-Xms64m” to “-Xms256m” – Change vmarg.Xmca=-Xmca512k
The implicit rules – Do not change vmarg.Xmx to more than -Xmx1024m regardless of how much memory you have – vmarg.Xms should always be ¼ of what you changed Xmx to
Note – The above defaults have been changed by IBM/HCL over the last couple of Notes releases. You might find other values than Xmx256m and Xms64m in your setups. The Out Of Memory Mystery In Notes 11
• Some of our customers who upgraded to Notes 11 have experienced out of memory errors in the Notes client. That always ended up in freezes or crashes of HCL Notes 11.x
• The weird thing about this: - The computer had plenty of available memory, and - Task Manager showed that Notes was using a normal amount of memory, like 250MB or so
• Using Performance Monitor for more detailed troubleshooting - “Private Bytes” - the amount of committed memory the application is using - “Virtual Bytes” - the amount of virtual address space the application has reserved
More information: https://www.panagenda.com/2020/08/the-out-of-memory-mystery-in-notes-11/ The Out Of Memory Mystery In Notes 11 (cont.)
More information: https://www.panagenda.com/2020/08/the-out-of-memory-mystery-in-notes-11/ The Out Of Memory Mystery In Notes 11 (cont.)
• When we were testing, we found that out of memory errors were happening at the point that the client hit the 2GB mark in the virtual address space monitor.
• This 2GB mark is important because the Notes client is still a 32-bit application. One of the side-effects of this is it only has 2GB of virtual address space to play with (really 4GB, but half of that is claimed by the kernel). 64-bit applications have a much, much larger virtual address space (they sometimes reserve terabytes of space), but 32-bit applications need to be careful.
• SIDEBAR: A few clarifications about virtual address space: - it’s completely different from virtual memory - it doesn’t tell you how much memory the application is actually using - it has nothing to do with how much memory is on the computer - it’s a way of mapping memory that might be used by the application - More information here about virtual address space https://docs.microsoft.com/en-us/windows-hardware/drivers/gettingstarted/virtual-address-spaces
More information: https://www.panagenda.com/2020/08/the-out-of-memory-mystery-in-notes-11/ The Out Of Memory Mystery In Notes 11 (cont.)
• Solution - The immediate fix was to reduce the Java heap size used by the Notes client vmarg.Xmx=-Xmx512m - See HCL‘s Technote about this: https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0038188
• Why wasn’t this a problem before? - When you compile a 32-bit application, there is a flag you can set called /LARGEADDRESSAWARE. This allows the application’s process to use a full 4GB of virtual address space on a 64-bit operating system, instead of the 2GB we were seeing above. - It turns out that the java.exe and notes2.exe files on the Notes 9 client were compiled with the /LARGEADDRESSAWARE flags, but the java.exe and notes2.exe files on the Notes 11 client were not. - Notes 9 used the IBM version of the Java JVM but Notes 11 uses the OpenJDK OpenJ9 version. The OpenJ9 32- bit distribution wasn’t compiled with /LARGEADDRESSAWARE until just recently (version 8u262, from June 2020), so the Notes 11 client didn’t compile with that flag either. - In other words, Notes 9 had a full 4GB virtual address space, but Notes 11.x only has 2GB.
More information: https://www.panagenda.com/2020/08/the-out-of-memory-mystery-in-notes-11/ HCL Notes 11.0.1 FP2 - jvm.properties file
To speed up the HCL Notes JVM it’s useful to optimize memory settings
Open %ProgramFiles(x86)%\HCL\Notes\framework\rcp\deploy\jvm.properties – Change vmarg.Xmx=-Xmx512m à or maybe to -Xmx768m – Change vmarg.Xms=-Xms256m – Change vmarg.Xmca=-Xmca512k – Remove the value “singleJVM,” from the parameter “vmarg.Dshare=” Workspace Improvement
• Every HCL Notes Client start in a virtual environment is a first (setup) start and takes ~20 secs on average
• ~65% - 75% of time during first starts is used to build the workspace folder (at least 13 sec)
• A typical new workspace folder (HCL Notes 11.0.1 FP2) consists of – >220 files – >160 folders – >85 MB (including 64MB *_xpdplat_.jvm_G41L00 file)
• Building the workspace folder causes lots of file I/O Workspace Improvement (cont.)
Accelerate the HCL Notes Client start to be >50% faster (with less file I/O)
1. With a non-admin user (Windows + HCL Notes) run an initial setup of HCL Notes 2. Start HCL Notes and wait for 1,5 minutes 3. Exit HCL Notes 4. Start HCL Notes again and wait for another 1,5 minutes 5. Exit HCL Notes 6. Create the following folder à %ProgramData%\HCL\Notes\Data\Common 7. Copy the workspace folder from %LocalAppData%\HCL\Notes\Data to %ProgramData%\HCL\Notes\Data\Common Workspace Improvement (cont.)
8. Delete the following files / folders in %ProgramData%\HCL\Notes\Data\Common\workspace a) UDM b) Logs à all but .prov2install c) .metadata\.plugins\com.ibm.collaboration.* d) .metadata\.plugins\com.ibm.rcp.personality.framework e) .metadata\.plugins\com.ibm.rcp.security.auth f) .metadata\.plugins\com.ibm.rcp.managedsettings g) .metadata\.plugins\org.eclipse.e4.workbench Workspace Improvement (cont.)
Important: • You must recreate the workspace folder when your Installation changes (another Notes version, Fix Pack, etc.) • If you want to include the workspace folder to your installation package, you also must include the following file: – C:\Program Files (x86)\HCL\Notes\framework\rcp\rcplauncher.properties Workspace Improvement
Live
Make Your Data Work For You Windows Registry
If you change the Data Directory – let’s say from “H:\Notes\Data” to “%LocalAppData%\HCL\Notes\Data”
• Please ensure that the following Registry-Entry will be deleted during Windows-Logon (always) HKEY_CURRENT_USER\Software\Lotus\Notes\NotesIniPath DEP
• Data Execution Prevention • Exclude – C:\Program Files (x86)\HCL\Notes\notes.exe – C:\Program Files (x86)\HCL\Notes\nlnotes.exe – C:\Program Files (x86)\HCL\Notes\nsd.exe – C:\Program Files (x86)\HCL\Notes\framework\rcp\eclipse\plugins\ com.ibm.rcp.base_10.0.0.20200318-1334\win32\x86\notes2.exe