<p>Citrix XenApp 7.6 Logon Slow – Long Black Screen Phase by Helge Klein on April 27, 2015 in Citrix / Terminal Services / Remote Desktop Services</p><p>Update 2015-04-28: Citrix provides the limited release hotfix ICATS760WX64009 that fixes this issue. More information below.</p><p>During the research for my session about the XenApp 7.6 logon process, to be presented at Citrix </p><p>Synergy and BriForum London, I noticed that the logon to my XenApp 7.6 lab server was taking a bit long. Longer, in fact, than the combined durations of the main logon phases user profile loading, group policy processing, logon script execution and shell startup. Much longer. Also much longer than on an otherwise similar XenApp 6.5 machine.</p><p>Examining uberAgent’s brand-new logon process performance dashboard I found a 10 second gap:</p><p>After the user profile has been loaded and group policy has been processed, which are both finished after three seconds, nothing happens for about ten seconds. Finally, at 13 seconds into the login is the Userinit process started which ultimately loads the shell, Explorer.exe. During most of that time the end user is presented with a black screen, getting no feedback as to what might or might not be happening as you can see in this video: I experiemented a lot trying to find the root cause of the prolonged black screen phase, disabling all kinds of redirection, printer mapping, even IPv6. None of that fundamentally changed the duration of the black screen phase. Then I had the idea to log in via RDP (for which you have to make the user a member of the group Direct Access Users on the XenApp machine):</p><p>Now, that is a lot faster! RDP logons are finished in less than 4 seconds compared to around 14 seconds for ICA logons.</p><p>Digging around some more I stumbled upon CTX135782 which suggests setting the following registry value:</p><p>Key: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Logon Value: DisableStatus Type: REG_DWORD Data: 1</p><p>The effect was immediate, bringing the ICA logon from 14 seconds down to around 5. Happy End?</p><p>Not quite. With DisableStatus set to 1, every time I logged on the LogonUI.exe process crashed due to a fault in the module CtxWinlogonProv.dll. Events like the following were logged:</p><p>Faulting application name: LogonUI.exe, version: 6.3.9600.16384, time stamp: 0x5215f6c5 Faulting module name: CtxWinlogonProv.dll_unloaded, version: 7.6.0.5018, time stamp: 0x541cfd05 Exception code: 0xc0000005 Fault offset: 0x00000000000019e1 Faulting process id: 0x1e2c Faulting application start time: 0x01d080da8b0d6ae2 Faulting application path: C:\Windows\system32\LogonUI.exe Faulting module path: CtxWinlogonProv.dll Report Id: cb285960-eccd-11e4-80c0-00155d001b10 Faulting package full name: Faulting package-relative application ID:</p><p>CtxWinlogonProv.dll, also called CitrixRemoteLogonFilter, is registered as a credential provider filter via the registry keyHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider </p><p>Filters\{3571A91D-713C-427c-AA0C-BBF4F618A819}. When that key is renamed or deleted LogonUI.exe does not try to load CtxWinlogonProv.dll and consequently does not crash any more while logons are still fast.</p><p>However, since I do not know what the consequences of disabling the registration of CtxWinlogonProv.dll are I cannot really recommend that approach. Ultimately Citrix are the only ones who can truly fix this. Update 2015-04-28: a Hotfix</p><p>Shortly after the initial publication of this article Shane Kleinert pointed me to limited release hotfix ICATS760WX64009 </p><p>(CTX142036) which seems to be available for server OS (i.e. XenApp) only. As he explains in the comments below </p><p>Shane had noticed similar issues at customer sites and found CTX142036 in spite of the more than vague problem description.</p><p>I am happy to report that CTX142036 fixed this long black screen phase issue on my machines, too. So this story does have a happy end, after all! https://helgeklein.com/blog/2015/04/citrix-xenapp-7-6-logon-slow-long-black-screen-phase/ </p><p>For those wondering the hotfix is “ICATS760WX64009” This is specifically applied to the Server VDA, NOT the workstation VDA.</p><p>Installing uberAgent for Splunk is quick and straightforward. Due to the product’s flexibility there is often more than one way of doing things. In such a case we describe the recommended configuration here and link to supported alternatives. Prerequisites  Read about the architecture options.  If you do not have a running Splunk installation yet set up Splunk.  Download uberAgent and extract the contents of the archive. You should have a directoryuberAgent components with the following content: o A subdirectory uberAgent_endpoint o Packed Splunk app uberAgent_indexer.tgz o Packed Splunk app uberAgent_searchhead.tgz Preparing the Splunk Server Manual Installation  Go to the Splunk console’s home page by navigating to http://servername:8000 in your browser.  Click Manage apps:</p><p> Click Install app from file:</p><p> Select uberAgent_indexer.tgz and click Upload  The uberAgent indexer app is now installed  Install uberAgent_searchhead.tgz in the same way  Click Settings -> Server controls -> Restart Splunk Distributed Splunk Deployment If you have a distributed Splunk deployment with separate search heads and indexers please deploy the indexer app to all indexers and the search head app to all search heads.</p><p>Alternative Architectures Note: This is optional and not required for the recommended architecture. If you decided to have uberAgent send data to Splunk through a locally installed Universal Forwarder on the monitored endpoints you need to enable receiving Universal Forwarder data as described here, i.e. through Settings -> Forwarding and receiving -> Receive data -> Add new -> 9997 -> Save. Installing uberAgent on the Endpoints The agent installer is available as an MSI package. The MSI can either be installed manually or unattended through existing software deployment tools or Splunk’s Deployment Server.</p><p>Expected result after the installation of the MSI: the service uberAgent is installed and running. Manual Installation  Run the batch file uberAgent_endpoint\bin\manual-install.cmd  On the screen Receiver Configuration specify the name(s) of your Splunk indexer(s) and the port configured earlier (default: 19500) Installation Through a Software Deployment Tool  Install the appropriate MSI file from the directory uberAgent_endpoint\bin depending on the bitness of your machine: uberAgent-32.msi or uberAgent-64.msi  Specify the following MSI parameters: o INSTALLDIR [optional]: installation directory o SERVERS [required]: comma-separated list of receiving servers/ports. o Example:</p><p> o SERVERS="splunk1:19500,splunk2:19500"</p><p>Alternative Architectures Note: This is optional and not required for the recommended architecture. If you decided to implement one of the alternative architectures you need to install Universal Forwarder on each endpoint. In order to deploy uberAgent through Splunk’s Deployment Server follow these steps. Alternatively, if you want to run uberAgent in legacy mode (not recommended) follow this documentation. Configuration uberAgent can be configured very flexibly. By editing the configuration file you can switch metrics on or off, change the data collection frequency and significantly reduce the data volume. License File If you have a license file for uberAgent copy it to the installation directory (default: C:\Program Files\vast limits\uberAgent). Without a license file uberAgent displays a splash screen during logon. Contact us for an evaluation license. Imaging / Citrix PVS If you intend to copy the agent installation via an imaging method or Citrix PVS we recommend you remove instance-specific information. To do that follow these steps right before capturing the image:  Stop the service uberAgent (but leave the start type at automatic)  Open an administrative command prompt  Run the command: reg delete “HKLM\SOFTWARE\vast limits\uberAgent” /f /reg:64  Prepare the machine for cloning as necessary, but do not reboot If you have Splunk Universal Forwarder installed, please follow the steps listed here, too. Monitoring uberAgent The operation of the uberAgent service can be monitored by checking the log file. Centralized monitoring is possible through the Splunk app uberAgent Log Collector. All Done! You have successfully deployed uberAgent. Now take a look at the dashboards by pointing your browser athttp://servername:8000/app/uberAgent. Have fun!</p><p>If you do not see data from some or all endpoints please follow these troubleshooting steps. Questions?</p><p>Do you have questions that were not answered here? Please ask us, we are happy to help!</p><p> https://uberagent.com/documentation/install-and-deploy-uberagent/ </p><p>Hotfix ICATS760WX64009 - For VDA Core Services 7.6 for Windows Server OS (64-bit) - English</p><p>Software Updates | Public | 16 found this helpful | Created: 10 Apr 2015 | Modified: 03 Aug 2015 Languages N/A Download</p><p>ICATS760WX64009.msp</p><p>Was this helpful? Yes No</p><p>Filename: ICATS760WX64xxx.msp</p><p>For: XenApp 7.6; XenDesktop 7.6 VDA Core Services for Windows Server OS (64-bit). or operating system </p><p> requirements, seeSystem requirements on the Citrix Product Documentation site.</p><p>Replaces: None Date: November, 2014</p><p>Languages supported: English (US), German (DE), Spanish (ES), French (FR), Japanese (JA), Simplified </p><p>Chinese (SC)</p><p>Readme version: 1.01</p><p>Readme Revision History Version Date Change Description 1.01 May, 2015 Updating for general availability 1.00 March, 2015 Initial release</p><p>Important Note(s)</p><p> This release is based on XenApp 7.6/XenDesktop 7.6 VDA Core Services for Windows Server (64-bit).</p><p>It patches only systems where Version 7.6 of the product is installed. Any known issues in Version 7.6 of </p><p> the product, except for the specific issues resolved in this limited release, still apply.</p><p> You can upgrade the VDA from a console session or from an RDP session. Upgrading from within an </p><p>ICA session is not supported.</p><p> Caution! This release may require you to edit the registry. Using Registry Editor incorrectly can cause </p><p> serious problems that may require you to reinstall your operating system. Citrix cannot guarantee that </p><p> problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your </p><p> own risk.</p><p>Where to Find Documentation</p><p>This document describes the issue(s) resolved by this release and includes installation instructions. For additional product information, including supported operating systems and system requirements, see Citrix Product </p><p>Documentation.</p><p>New Fixes in This Release</p><p>1. When starting a published application, users can experience slow logons.</p><p>[From ICATS760WX64009][#LC0596]</p><p>Fixes from Replaced Hotfixes</p><p>No hotfixes were replaced by this release.</p><p>Installing and Uninstalling this Release</p><p>Notes:  This release is packaged with Microsoft Windows Installer 3.0 as a .msp file. For more information </p><p> about deploying .msp files, see Microsoft article 884016 or visit the Microsoft Web site and search on </p><p> keyword msiexec.</p><p> This installer program complies with Microsoft User Account Control (UAC). If UAC is enabled, you </p><p> must run the installer program in elevated mode; that is, with administrative privileges enabled. For more </p><p> information about UAC, see Microsoft TechNet or visit the Microsoft Web site and search on keyword </p><p>UAC.</p><p> To install this release successfully, computers must not have registry modification restrictions in place.</p><p> Slipstreaming - installations of the VDA for Windows Server OS base product packaged with one or </p><p> more individual hotfixes - is not supported and might leave servers in an unstable state.</p><p> You must restart the server after the installation of this hotfix completes. However, if you are installing </p><p> multiple hotfixes at the same time, there is no need to restart the server after the installation of each hotfix.</p><p>It is sufficient to restart the server after the installation of the final hotfix completes.</p><p> If the need arises to restore the original settings and functionality provided by this release, you must </p><p> uninstall the release before reinstalling it according to the installation instructions below.</p><p>To install this release:</p><p>1. Copy the release package to an empty folder on the hard drive of the computer you want to update.</p><p>2. Close all applications.</p><p>3. Run the executable.</p><p>4. Restart the computer.</p><p>To uninstall this release:</p><p>1. From the Start menu, select Control Panel > Programs and Features.</p><p>2. Highlight the release you want to uninstall and click Uninstall.</p><p>3. Follow the directions on-screen.</p><p>Files Updated (All Dates/Times UTC)</p><p>File Name Date Time Size</p><p>CPatch.exe 01/26/2015 13:42 338,288</p><p>CtxHfLoader.dll 01/26/2015 13:42 395,648</p><p> ctxregasm.exe 01/26/2015 13:42 26,576 msi50ca.dll 01/26/2015 13:42 633,184</p><p>SDKv06_RunTime_Xml 01/26/2015 13:42 334 statui.dll 01/26/2015 13:42 103,192</p><p>CPatchUI.dll (x32\de) 01/26/2015 13:42 14,200</p><p>CPatchUI.dll (x32\en) 01/26/2015 13:42 13,688</p><p>Error.idt (x32\en) 01/26/2015 13:42 35,673</p><p>CPatchUI.dll (x32\es) 01/26/2015 13:42 14,712</p><p>CPatchUI.dll (x32\fr) 01/26/2015 13:42 14,200</p><p>CPatchUI.dll (x32\ja) 01/26/2015 13:42 12,152</p><p>CPatchUI.dll (x32\zh-cn) 01/26/2015 13:42 11,640</p><p>File Name MD5 Checksum</p><p>ICATS760WX64009.msp 992F0BEC3BFA0FCC24DA274F479AE857</p><p>CPatch.exe B552C4ADFC4EBDBC198EB2620F282709</p><p>CtxHfLoader.dll 3342F05F9EE6F862172AC3150F804149 ctxregasm.exe 07207F40A7704D248E14594672B72CC4 msi50ca.dll C0E3ED1736A69FE8B6D6EBC6E5F2421E</p><p>SDKv06_RunTime_Xml 02C86CD1A05CE7F5BF41291C7AA6B595 statui.dll 8976157104431E08A2530F1026D0192C</p><p>CPatchUI.dll (x32\de) E4F3BD678E21553FD6B3BB48F2BDB297</p><p>CPatchUI.dll (x32\en) 1D5BC99B2A180E49381C086804B70F69 Error.idt (x32\en) 8370B4898257DD82C2D81D3EF1BE893E</p><p>CPatchUI.dll (x32\es) EAFEEF8AAA7BEBE89139D1AA0150A5B5</p><p>CPatchUI.dll (x32\fr) E2AC887A998BBC67CC3BEC4A7705EECB</p><p>CPatchUI.dll (x32\ja) A86BB78D0F01E8A8D82868E7B3FFF9E5</p><p>CPatchUI.dll (x32\zh-cn) 4D5B49F6F74FAF217C543D21DBCD3F91</p>