While every precaution has been taken in the preparation of this installation manual, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
To get an idea of the Development Production Line take a look at the following Application Integration overview and Maven vs SonarQube Quality Assurance reports comparison.
17.3.5. Upgrade FOP ...... 246 17.3.6. Fonts ...... 252 17.3.7. PressGang Example ...... 258 17.3.8. Publication ...... 259 A. Tips & Tricks ...... 261 A.1. Archives ...... 261 A.1.1. How to extract content from MSI files ...... 261 A.1.2. How to determine absolute path of a loaded Class ...... 261 A.2. Version Control System ...... 261 A.2.1. GitLab Repository Mirroring ...... 261 A.2.2. SVN to Git Migration ...... 264 A.2.3. GitWeb enhancement ...... 271 B. Hosting a group Website in the Cloud ...... 275 B.1. Resources ...... 275 B.2. Bitbucket ...... 275 B.3. GitHub ...... 275 B.4. GitLab ...... 275 B.5. Multiple remote git locations ...... 276 C. Continuous Integration Pipeline ...... 277 C.1. Resources ...... 277 C.2. GitLab CI Pipeline ...... 277 C.2.1. Build Stage (test-)compile ...... 277 C.2.2. Test Stage deploy ...... 278 C.2.3. Test Stage no recompile - Changes detected ...... 278 C.2.4. Test Stage no recompile - Compiling xx source files ...... 279 C.2.5. Test Stage unit vs integration phase ...... 279 C.2.6. Test Stage unit vs integration job ...... 279 C.2.7. GitLab CI Cache ...... 280 C.2.8. Show Time ...... 281 C.2.9. Show Date/Time ...... 281 C.2.10. Suppress download messages ...... 282 C.2.11. Deploy Stage site:stage ...... 282 C.2.12. GitLab CI Deploy Pages Stage ...... 282 C.2.13. Maven Site Staging issue workaround ...... 283 C.2.14. GitLab CI Deploy Pages Stage take 2 ...... 283 C.2.15. Site Generation revisited ...... 284 C.2.16. Effective Pom ...... 285 C.2.17. Optional JDK Check ...... 285 C.2.18. Optional OWASP Test Stage ...... 285 C.2.19. Optional SonarQube Docs stage ...... 285 C.2.20. SonarQube Dashboard issue workaround ...... 286 C.2.21. Optional OWASP Docs Stage ...... 286 C.2.22. Artifacts verify vs deploy ...... 287 C.2.23. Artifacts verify on fork ...... 288 C.2.24. Javadoc Reports ...... 288 C.2.25. Dependency Check Report ...... 288 C.2.26. Site Generation on release only ...... 289 C.2.27. Docker Image ...... 289 C.3. Jenkins CI Pipeline ...... 290 C.3.1. Build Stage (test-)compile ...... 290 C.3.2. Test Stage deploy ...... 291 C.3.3. Test Stage unit vs integration phase ...... 292 C.3.4. Test Stage unit vs integration job ...... 293 C.3.5. Test Stage no recompile - Changes detected ...... 294 C.3.6. Show Time ...... 294 C.3.7. Show Date/Time ...... 295 C.3.8. Suppress download messages ...... 295 C.3.9. Deploy Stage site-deploy ...... 295 C.3.10. Jenkins CI Test Report ...... 296 C.3.11. Effective Pom ...... 296 C.3.12. Optional JDK Check ...... 297 C.3.13. Optional OWASP Test Stage ...... 297 C.3.14. Optional SonarQube Deploy stage ...... 298 C.3.15. Optional OWASP Docs Stage ...... 299 C.3.16. Jenkins CI (un)stash vs checkout scm ...... 299 C.3.17. Test Stage no recompile - Compiling xx source files ...... 301 C.3.18. Extra Open Source Stage ...... 301 C.3.19. Artifacts verify vs deploy ...... 301 C.3.20. Javadoc Reports ...... 302 C.3.21. Release Tags ...... 302 D. Maven Versions Rules ...... 307 E. Custom skins ...... 311 E.1. Custom Paper Skin ...... 312 E.1.1. Customization ...... 313
vi E.2. Custom Fluido Skin ...... 316 E.3. Custom Reflow Skin ...... 318 F. Quality Assurance Sandbox ...... 327 F.1. Javadoc report ...... 328 F.2. Cross-reference ...... 329 F.3. Maven Quality Assurance reports ...... 329 F.3.1. Checkstyle ...... 330 F.3.2. PMD/CPD ...... 331 F.3.3. FindBugs ...... 332 F.3.4. JavaNCSS ...... 332 F.3.5. JDepend ...... 333 F.3.6. Code Coverage ...... 334 G. Source [ToDo svn2git] ...... 337 H. Source ...... 339 H.1. Available Documentation ...... 339 H.2. Available Projects ...... 339 H.3. Development Production Line - The Short Story Reference ...... 339 H.3.1. Additional Maven Documentation ...... 339 H.3.2. Releases ...... 340 H.3.3. Quality Assurance ...... 340 H.3.4. Open Source Patches ...... 341
vii viii Chapter 1. Chapter 1. Operating System 1.1. Windows
Microsoft Windows 7 is an operating system for personal and business computers, including both desktops and laptops. 1.1.1. Resources
• Leaving Microsoft to Change the World. • Microsoft Security Advisories are a supplement to the Microsoft Security Bulletins. They address security changes that may not require a security bulletin but that may still affect customers' overall security. • Sysinternals utilities to help manage, troubleshoot and diagnose your Windows systems and applications. • Everything You Need To Know About the Blue Screen of Death. How to configure Windows to create MiniDump files on BSOD. BlueScreenView scans all your minidump files created during 'blue screen of death' crashes, and displays the information about all crashes in one table. • MemTest86 is the original, free, stand alone memory testing software for x86 computers. MemTest86 boots from a USB flash drive or CD and tests the RAM in your computer for faults using a series of comprehensive algorithms and test patterns. • Drivers can be selected for verification by using the Verifier Command Line, or by using Driver Verifier Manager. • LessMSI is a tool to view and extract the contents of a Windows Installer (.msi) file.. • SharpKeys is a Registry hack that is used to make certain keys on a keyboard act like other keys. For example you could use this utility to map Caps Lock to a Left Shift. • The Windows 7 forum covers news and updates and has an extensive Windows 7 tutorial section that covers a wide range of tips and tricks. • A List of Run Commands for Windows 7. • Exploring Windows 7's New Search Features. • Netstat displays protocol statistics and current TCP/IP network connections. • PortQry is a command-line utility that you can use to help troubleshoot TCP/IP connectivity issues. The Port- QueryUI tool provides a graphical user interface for the PortQry Command Line Port Scanner. New features and functionality in PortQry version 2.0. • NMAP (“Network Mapper”) is a free and open source utility for network exploration or security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. • The Microsoft Visual C++ Redistributable Packages install runtime components that are required to run C++ applications built with Visual Studio. • Quick fix for disappearing system tray icons. • Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7 re- sulting in “JVM_Bind java.net.SocketException: No buffer space available (maximum connections reached?)” • Windows 7 FOUND.000 File folder. • NVIDIA Driver Downloads. Disable Nvidia Telemetry tracking on Windows. • ASUS Download Center download the latest drivers, software, firmware and user manuals. Such as Rampage IV Extreme. • security.nl. 1.1.2. Desktop
To configure the Desktop minimize all open windows with +M and right click on the desktop Personalize it. This will open Control Panel → Appearance and Personalization → Personalization where you can set Desktop Background, Window Color, Sounds and Screen Saver. After Save Changes also Save theme.
To add your own folders to the Picture location of the Windows Desktop Backgrounds create the following registry file P:\dev\apps\windows\registry\wallpapers.reg:
Import these settings with regedit "P:\dev\apps\windows\registry\wallpapers.reg". 1.1.3. Explorer
Windows 7 introduces a brand new Windows Explorer which is superior in many ways to its predecessors. In Win- dows Explorer select Organize → Layout to activate the Menu bar, Display pane and Navigation pane. In the Menu bar select View → Details
1 Chapter 1.
To configure the Windows Explorer first select the Local Disk (C:) and then Organize → Folder and search op- tions or Tools → Folder options… where you can set:
• General: • Open each folder in the same window. • Double-click to open an item (single-click to select). • Show all folders. • Automatically expand to current folder. • View: • Always show menus. • Display file icon on thumbnails. • Display file size information in folder tips. • Show hidden files, folders, or drives. • Uncheck Hide extensions for known file types. • Uncheck Hide protected operating system files (Recommended). • Show drive letters. • Show encrypted or compressed NTFS folders in color. • Optionally uncheck Use Sharing Wizard (Recommended). • Select the typed item in the view
and Apply to Folders. • Search: • In indexed locations, search file names and contents. • Include subfolders in search results when searching in file folders. • Find partial matches. • Include system directories.
Tip
Advanced search is available by holding down +F keys.
For every non system drive select it and then Organize → Properties → Customize (see also Customize Tab - Add or Remove from Properties) or View → Customize this folder… to Optimize this folder for General items, check Also apply this template to all subfolders and click OK. 1.1.4. Windows 7 Start Menu
First go to Start Menu → Control Panel → Ease of Access → Ease of Access Center → Make the keyboard easier to use and select Underline keyboard shortcuts and access keys.
To customize the Start Menu right click on the Taskbar and select Properties.
• Taskbar • Check Lock the taskbar which is also available with right click on the Taskbar and select Properties. • Check Use small icons. • Taskbar buttons: Combine when taskbar is full. • Customize… which icons and notifications appear in the notification area. • Check Use Aero Peek to preview the desktop.
• Start Menu • In Privacy disable Store and display recently opened programs in the Start menu and Store and display recently opened items in the Start menu and the taskbar. • Customize… • Computer: Display as a link. • Control Panel: Display as a link. • Uncheck Default Programs. • Check Devices and Printers. • Documents: Don't display this item. • Downloads: Don't display this item. • Check Enable context menus and dragging and dropping. • Uncheck Favorites menu. • Games: Don't display this item. • Uncheck Help. • Uncheck Highlight newly installed programs. • Uncheck Homegroup. • Music: Don't display this item. • Check Network. • Personal folder: Don't display this item. • Pictures: Don't display this item. • Uncheck Recent Items. • Recorded TV: Don't display this item.
2 Operating System
• Check Run command. • Search other files and libraries: Search without public folders. • Check Search programs and Control Panel. • Check Sort All Programs menu by name. • System administrative tools: Display on the All Programs menu and the Start menu. • Uncheck Use large icons. • Videos: Don't display this item.
• Quick Launch Toolbar is restored with right click on the Taskbar and select Toolbars → New toolbar… and point to Folder C:\Users\\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch. Right click on the Quick Launch Toolbar to uncheck Show Text and Show title. Make sure the taskbar is unlocked (Lock the taskbar is unchecked), otherwise these menu options won't be available. 1.1.5. Task Manager replacement
Process Explorer shows you information about which handles and DLLs processes have opened or loaded.
Download the archive: ProcessExplorer.zip [version 16.21].
Extract the .zip file to P:\dev\apps\windows\process-explorer. Start the Process Explorer with procexp.exe and select Options → Replace Task Manager and Options → Hide When Minimized.
Create shortcut named Process Explorer for it and drag this shortcut to Start → Programs → Startup to be able to access Properties → Run → Minimized to always start it in the system tray on startup.
Important
In case of the following errors:
• “[SC] DeleteService FAILED 1072: The specified service has been marked for deletion.” • “CreateService failed - The specified service has been marked for deletion. (0x430).”
just Close Process Explorer and/or any services.msc console.
If a service still hangs with the following error “The service is starting or stopping. Please try again later.” see Kill stuck Windows service using sc queryex and taskkill.
1.1.6. Resource Monitor
Many folks use Windows SysInternals Process Explorer to gather information about running processes and their open handles. There is an excellent but less familiar utility in-built with Windows 7 and higher, called Resource Monitor which provides complete details of running processes, their associated handles and associated modules (for example, DLL files). Start the Resource Monitor with +R and resmon (see also How to Use Resource Monitor to Find Which Process Has Locked a File?).
Figure 1.1. Resource Monitor
1.1.7. Symbolic links
Junction (also called a soft link) differs from a hard link in that the storage objects it references are separate di- rectories, and a junction can link directories located on different local volumes on the same computer. Otherwise, junctions operate identically to hard links. Junctions are implemented through reparse points.
3 Chapter 1.
Download the archive: Junction.zip [version 1.07].
Extract the .zip file to C:\Windows\System32.
Verify the installation with junction. 1.1.8. User Account Control replacement
As a robust Security Monitor, WinPatrol will alert you to hijackings, malware attacks and critical changes made to your computer without your permission.
Download the binary: wpsetup.exe [version 35.5.2017.8].
Run this .exe file to install WinPatrol in P:\dev\apps\windows\win-patrol. 1.1.9. Windows Update
Go to Start Menu → Control Panel → System and Security → Windows Update → Change settings:
• Important updates: Check for updates but let me choose whether to download and install them. • Recommended updates: Give me recommended updates the same way I receive important updates.
Important
Before upgrading to SP1 uninstall all language packs except the default with lpksetup to avoid error “C000009A”.
The System Update Readiness Tool for Windows 7 for x64-based Systems (KB947821) is being offered because an inconsistency was found in the Windows servicing store which may prevent the successful installation of future updates, service packs, and software. 1.1.10. Windows Languages
Go to Start Menu → Control Panel → Region and Language:
• Formats: English (United States): • Short date: yyyy-MM-dd. • Long date: dddd, dd MMMM, yyyy. • Short time: HH:mm. • Long time: HH:mm:ss. • First day of week: Monday. • Additional settings… Currency symbol: €. • Current location: United States. • Keyboards and Languages: • Change keyboards… Default input language: English (United States) - US. • Install/uninstall languages… Choose a display language: English. • Administrative: • Copy settings… to Welcome sceen and system accounts and New user accounts. • Change system locale… to English (United States). 1.1.11. Windows Disks
Go to Start Menu → Computer → Manage → Storage → Disk Management and right click on new disk to:
• Initialize Disk (GPT). • New Simple Volume and click Next (twice). • Assign the following drive letter: K and click Next. • Format this volume with the following settings: Volume Label Knowledge and click Next and Finish. 1.1.12. Windows Mouse and Keyboard
Microsoft Garage Mouse Without Borders is an immensely useful tool. It might come as something of a surprise that it is a tool associated with Microsoft, and even more of a surprise that it has been kept a secret. Mouse Without Borders is a product that makes you the captain of your computer fleet by allowing you to control up to four computers from a single mouse and keyboard. This means that with Mouse without Borders you can copy text or drag and drop files across computers.
ShareMouse lets you share your mouse and keyboard with multiple networked computers. 1.1.13. Antivirus
All-inclusive and comprehensive protection Avast antivirus Home Edition includes ANTI-SPYWARE protection, certified by the West Coast Labs Checkmark process, and ANTI-ROOTKIT detection based on the best-in class GMER technology.
4 Operating System
Download the binary (Forum for latest offline installer): avast_free_antivirus_setup_offline.exe [ver- sion 18.5.2342].
Run this .exe file for Customize installation of Avast in P:\dev\apps\antivirus\avast. Optionally uncheck the following components: Behavior Shield, Ransomware Shield, Anti-spam, Real Site, Firewall, Sandbox, Software Updater, Browser Cleanup, Rescue Disk, Wi-Fi Inspector, Security browser extension, SafePrice browser extension, SecureLine VPN, Passwords, Sensitive Data Shield, Cleanup and Game mode.
Disable WebRep and Community settings with Open Avast user interface → Settings → General:
• disable Participate in the Avast community. • disable Enable reputation services. • disable Enable DeepScreen since 12.1.2272 CyberCapture against zero-second attacks. • disable Enable Hardened mode. • disable Enable Avast email signature. • Sounds: disable Use voiceover sounds (when available). • Privacy: disable Participate in the Avast community and/or disable Participate in data sharing to Opt-out.
Sometimes it´s not possible to uninstall Avast the standard way in the control panel. In this case, you can use our uninstallation utility avastclear.exe.
Note
See also Apache James “James.Mailet: RemoteDelivery” issue.
1.1.13.1. Resources
• How-To Geek article Avast Antivirus Was Spying On You with Adware (Until This Week) and Avast's response to the article on their forum. • Avast 2015: Managing participation in statistics to opt-out after installation from the data gathering. 1.1.14. Command Processor
To use the Tab for filename and directory name completion in the Command Processor create the following registry file P:\dev\apps\windows\registry\command-processor.reg:
Import these settings with regedit "P:\dev\apps\windows\registry\command-processor.reg".
After starting the Command Processor from the Quick Launch Toolbar configure the following settings by click- ing Alt+Space and selecting Properties and / or Defaults (for Start → Run… → cmd):
• Options: in Command History enable Discard Old Duplicates and in Edit Options enable Quick Edit Mode and Insert Mode. • Font: Font Lucida Console with Size 16. • Layout: Uncheck Let system position window and set Window Position Left 0 and Top 0. Based on font size and display resolution set for example: • 3840 x 2160: Screen Buffer Size Width 380 x Height 9999 and Window Size Width 380 x Height 128. • 1920 x 1080: Screen Buffer Size Width 188 x Height 9999 and Window Size Width 180 x Height 61.
or in case of Windows 10 Shortcut just Run: Maximized and in Layout only change Screen Buffer Size Height 9999 and check Wrap text output on resize.
To quickly repeat any previously entered command use the ▲ and ▼ arrow keys and use the F7 key to view a history of all the commands that have been entered in that Command Processor. 1.1.14.1. Elevated Command Prompt
An elevated command prompt is a command prompt that you run with administrator privileges.
Create for example a another shortcut for the Command Proccessor in the Quick Launch Toolbar. Right click on this shortcut and select Properties:
• Select Advanced to Run as administrator.
5 Chapter 1.
• And for example Change Icon… and Browse… %windir% \system32\UserAccountControlSettings.exe to the User Account Control (UAC) icon.
1.1.14.2. User Account Control Consent Prompt
The consent prompt notification is presented when a user attempts to perform a task that requires a user's ad- ministrative access token. To disable those annoying prompts for administrators go to Start Menu → Control Panel → System and Security → Administrative Tools → Local Security Policy or run secpol.msc. Select Security Settings → Local Policies → Security Options → User Account Control: Behavior of the elevation prompt for administrators in Admin Approval Mode and switch from Prompt for consent for non-Windows binaries to Elevate without prompting. Now it is possible to run the elevated command prompt without having to consent every time.
1.1.14.3. Edit the Hosts File
When editing the hosts file results in the following error “You don't have permission to save in this location” open an elevated command prompt and run notepad %windir%\system32\drivers\etc\hosts.
1.1.14.4. Batch files
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %BATCH_HOME%;. Also add a New system variable BATCH_HOME pointing to P:\dev\apps\windows\batch. 1.1.15. Services
The Service Control command (sc.exe) provided by Microsoft is a command line program used for communicat- ing with the NT Service Controller and services. Some of the options are:
• sc qc [service_name] queries the configuration information for a service. • sc start [service_name] starts a service. • sc query [service_name] queries the status for a service. • sc stop [service_name] sends a stop request to a service. • sc create service_name creates a service in the registry. • sc delete service_name deletes a service from the registry.
To edit the Service properties launch the Service Management Console with services.msc.
If a service fails to start check the Application Log in Start → Administrative Tools → Event Viewer or launch the Event Viewer Console from the command line with eventvwr.msc and select Event Viewer (Local) → Windows Logs → Application.
Note
If you see something like “The description for Event ID ( 0 ) in Source ( … ) cannot be found. … You may be able to use the /AUXSOURCE= flag to retrieve this description; … The following information is part of the event: …” it's Windows telling you that it can't find the template to nicely format the event content, so it'll give you the raw event data.
1.1.15.1. Delayed service loading
To make sure that for example the Database Service is available when the Application Server Service starts create a dependency between both services as follows:
1. Start regedit. 2. Select HKEY_LOCAL_MACHINE → SYSTEM → CurrentControlSet → services → appserverService. 3. Right click on appserverService and select New → Multi-String Value. 4. Set the name to DependOnService. 5. Right click on DependOnService and select Modify. 6. Set the Value data to databaseService (in case of multiple services create a space separated list). 7. Run: sc qc appserverService to verify the dependencies.
Note
In case of the following error: “[SC] GetServiceConfig needs nnn bytes” try sc qc appserverService , where the specified buffersize is equal or larger than the number of bytes in the error message.
6 Operating System
For an alternative take a look at the manual Startup and Shutdown batch files.
1.1.15.2. Service Wrapper
WinSW creates a wrapper executable that can be used to host any executable as a Windows Service with a less restrictive license than the Java Service Wrapper.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %SERVICES_HOME%;. Also add a New system variable SERVICES_HOME pointing to P:\dev\apps\windows\services.
To configure any executable as a Windows Service create a service configuration file exampleService.xml in P:\dev\apps\windows\services:
exampleExample ServiceSample service configuration.SpoolerMessengerP:\dev\logs\windows\servicesrollrunstop%TMPDIR%exampleApplication.exe
Get a copy of the latest binary [version 1.18], rename it to exampleService.exe and place it next to the exampleService.xml in P:\dev\apps\windows\services.
Install the dummy service with exampleService install. Start it with exampleService start or sc start example. Other winsw options are status, stop and uninstall.
Verify the installation with services.msc and view the event logs with eventvwr.msc.
Warning
Windows XP kills all services that take longer than the WaitToKillServiceTimeout registry set- ting (default is 20000 milliseconds) on shutdown. When experimenting with WinSW usage for MySQL this resulted in the following error message “InnoDB: Database was not shut down nor- mally!”.
1.1.15.3. Startup and Shutdown batch files
To start and stop the services in a certain order make use of the net command instead of sc because this one waits for the service stopped signal (see also warning about WaitToKillServiceTimeout). For the things to come create the following two batch files P:\dev\apps\windows\batch\services-startup.bat:
net start james net start mysqlmaster net start fisheye net start jira net start nexus net start sonarqube net start tomcat net start payara
REM pause 10 seconds ping -n 10 127.0.0.1 > NUL
and P:\dev\apps\windows\batch\services-shutdown.bat:
net stop payara
7 Chapter 1.
net stop tomcat net stop sonarqube net stop nexus net stop jira net stop fisheye net stop mysqlmaster net stop james
REM pause 10 seconds ping -n 10 127.0.0.1 > NUL
1.1.16. Data Execution Prevention
Data Execution Prevention (DEP) is a set of hardware and software technologies that perform additional checks on memory to help prevent malicious code from running on a system.
When a program (for example java.exe version 1.3.1 update 20) won't start and displays the following error message “To help protect your computer, Windows has closed this program.”, you can manually exclude the pro- gram from the DEP feature.
Press +Break keys to open the Windows System Properties. Select Advanced → Settings → Data Execution Prevention → Turn on DEP for all programs and services except those I select and Add… the program. 1.1.17. Text editor
A text editor is a type of program used for editing plain text files.
1.1.17.1. Notepad++
Notepad++ is a free (as in “free speech” and also as in “free beer”) source code editor and Notepad replacement that supports several languages.
Download the binary: npp.6.8.2.Installer.exe [version 6.8.2].
Run this .exe file to install Notepad++ in P:\dev\apps\editor\notepad++.
Notepad++ supports a few command line parameters to control its startup.
1.1.17.2. UltraEdit
UltraEdit is the ideal text, HTML and hex editor, and an advanced PHP, Perl, Java and JavaScript editor for programmers.
Download the archive: ue_english.zip [version 16.30.0.1003]* and any required dictionaries and manuals.
Extract the .zip file to C:\tmp. Run the MSI Installer ue_english.msi to custom install UltraEdit in P:\dev \apps\editor\ultraedit.
Install the dictionaries with the MSI Installer, for example Dutch.msi, in P:\dev\apps\editor\ultraed- it\GNU\aspell. 1.1.18. Archive Tools
A file archiver is a computer program that combines a number of files together into one archive file, or a series of archive file, for easier transportation or storage. Many file archivers employ archive formats that provide loss- less data compression to reduce the size of the archive which is often useful for transferring a large number of individual files over a high latency network like the Internet.
1.1.18.1. WinRAR
WinRAR is a powerful archive manager.
Download the binary: winrar-x64-560.exe [version 5.60].
Run this .exe file to install WinRAR in P:\dev\apps\archive\winrar.
Configure Options → Settings… → Viewer → External viewer name to point to for example P:\dev\apps \editor\ultraedit\Uedit32.exe. Use Ctrl+H to switch to and from Flat folders view.
1.1.18.2. 7-Zip
7-Zip is a file archiver with a high compression ratio.
8 Operating System
Download the binary (MSI Installer): 7z1805-x64.msi [version 18.05].
Run this .msi file to install 7-Zip in P:\dev\apps\archive\7zip.
1.1.18.3. HJ-Split
HJ-Split multi-platform file splitter and joiner.
Download the archive: hjsplit.zip [version 3.0].
Extract the .zip file to P:\dev\apps\archive\hjsplit.
1.1.19. GnuWin32
GnuWin32 provides ports of tools with a GNU or similar open source license to MS-Windows.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environ- ment Variables and Ed it the Path to append %GNUWIN32_HOME%\bin; . Also add a New system variable GNUWIN32_HOME pointing to P:\dev\apps\gnuwin32.
Note
Why big open-source projects are fleeing SourceForge's free software hub.
1.1.19.1. DiffUtils
DiffUtils shows differences between files.
Download the Win32 binary: diffutils-2.8.7-1.exe [version 2.8.7.1].
Run this .exe file to install DiffUtils in P:\dev\apps\gnuwin32.
Verify the installation with diff -version.
1.1.19.2. Patch
Patch applies a diff file to an original.
Download the Win32 binary: patch-2.5.9-7-setup.exe [version 2.5.9.7].
Run this .exe file to install Patch in P:\dev\apps\gnuwin32.
Verify the installation with patch -version.
Sample command patch -i viewvc-1.1.5.patch.diff -p 0 --verbose (add the option --dry-run if you want to check the patch first). You can revert the patch by adding the option -R.
Important
If you run patch and get one of the following error messages: “can't find file to patch at input line …” or “Assertion failed: hunk, file …, line …”, make sure the directory separator (for the entries: Index:, --- and +++) is the backslash (\) and not the forwardslash (/) and the file contains CR-LF as line endings instead of just LF.
1.1.19.3. LibIconv
LibIconv converts from one character encoding to another through Unicode conversion. It has also limited support for transliteration, i.e. when a character cannot be represented in the target character set, it is approximated through one or several similar looking characters.
Instead of the older binary libiconv-1.9.2-1.exe download the one maintained at ftp.zlatkovic.com (or mir- ror): iconv-1.9.2.win32.zip [version 1.9.2].
Extract this .zip file to C:\tmp and move the three subdirectories bin, include and lib to P:\dev\apps \gnuwin32.
Verify the installation with iconv --version.
9 Chapter 1.
1.1.19.4. ZLib
ZLib is designed to be a free, general-purpose, legally unencumbered -- that is, not covered by any patents -- lossless data-compression library for use on virtually any computer hardware and operating system. The zlib data format is itself portable across platforms.
Instead of the older binary zlib-1.2.3.exe download the one maintained at ftp.zlatkovic.com (or mirror ): zlib-1.2.5.win32.zip [version 1.2.5].
Extract this .zip file to C:\tmp and move the three subdirectories bin, include and lib to P:\dev\apps \gnuwin32.
1.1.19.5. LibXml2
LibXml2 is the XML C parser and toolkit developed for the Gnome project, but usable outside of the Gnome plat- form.
Instead of the older archive libxml2-2.4.12-bin.zip download the one maintained at ftp.zlatkovic.com (or mirror): libxml2-2.7.8.win32.zip [version 2.7.8].
Extract this .zip file to C:\tmp and move the three subdirectories bin, include and lib to P:\dev\apps \gnuwin32.
Important
To upgrade the libxml2.dll you might need to stop the Apache HTTP Server.
Verify the installation with xmllint -version. 1.1.20. Merging tool
Merging (also called integration) in revision control, is a fundamental operation that reconciles multiple changes made to a revision-controlled collection of files. Most often, it is necessary when a file is modified by two people on two different computers at the same time. When two branches are merged, the result is a single collection of files that contains both sets of changes.
1.1.20.1. KDiff3
KDiff3 compares or merges two or three text input files or directories.
Download the binary: KDiff3-32bit-Setup_0.9.98-3.exe [version 0.9.98].
Run this .exe file to install KDiff3 in P:\dev\apps\editor\kdiff3.
1.1.20.2. WinMerge
WinMerge is an Open Source differencing and merging tool for Windows. It can compare both folders and files, presenting differences in a visual text format that is easy to understand and handle. The WinMerge command line accepts several parameters in addition to the paths to compare.
Download the binary: WinMerge-2.14.0-Setup.exe [version 2.14.0].
Run this .exe file to install WinMerge in P:\dev\apps\editor\winmerge.
To configure an different external editor select Edit → Options… → System and change External editor in P: \dev\apps\editor\notepad++\notepad++.exe.
Tip
If the WinMerge Windows Explorer context menu conflicts with the New menu usage then disable it by selecting Edit → Options… → Shell Integration and uncheck Add to context menu.
1.1.21. Secure Shell
Secure Shell or SSH is a network protocol that allows data to be exchanged using a secure channel between two networked devices.
10 Operating System
1.1.21.1. PuTTY
PuTTY is a free implementation of Telnet and SSH for Win32 and Unix platforms, along with an xterm terminal emulator.
Download the 64-bit binaries: putty.exe, puttygen.exe, pageant.exe and plink.exe [version 0.70].
Just copy those .exe file in P:\dev\apps\shell\putty.
Tip
Exporting PuTTY Sessions or Putty Settings Migration.
1.1.21.1.1. Default Settings
To change PuTTY's default settings run PuTTY make the required changes and set Saved Sessions: Default Settings and click Save.
Tip
If you find your sessions are closing unexpectedly (most often with “Connection reset by peer”) after they have been idle for a while, you might want to try using the Connection option Se- conds between keepalives (0 to turn off): 60.
In Window → Appearance Change… Font Lucida Console, Font Style Regular and Size 12. 1.1.22. The Best Application Launcher for Windows
Do you still launch applications by pressing the Windows key and searching for your app? That's a serviceable way to get your apps up and running, but it's pretty limited. If you've never used an app launcher, now's the time to try one: they're faster and more powerful than any built-in search system, and they can do a lot more than just launch apps. Here's why you should be using one (and everything you can do with it).
Launchy is a free cross-platform utility designed to help you forget about your start menu, the icons on your desktop, and even your file manager.
Download the binary: Launchy2.5.exe [version 2.5].
Run this .exe file to install Launchy in P:\dev\apps\windows\launchy.
Additionally install required plugins sucha as putty-launchy-plugin [version 2.4]. 1.1.23. File Transfer
File transfer is a generic term for the act of transmitting files over a computer network or the Internet. There are numerous ways and protocols to transfer files over a network. Computers which provide a file transfer ser- vice are often called file servers. Depending on the client's perspective the data transfer is called uploading or downloading. 1.1.23.1. WinSCP
WinSCP is an open source free SFTP client and FTP client for Windows. Legacy SCP protocol is also supported. Its main function is safe copying of files between a local and a remote computer.
Download the binary: WinSCP-5.13.3-Setup.exe [version 5.13.3].
Run this .exe file to install WinSCP in P:\dev\apps\ftp\winscp.
Integration with PuTTY through Tools → Preferences… → Integration → Applications → Browse to select P:\dev\apps\shell\putty\putty.exe as PuTTY/Terminal client path.
11 Chapter 1.
Use Ctrl+P to Manage → Open in PuTTY of current session Using WinSCP as PuTTY session manager with Folders and Incremental search.
Tip
Transferring the Configuration to another computer/location or How do I import/export my settings. Alternatively use Tools → Import Sites… → PuTTY/FileZilla.
1.1.23.2. All-in-one FTP/SFTP/HTTP/WebDAV Client
BitKinex integrates the functionality of an innovative FTP, SFTP and WebDAV client for Windows.
Download the binary: bitkinex323.exe [version 3.2.3].
Run this .exe file to install BitKinex in P:\dev\apps\ftp\bitkinex. 1.1.24. Graph Visualization Software
Graphviz is open source graph visualization software. Graph visualization is a way of representing structural information as diagrams of abstract graphs and networks. Automatic graph drawing has many important appli- cations in software engineering, database and web design, networking, and in visual interfaces for many other domains.
Download the binary (MSI Installer): graphviz-2.38.msi [version 2.38].
Run this .msi file to install Graphviz in P:\dev\apps\editor\graphviz.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environ- ment Variables and Ed it the Path to append %GRAPHVIZ_HOME%\bin; . Also add a New system variable GRAPHVIZ_HOME pointing to P:\dev\apps\editor\graphviz.
Verify the installation with dot -v and click Ctrl+C. 1.1.25. Debugging Tools
The Microsoft debuggers are fully capable of running on computers with x86-based, Itanium, or x64-based proces- sors. The debuggers can debug the Windows operating system, applications, services, and drivers that run on the operating system.
How to Analyse Bugcheck and Process Crash Dumps. 1.1.26. Viewers
A file viewer is application software that presents the data stored in a computer file in a human-friendly form. Discontinued versions of Office viewers.
1.1.26.1. Excel
Excel Viewer lets you view and print Microsoft Excel documents on a computer that does not have Excel in- stalled.
Download the binary: ExcelViewer.exe and the following list of patches [version SP3 up to KB2965209].
Run this .exe file and patches to install Excel Viewer in P:\dev\apps\editor\excel. 1.1.26.2. Word
Word Viewer lets you view and print Microsoft Word documents on a computer that does not have Microsoft Word installed.
Download the binary: wordview_en-us.exe and the following list of patches [version SP3 up to KB3055054].
Run this .exe file to install Word Viewer in P:\dev\apps\editor\word. 1.1.26.3. PowerPoint
PowerPoint Viewer lets you view full-featured presentations created in PowerPoint 97 and later versions.
Download the binary: PowerPointViewer.exe and the following list of patches [version SP1 up to KB3054840].
12 Operating System
Run this .exe file and patches to install Excel Viewer in P:\dev\apps\editor\powerpoint. 1.1.26.4. Foxit PDF Reader
Foxit Reader is a free PDF document viewer, with incredible small size, breezing-fast launch speed and rich feature set.
Download the binary: FoxitReader91_enu_Setup_Prom.exe and available add-ons [version 9.1.0.5096].
Important
FoxitReader91_Setup_Prom_IS.exe installer doesn't allow to choose installation directo- ry.
Run this .exe file to custom install Foxit Reader (and optionally install ConnectedPDF and the Plug-ins: Word, PPT and Excel) in P:\dev\apps\editor\foxit (see also How to disable or opt-out from ConnectedPDF). Decline or Accept the optional offer.
After installation take a look at the following settings in File → Preferences:
• ECM Integration: Allow you to choose when opening the document. • Foxit Drive: Always show Extras tab. • General: • User Experience Improvement: Collect data to improve user experience. • Application Startup: Show floating promotion page and Show Advertisement. • Trust Manager: Enable/disable the Safe Reading Mode. 1.1.27. Graphic Design and Digital Media Design 1.1.27.1. Snipping Tool
Snipping Tool to capture a screen shot, or snip, of any object on your screen, and then annotate, save, or share the image. It is available at %windir%\system32\SnippingTool.exe.
Tip
To capture context menus first start a New snip but cancel with Esc. Now open the desired menu and submenus followed by Ctrl+PrintScreen to restart capturing (see for example Fig- ure 15.9, “Eclipse Mylyn Task List view”).
1.1.27.2. IrfanView
IrfanView is a very fast, small, compact and innovative graphic viewer for Windows.
Download the binary: iview451_setup.exe [version 4.51].
Run this .exe file and patches to install Excel Viewer in P:\dev\apps\gfx\irfanview.
Download the binary: iview451_plugins_setup.exe [version 4.51].
Run this .exe file and patches to install Excel Viewer in P:\dev\apps\gfx\irfanview. 1.1.27.3. Paint.NET
Paint.NET is just about perfect for the graphic design needs of a non-graphic-designer when you need to crop, cut, or otherwise edit an image and Windows' built-in Paint isn't quite enough [version 4.0.21]. 1.1.27.4. GIMP
GIMP is the GNU Image Manipulation Program. It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages [version 2.10.2]. 1.1.27.5. Inkscape
Inkscape is professional quality vector graphics software for creating a wide variety of graphics such as illustra- tions, icons, logos, diagrams, maps and web graphics. Inkscape uses the W3C open standard SVG (Scalable Vector Graphics) as its native format [version 0.92.3].
13 Chapter 1.
1.1.27.6. VLC
VLC (VideoLAN Client) is a free and open source cross-platform multimedia player and framework that plays most multimedia files as well as DVDs, Audio CDs, VCDs, and various streaming protocols [version 3.0.3]. 1.1.28. Utilities
There are some excellent free software utilities available that are every bit as good as their commercial coun- terparts, and sometimes even better. One software publisher, with an almost unknown name, but with very pop- ular products is Piriform Software. 1.1.28.1. CCleaner
CCleaner is a system optimization, privacy and cleaning tool. It removes unused files from your system - allowing Windows to run faster and freeing up valuable hard disk space. It also cleans traces of your online activities such as your Internet history. Additionally it contains a fully featured registry cleaner.
Warning
Update to the CCleaner 5.33.6162 Security Incident clarifying what happened and outlining our next steps in protecting CCleaner customers. Infected CCleaner downloads from official servers.
Download the binary: ccsetup544.exe [version 5.44.6575].
Run this .exe file to install CCleaner in P:\dev\apps\windows\ccleaner. 1.1.28.2. Defraggler
Use Defraggler to defrag your entire hard drive, or individual files - unique in the industry.
Download the binary: dfsetup222.exe [version 2.22.995].
Run this .exe file to install Defraggler in P:\dev\apps\windows\defraggler. 1.1.28.3. Recuva
Accidentally deleted an important file? Lost something important when your computer crashed? No problem! Recuva recovers files deleted from your Windows computer, Recycle Bin, digital camera card, or MP3 player.
Download the binary: rcsetup153.exe [version 1.53.1087].
Run this .exe file to install Recuva in P:\dev\apps\windows\recuva. 1.1.28.4. Speccy
Speccy is an advanced System Information tool for your machine.
Download the binary: spsetup132.exe [version 1.32.740].
Run this .exe file to install Defraggler in P:\dev\apps\windows\speccy. 1.1.28.5. Resource Hacker
Resource Hacker is a utility to view, modify, rename, add, delete and extract resources in 32bit Windows exe- cutables and resource files.
Download the archive: reshacker_setup.exe [version 5.1.1].
Extract the .zip file to P:\dev\apps\windows\resource-hacker.
• Download a Firefox version that speaks your language. Keeping your plugins up to date helps Firefox run safely and smoothly. Skip Firefox's Add-On Compatibility Check with This Small Extension. • Waterfox is a Faster, 64-Bit Optimized Version of Firefox for Windows PCs. Firefox 64-Bit Now Available for Windows with Improved Performance. Users will notice that NPAPI Plugins in Firefox will not work in this 64- bit version. • How to stop Firefox from making automatic connections. OpenH264 Now in Firefox. • The Mozilla Developer Center (MDC) supports the growth and development of Firefox and the web by pro- viding comprehensive, accurate, and up-to-date documentation and news about Firefox and web development technologies. • LifeHacker tips & downloads for getting things done. • The POODLE SSL vulnerability: disabling SSL v3 in your Azure Virtual Machines and Web Roles. POODLE Vul- nerability Affects TLS 1.0, TLS 1.1. • Mozilla Improves Firefox's Certificates Visibility and Mozilla To Remove Favicons From Firefox URL Bar. • 15 Power User Tips for Tabs in Firefox. • Add-ons extend Firefox, letting you personalize your browsing experience (see also The Best Firefox Ad- dons):
• Firebug is a discontinued free and open-source web browser extension for Mozilla Firefox that facilitated the live debugging, editing, and monitoring of any website's CSS, HTML, DOM, XHR, and JavaScript.
• Firefox Developer Tools to examine, edit, and debug HTML, CSS, and JavaScript on the desktop and on mobile. Migrating from Firebug .
• NoScript allows JavaScript and Java execution only for trusted domains of your choice (e.g. your home- banking web site). NoScript optionally blocks Flash and other potentially exploitable plugins too, and pro- vides the most powerful Anti-XSS protection available in a browser.
• Video DownloadHelper is a tool for web content extraction. Its purpose is to capture video and image files from many sites.
• Redirector is a browser add-on for Firefox, Chrome and Opera. The add-on lets you create redirects for specific webpages, e.g. always redirect http://bing.com to http://google.com. Don't you hate it when a website automatically redirects you to a localized version?
• Open Add-ons page with Ctrl+Shift+A to select Themes → Dark → Enable the native Mozilla dark color scheme. Possible alternative for FT DeepDark is a smooth dark theme for Firefox.
• Bulk Media Downloader grabs and downloads media (image, audio, and video) resources by monitoring network (like FlashGot or DownThemAll!).
• Tab Rotator automatically switches between the open tabs of a browser window after a certain amount of time.
• Tab Session Manager to save and restore the state of windows and tabs. It also supports automatic saving. Possible alternative for Session Manager.
• Conex tries to replace some functionality from the discontinued Tab Groups. Another possible alter- native: Sync Tab Groups is an easy way to organize your current tabs depending on a topic. Each open group is synchronized with a window so that all tabs change are saved. Hence, you can close a current tabs- flow while opening another one.
• Forget Me Not to make the browser forget website data (like cookies, local storage, etc.), except for the data you want to keep by adding domains to a whitelist, graylist, or redlist. Possible alternative for Remove Cookie(s) for Site. • The about:about page is an index of Firefox's about pages.The about:support page provides information you might need to troubleshoot problems with Firefox. About:config is the about: page most geeks have heard of. It provides access to all sorts of internal Firefox options that aren't exposed in the user interface - it's the go-to place for tweaking Firefox. 2.1.2. Firefox installation guide
Download the Win64 binary: Firefox Setup 61.0.exe [version 61.0].
15 Chapter 2.
Run this .exe file to custom install Firefox in P:\dev\apps\browser\firefox.
Optionally install useful extensions and pimp the browser theme for example with the the above mentioned add- ons. To reactivate the menu bar type Alt and check View → Toolbars → Menu Bar.
Tip
http://www.google.com/ncr stops google.com from redirecting to local country or lan- guage version of Google.
2.1.2.1. Custom profile folder
Mozilla Firefox stores all your personal settings, such as bookmarks, passwords and extensions, in a profile folder.
To move it to a new location, all you have to do is close Firefox and move the default profile folder and set the absolute path for the new location in %APPDATA%\Mozilla\Firefox\profiles.ini:
To display a list of used preferences, as well as a search bar, type about:config (see also About protocol) in the Firefox address bar:
security.tls.version.(min / max / fallback-limit) Set to 3 to make TLS 1.2 is the minimum required / maximum supported encryption protocol (verify with POODLE Test).
browser.cache.disk.parent_directory To specify in which folder the Cache is stored, add a New → String preference browser.cache.disk.parent_directory, and set the value to C:\tmp\firefox. The Cache directory location can be viewed with about:cache.
browser.tabs.closeButtons The preference controls how the close button for tabs is displayed. The value 3 displays a single close button at the end of the tab strip.
Tip
A handy keyboard shortcut for undoing accidentally closed tabs is Ctrl+Shift+T and acci- dentally closed windows is Ctrl+Shift+N.
browser.tabs.onTop Disable this preference to put the tabs back on bottom like they used to be.
browser.urlbar.trimURLs Disable url trimming in Firefox so that the http protocol is shown on all tabs again.
browser.urlbar.formatting.enabled Disable url formatting so that the url is displayed in one color.
network.prefetch-next To stop Firefox from silently prefetching hinted documents set the preference to false.
Note
The saving of tabs when quitting Firefox 4 has been disabled by default. To (re)activate all warn- ing dialogs enable the following 4 settings and reset the last one:
16 Browser
browser.showQuitWarning To restore the “Quit Dialog”: Save and Quit, Quit or Cancel set the preference to true. Unless browser.startup.page is 3 (see also below).
browser.tabs.warnOnClose To restore the “Confirm Close Dialog”: Close tabs or Cancel set the preference to true. browser.warnOnQuit To restore the “Exit Dialog”: Save and Quit, Quit or Cancel set the preference to true. browser.startup.page Each time the web browser starts, this preference is consulted to determine what to dis- play. It has superseded browser.sessionstore.resume_session as the preference determin- ing whether saved sessions are restored. The value 3 resumes the previous browser ses- sion.
2.2. Safari
Safari renders web pages at lightning speed. It works on your iPad, iPhone, iPod touch, Mac, and PC. 2.2.1. Resources
• Download a Safari version for Mac and PC. • Learn about the innovative features available in Safari. • Safari Extensions are a great way for you to add new features to Safari. Built by developers, Safari Extensions use the latest HTML5, CSS3, and JavaScript web technologies. They're digitally signed and sandboxed for im- proved security. You can install extensions with one click — no need to restart Safari. 2.2.2. Safari installation guide
Download or download.cnet.com the Win32 binary: SafariSetup.exe [version 5.1.7].
Run this .exe file to custom install Safari in P:\dev\apps\browser\safari. 2.3. Chrome
Chrome runs websites and applications with lightning speed. 2.3.1. Resources
• Download a Chrome version. • Chrome Cleanup Tool application will scan and remove software that may cause problems with Chrome, such as crashes, unusual startup pages or toolbars, unexpected ads you can't get rid of, or otherwise changing your browsing experience. • Chrome has many useful features built in, including translation in the browser, apps, extensions, themes, and more. • Chrome Web Store is an online marketplace where you can discover thousands of apps, extensions and themes for Google Chrome. • POODLE Disabling SSLv3 Support in Browsers. • Quickly Save Tab Sessions in Chrome Without Installing Extensions. 2.3.2. Chrome installation guide
Download the Win32 binary: ChromeSetup.exe for the Google Chrome Web Browser only [version 49.0.2623.112]. Or as an alternative download the standalone installer ChromeStandaloneSetup.exe (see also Disabling SSLv3 Support in the above mentioned Resources).
Run one of those .exe file to install Chrome.
To simulate custom installation of Chrome in P:\dev\apps\browser\google create a symbolic link using Junc- tion as follows:
The extra junction for the browser itself is now reverted because it caused “The application has failed to start because its side-by-side configuration is incorrect.” for new_chrome.exe af-
17 Chapter 2.
ter the update to 49.0.2623.110. Reinstall resulted in the following recurring error “An error occurred while checking for updates: Update check failed to start (error code 1: 0x80004005).” when viewing, in the top right, the Chrome menu → Help → About Google Chrome version information (Chrome will check for updates when you're on this page).
rem Note: on 32-bit use %PROGRAMFILES% mkdir P:\dev\apps\browser\chrome mkdir "%PROGRAMFILES(X86)%\Google"
Java Platform, Standard Edition (Java SE) lets you develop and deploy Java applications on desktops and servers, as well as in today's demanding embedded environments. Java offers the rich user interface, perfor- mance, versatility, portability, and security that today's applications require.
3.1.1. Resources
• Download the latest complete environment and runtime environment. Also available are the Next Release (Early Access) and Previous Releases. The Archive is providing the products available as a courtesy to develop- ers for problem resolution. The products available here have completed the Sun EOL process. • Security: Critical Patch Updates, Security Alerts and Bulletins. • Upgrade checklist: • Related products: -. • References: Apache James - Windows Service wrapper.java.command, Tomcat - Windows Ser- vice tomcat8.exe or tomcat8w.exe, Payara - asenv.bat AS_JAVA, FishEye - Windows Service wrapper.java.command, JIRA - Windows Service tomcat7.exe or tomcat7w.exe, Nexus - Windows Service wrapper.java.command, Jenkins - Slave Configuration jeeves\jenkins-slave.xml, Eclipse - installation guide eclipse.ini -vm [119], Eclipse - Maven integration Installed JREs and SonarQube - Windows Service wrapper.java.command. • Integration configuration changes: Java - Security and Usability JCE. • What is Java Auto Update? How do I change notify settings? Where is the Java Control Panel on Windows? • Oracle Java on Windows: C:\ProgramData\Oracle\Java\javapath. How to set custom Java path after installing JDK 8. • java -version results in “Error occurred during initialization of VM java/lang/NoClassDefFoundError: ja- va/lang/Object” seems to indicate that the JVM can't find its own rt.jar file. The problem comes from an improper Java installation. Try uninstalling and then reinstalling Java. • Java Platform, Standard Edition (Java SE) 8. • Mkyong.com is created, written by, and maintained by Yong Mook Kim, aka Mkyong. • Networking IPv6 User Guide and IPv6 Test.java (see also IPv6Test.java). • A tutorial example on how to use JVM property, javax.net.debug, to turn on the SSL socket communication debug option. The debug output message can help you to know what exactly happens at the SSL layer. • Developer Tools. • TinyMCE is a platform independent web-based JavaScript HTML WYSIWYG editor control released as open source under LGPL. TinyMCE enables you to convert HTML textarea fields or other HTML elements to editor instances. Integrations of TinyMCE into your favorite framework or CMS. TinyMCE Advanced plugin will let you add, remove and arrange the buttons that are shown on the Visual Editor toolbar. • Tired of Null Pointer Exceptions? Consider Using Java SE 8's Optional! • Inversion of Control Containers and the Dependency Injection pattern by Martin Fowler. • Dependency Injection Containers are Code Polluters. • My case against autowiring. • I was wrong: constructor vs. setter injection. • Fluent Interface by Martin Fowler. • Creation, dynamic loading and instrumentation with javaagents. • Oracle Technology Network (OTN) Java is Oracle Corporation's official, interactive online/offline commu- nity for Oracle technical professionals. • java.net the Source for Java Technology Collaboration.
Important
The Java.net and Kenai.com forges will be going dark on April 28, 2017. The new location is the Java Community Space.
• OpenJDK is the place to collaborate on the open-source JDK. • Java Tools and Technologies Landscape Report 2016. • Shameless plugs:
• Article about the JModalWindow Project.
• GeeCON Prague 2015: Bas Knopper - Making Darwin Proud. Coding Evolutionary Algorithms (PDF). • MarI/O - Machine Learning for Video Games. • Destination NASA - Evolutionary Antenna Synthesis.
• Crealysm (BooH) by Cris Kop. 19 Chapter 3.
3.1.2. JDK installation guide
Download the Win64 binary: jdk-8u162-windows-x64.exe [version 1.8.0_162].
Note
What's the difference between odd (a Critical Patch Update [CPU]) and even (a corresponding Patch Set Update [PSU]) Java updates? See for example bug “Decode error with Tomcat version 7.x” noted as Known Issue in release notes of 8u151/152 JDK-8191040, which was fixed in Or- acle 8u162 JDK-8189789 and OpenJDK 8u171 JDK-8189789.
Run the 64-bit .exe file to install the JDK Development Tools with Change… into P:\dev\apps\prg\ja- va-x64\jdk1.8.0_162. After the JDK is installed Change… the JRE Destination Folder into P:\dev\apps \prg\java-x64\jre1.8.0_162.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %JAVA_HOME%\bin;. Also add a New system variable JAVA_HOME point- ing to P:\dev\apps\prg\java-x64\jdk1.8.0_162.
Verify the installation with java -version. 3.1.3. Thoughts on Java SE, Java Security and Usability
When building inter-connected applications, developers frequently interact with TLS-enabled protocols like HTTPS:
• Diagnosing TLS, SSL, and HTTPS. • JDK 8 will use TLS 1.2 as default. • Instructions to disable SSL v3.0 in Oracle JDK and JRE. • Release Notes for JDK 8. • Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8.
When working with https URLs that cause an error “SSLHandshakeException: Received fatal alert: handshake_failure” in Java but work in the browser, you might want to install the files, from the above mentioned JCE, local_policy.jar and US_export_policy.jar in:
JCE policy changes in Java SE 8u151, 8u152 and 8u162. It is now enabled by default,but can be set specifically in %JAVA_HOME%\jre\lib\security\java.security:
#crypto.policy=limited | unlimited
To test it, just run: jrunscript -e "print (javax.crypto.Cipher.getMaxAllowedKeyLength('AES'))".
3.1.3.1. Standard Edition Tools Reference
Java Platform, Standard Edition Tools Reference describes for example the general purpose options that are specific to the Java HotSpot Virtual Machine such as -Xrs, which reduces the use of operating system signals by the JVM.
Shutdown hooks enable orderly shutdown of a Java application by running user cleanup code (such as closing database connections) at shutdown, even if the JVM terminates abruptly.
The JVM watches for console control events to implement shutdown hooks for unexpected termination. Specif- ically, the JVM registers a console control handler that begins shutdown-hook processing and returns TRUE for CTRL_C_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, and CTRL_SHUTDOWN_EVENT.
The JVM uses a similar mechanism to implement the feature of dumping thread stacks for debugging purposes. The JVM uses CTRL_BREAK_EVENT to perform thread dumps.
If the JVM is run as a service (for example, as a servlet engine for a web server), then it can receive CTRL_LOGOFF_EVENT but should not initiate shutdown because the operating system will not actually termi- nate the process. To avoid possible interference such as this, the -Xrs option can be used. When the -Xrs op- tion is used, the JVM does not install a console control handler, implying that it does not watch for or process CTRL_C_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, or CTRL_SHUTDOWN_EVENT.
20 Programming Languages
Note
Testing Log off for user .\Tomcat under which the windows services, without the option - Xrs, are running, didn't result in the service stopping. Tested with services tomcat, jeeves and nexus.
There are two consequences of specifying -Xrs:
• Ctrl+Break thread dumps are not available. • User code is responsible for causing shutdown hooks to run, for example, by calling System.exit() when the JVM is to be terminated. 3.2. Visual Studio C# and C++
Visual Studio Community is a free, fully-featured, and extensible IDE for creating modern applications for Win- dows, Android, and iOS, as well as web applications and cloud services. Multi-language support includes C#, Visual Basic, F#, C++, JavaScript, TypeScript, Python, and more. Visual Studio guides you as you write, debug, and test code — no matter what language you choose.
Download the Win32 binary: vcs_web.exe [version 10.0.30319.1].
Run this .exe file to install it for from the Internet. Install without the optional products into a new destination folder P:\dev\apps\prg\visualstudio-2015. 3.3. Python
Python is a dynamic object-oriented programming language that can be used for many kinds of software de- velopment. It offers strong support for integration with other languages and tools, comes with extensive stan- dard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code. Fans of Python use the phrase “batteries included” to describe the standard library, which covers everything from asynchronous processing to zip files. 3.3.1. Resources
• Download Standard Python Software. • Upgrade checklist: • Related products: Python - installation guide Setuptools, Python - installation guide Pygments and Apache HTTP Server - Module configuration mod_wsgi. • References: -. • Integration configuration changes: -. • Python's standard documentation. • Python for Windows Extensions. • The Python Package Index is a repository of software for the Python programming language such as setup- tools. 3.3.2. Python installation guide
Download the Win64 binary (MSI Installer): python-3.4.3.amd64.msi [version 3.4.3].
Run this .msi file to install it for all users. Install all the features into a new destination folder P:\dev\apps \prg\python-3.4.3.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %PYTHON_HOME%;%PYTHON_HOME%\Scripts;. Also add a New system variable PYTHON_HOME pointing to P:\dev\apps\prg\python-3.4.3 and reboot the server.
Verify the installation with python --version.
Download the Python for Windows Extensions Win64 binary: pywin32-219.win-amd64-py3.4.exe [version 219].
Run this .exe file to install the Windows extensions in P:\dev\apps\prg\python-3.4.3. 3.3.3. Python Enterprise Application Kit
PEAK is the Python Enterprise Application Kit. If you develop enterprise applications with Python, or indeed almost any sort of application with Python, PEAK may help you do it faster, easier, on a larger scale, and with fewer defects than ever before. The key is component-based development, on a reliable infrastructure.
Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.
21 Chapter 3.
3.3.3.1. Setuptools installation guide
Download the install script ez_setup.py that will download the appropriate setuptools-x.x.x-pyx.x.egg to for example python P:\dev\tools\prg\python\ez_setup.py [version 18.3.1].
Run python P:\dev\tools\prg\python\ez_setup.py to install setuptools-18.3.1-py3.4.egg into P:\dev\apps\prg\python-3.4.3.
Note
In case of an upgrade of the setuptools delete the setuptools.pth and setup- tools-x.x.x-pyx.x.egg from P:\dev\apps\prg\python-3.4.3\Lib\site-pack- ages to avoid the following error message: “zipimport.ZipImportError: bad local file header in …\lib\site-packages\setuptools-0.9.6-py3.3.egg”.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %PYTHON_HOME%\Scripts;.
Verify the installation with easy_install --version. 3.3.4. Pygments
Pygments is a generic syntax highlighter for general use in all kinds of software such as forum systems, wikis or other applications (such as ViewVC) that need to prettify source code.
3.3.4.1. Pygments installation guide
Just run easy_install -i http://pypi.python.org/simple/ Pygments [version 2.0.2]. 3.4. Perl
Perl is a dynamic programming language created by Larry Wall and first released in 1987. Perl borrows features from a variety of other languages including C, shell scripting (sh), AWK, sed and Lisp. Perl was widely adopted for its strengths in text processing and lack of the arbitrary limitations of many scripting languages at the time. 3.4.1. Resources
• ActivePerl is the industry-standard Perl distribution, available for Windows, Linux, Mac OS X, Solaris, AIX and HP-UX. • Download the complete, ready-to-install Perl distribution. • Upgrade checklist: • Related products: Perl - Module DBI, Perl - Module PerlMagick and Apache HTTP Server - Module config- uration mod_perl. • References: -. • Integration configuration changes: -. • Online documentation. 3.4.2. Perl installation guide
Download the Win64 Binary (MSI Installer): ActivePerl-5.22.0.2200-MSWin32-x64-299195.msi [ver- sion 5.22.0.2200].
Note
In case of Perl usage in combination with Bugzilla and MySQL check availabitly of DBD-mysql.
Run this .msi file to install it for all users. Install all the features into a new destination folder P:\dev\apps\prg \perl-5.22.0.2200 and do not Add Perl to the PATH environment variable.
Important
Installing Perl into a directory that contains a space (e.g. C:\Program Files) will break the Template-Toolkit installer.
22 Programming Languages
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %PERL_HOME%\site\bin;%PERL_HOME%\bin;. Also add a New system variable PERL_HOME pointing to P:\dev\apps\prg\perl-5.22.0.2200 and reboot the server.
Verify the installation with perl -V. 3.4.3. Perl Package Manager user guide
Start the Perl Package Manager with: ppm gui.
Switch to View → Installed Packages or Ctrl+2 and select File → Verify Packages to check for updates.
Switch to View → Upgradable Packages or Ctrl+3. If the column Repo is not shown select View → View Column → Repo to activate it. To upgrade the package from the original repository just right click on it and select Install … or +. Should there be a higher version of the package available in a different repository right click on the installed package and select Remove … or - and then right click on the latest version of that package and select Install … or +.
After selecting the packages that must be upgraded switch to View → Packages to Install/Remove or Ctrl+4 and select File → Run Marked Actions or Ctrl+Enter to update.
Tip
Run ppm upgrade to check for available upgrades.
To add alternative repositories such as Scraps of Perl run the following command ppm repo add bribes-de- perl http://www.bribes.org/perl/ppm64/.
Verify the repository installations with ppm repo list or start the ppm gui Perl Package Manager and look at the available repositories with Edit → Preferences or Ctrl+P and Repositories. 3.4.3.1. Module DBI
The Perl DBI module provides a generic interface for database access. You can write a DBI script that works with many different database engines without change. To use DBI, you must install the DBI module, as well as a Data- Base Driver (DBD) module for each type of server you want to access. For MySQL, this driver is the DBD::mysql module.
Install the DBI module inclusive the MySQL implementation with ppm install DBI [version 1.631] and ppm install DBD::mysql [version 4.022]. 3.4.3.2. Module PerlMagick
PerlMagick is an objected-oriented Perl interface to ImageMagick. Use the module to read, manipulate, or write an image or image sequence from within a Perl script. This makes it very suitable for Web CGI scripts such as Bugzilla.
ImageMagick is a software suite to create, edit, and compose bitmap images. It can read, convert and write images in a variety of formats (over 100) including DPX, EXR, GIF, JPEG, JPEG-2000, PDF, PhotoCD, PNG, Postscript, SVG, and TIFF. Use ImageMagick to translate, flip, mirror, rotate, scale, shear and transform images, adjust image colors, apply various special effects, or draw text, lines, polygons, ellipses and Bézier curves.
Download the Win64 binary: ImageMagick-6.9.2-0-Q16-x64-dll.exe [version 6.9.2-0].
Run this .exe file to install ImageMagick into the following folder P:\dev\apps\gfx\imagemagick and des- elect all additional tasks:
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Envi- ronment Variables and Edit the Path to append %IMAGEMAGICK_HOME%;. Also add a New system variable IMAGEMAGICK_HOME pointing to P:\dev\apps\gfx\imagemagick and reboot the server.
Important
Because C:\Windows\System32 also contains a convert.exe file, which is used to convert a FAT volume to NTFS, place %IMAGEMAGICK_HOME%; before it in the Path.
Verify the installation with convert -version or convert logo: c:\tmp\logo.gif followed by identify c:\tmp\logo.gif.
23 Chapter 3.
An example of the convert command to reduce the image size before it is written to the PNG format: convert c:\tmp\logo.gif -resize 50% c:\tmp\logo.png.
3.4.3.2.1. PDF issue
After resizing a PNG for usage with the Maven jDocBook Plugin to generate a PDF, one might run into the following error situation:
java.lang.NullPointerException: Parameter alpha must not be null at org.apache.fop.pdf.AlphaRasterImage.(AlphaRasterImage.java:57) at org.apache.fop.pdf.AlphaRasterImage.(AlphaRasterImage.java:71) at … .ImageRenderedAdapter.setup(ImageRenderedAdapter.java:125) at org.apache.fop.pdf.PDFDocument.addImage(PDFDocument.java:809) at … handleImage(PDFImageHandlerRenderedImage.java:79)
Just remove the transparency from the PNG with the option -flatten, for example: convert 235283-32.png -flatten -resize 50% status4evar.png. 3.5. PHP
PHP is a widely-used general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. 3.5.1. Resources
• Please visit our downloads page, Windows binaries can be found on windows.php.net/download/. • Upgrade checklist: • Related products: Apache HTTP Server - Module configuration mod_php5. • References: -. • Integration configuration changes: -. • Online documentation. 3.5.2. PHP installation guide
Download the archive: php-5.6.13-Win32-VC11-x64.zip [version 5.6.13].
Extract this .zip file to P:\dev\apps\prg\php-5.6.13.
Note
Navigate into this directory and rename the file php.ini-development or php.ini-pro- duction to php.ini.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %PHP_HOME%;. Also add a New system variable PHP_HOME pointing to P:\dev\apps\prg\php-5.6.13 and reboot the server.
The VC11 build requires to have the Microsoft Visual C++ 2012 Redistributable Package SP4 (x64) [1] installed.
Verify the installation with php -v. 3.6. Simplified Wrapper and Interface Generator
SWIG is an interface compiler that connects programs written in C and C++ with scripting languages such as Perl, Python, Ruby, and Tcl. It works by taking the declarations found in C/C++ header files and using them to generate the wrapper code that scripting languages need to access the underlying C/C++ code. 3.6.1. Resources
Download the archive: swigwin-3.0.12.zip [version 3.0.12].
24 Programming Languages
Extract this .zip file to P:\dev\apps\prg.
Verify the installation with P:\dev\apps\prg\swigwin-3.0.12\swig.exe -version.
25 26 Chapter 4. Chapter 4. Build Tool 4.1. Ant
Apache Ant is a Java-based build tool. In theory, it is kind of like Make, but without Make's wrinkles. 4.1.1. Resources
• Download a binary distribution of Ant from one of the mirrors. • Upgrade checklist: • Related products: -. • References: Jenkins - configuration Ant. • Integration configuration changes: -. • Apache Ant Manual and an overall view of what each task can do. • External resources for Apache Ant: Tasks, Compiler Implementations, IDE integration tools, Source Control Systems, loggers, you name it. 4.1.2. Ant installation guide
Download the archive: apache-ant-1.10.4-bin.zip [version 1.10.4].
Extract this .zip file to P:\dev\apps\build.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %ANT_HOME%\bin;. Also add a New system variable ANT_HOME pointing to P:\dev\apps\build\apache-ant-1.10.4.
Verify the installation with ant -version. 4.2. Maven
Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of informa- tion. 4.2.1. Resources
• Download a binary distribution of Maven. • Upgrade checklist: • Related products: -. • References: Jenkins - configuration Maven, Eclipse - configuration Maven and Project - M2Eclipse life- cycle configuration [211]. • Integration configuration changes: Maven - settings.xml localRepository , Maven - Logging simplelogger.properties, Maven - Password Encryption server nexus, Nexus - Maven integration - mirror nexus-central / profile development / activeProfile development / server nexus +ossrh, Project - Site WebDAV server mvn-sites, Project - PGP Signatures - profile gnupg / server gpg.keyname and SonarQube - Login Token profile sonarCloud.
Note
Or just run P:\dev\apps\editor\winmerge\WinMergeU.exe P:\dev \apps\build\apache-maven-3.5.2\conf\ P:\dev\apps\build\apache- maven-3.5.4\conf\ to copy the changes in for example settings.xml and log- ging\simplelogger.properties.
• Maven Reference such as core. • Maven XSDs. • Maven Documentation, Introduction to the Build Lifecycle, Guide to Configuring Plug-ins and Available Plugins at Maven (issues) and MojoHaus (see also SoftwareEntwicklung Beratung Schulung for latest release notes or see Section 16.1.10, “Versions” with mvn versions:display-plugin-updates). • Books such as Maven: The Complete Reference (Maven Command Line Options) and Maven by Example. • For some examples of upgrading direct and transitive dependencies of a plugin see also Section 17.3.5, “Up- grade FOP” and Section 17.3.4, “Upgrade DocBook XSL stylesheets”. 4.2.2. Maven installation guide
Download the archive: apache-maven-3.5.4-bin.zip [version 3.5.4].
Extract this .zip file to P:\dev\apps\build.
27 Chapter 4.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %MVN_HOME%\bin;. Also add a New system variable MVN_HOME pointing to P:\dev\apps\build\apache-maven-3.5.4.
Note
MAVEN_HOME, M2_HOME or M3_HOME (see also issue MNG-5607) don't seem to matter since MAVEN_HOME is set in mvn.cmd anyway. In that case we are only using MVN_HOME to point to the Maven installation location and thus introduce a different one altogether.
The location of your local repository can be changed in the Maven configuration file P:\dev\apps\build \apache-maven-3.5.4\conf\settings.xml:
C:/dev/data/repo/maven/local
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and supply extra options to Maven with a New system variable MAVEN_OPTS and for example the fol- lowing value -Xmx512m -Xms128m.
Note
Since JDK 8 JEP 122: Remove the Permanent Generation from the Hotspot JVM and thus the need to tune the size of the permanent generation with for example: -XX:MaxPermSize=192m.
Tip
Should the Surefire Plugin fail to run with one of the following errors:
• “Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine.” • “RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?”
just temporarily overrule the maximum Java heap size setting with set _JAVA_OPTIONS=- Xmx256m or export _JAVA_OPTIONS=-Xmx256m, which will be notified with this message “Picked up _JAVA_OPTIONS: -Xmx256m” (see also JAVA_TOOL_OPTIONS).
Important
JDK 7 brings support for IPv6 on Windows. If you attempt to connect to an IPv4 address then under the covers it will use an IPv4-mapped IPv6 address. This might cause an org.apache.maven.lifecycle.LifecycleExecutionException with “Con- nection timed out: connect”. Then add -Djava.net.preferIPv4Stack=true also to the MAVEN_OPTS.
Verify the installation with mvn -version or mvn -v. 4.2.3. Logging
Configure Maven to include the current date and time in the output in the Maven logging configuration file P: \dev\apps\build\apache-maven-3.5.4\conf\logging\simplelogger.properties:
# MNG-6181: mvn -X also prints all debug logging from HttpClient # Be aware that the shaded packages are used # org.apache.http -> org.apache.maven.wagon.providers.http.httpclient org.slf4j.simpleLogger.log.org.apache.maven.wagon.providers.http.httpclient=off org.slf4j.simpleLogger.log.org.apache.maven.wagon.providers.http.httpclient.wire=off
Because the INFO color blue isn't readable in Windows CMD console and in Git Bash it is just ok, change it with an extra option -Dstyle.info=bold in MVN_OPTS to white (see also How to customize colors in maven 3.5 console output). 4.2.4. Repository cleanup
Instead of deleting the local repository completely remove (current) project artifacts with:
Maven 2.1.0+ now supports server password encryption.
Warning
maven-settings-decoder lets you decrypt these passwords as long as you have access to both the settings.xml file and the settings-security.xml file. To use it download the com- piled distributable or build it from source (using gradle, because of irony).
Create an encrypted master password with mvn --encrypt-master-password password and save it in the file %USERPROFILE%\.m2\settings-security.xml:
{encrypted master-password}
Now encrypt for example the nexus server password with mvn --encrypt-password password and place it in the Maven configuration file P:\dev\apps\build\apache-maven-3.5.4\conf\settings.xml:
nexusdeployment{encrypted password}
29 30 Chapter 5. Chapter 5. Documentation
5.1. DocBook
DocBook is a schema (available in several languages including RELAX NG, SGML and XML DTDs, and W3C XML Schema) maintained by the DocBook Technical Committee of OASIS . It is particularly well suited to books and papers about computer hardware and software (though it is by no means limited to these applications).
5.1.1. Resources
• DocBook V5.x release is a complete rewrite of DocBook in RELAX NG. • DocBook 5: The Definitive Guide. It is a “work in progress” and purports to document DocBook V5.0. • Writing documentation with DocBook-XML 5.0. • A DocBook 5.0 customization layer. • DocBook to LaTeX Publishing. • XMLmind XML Editor allows to edit large, complex, modular, XML documents. How to be productive with XXE and get a copy of the quick reference card. • Maven jDocBook.
5.1.2. XML Editor installation guide
XMLmind XML Editor allows to author large, complex, modular, XML documents. It makes it easy mastering XML vocabularies such as DocBook or DITA.
Download the Win32 binary: xxe-perso-5_1_1-setup-nojvm.exe or xxe-eval-6_4_0-set- up-nojvm.exe [version 6.4.0].
Run this .exe file to install XMLMind XML Editor in P:\dev\apps\editor\xmlmind.
5.1.3. XML Editor customization
Start the editor and select Options → Preferences… to edit the following options:
• Tools → Validate and check Automatically show Validity tool. • Window and check Show both tree and styled views.
Reopen your document or restart the editor for these changes to take effect.
Tip
To move the Search and Validity tools to the top next to the Edit tool, activate it and move the mouse cursor towards the tear-off line until the cursor changes into , then just right click. This way the Attributes tool is better accessible in combination with search and validity actions.
To customize the editor GUI, for example by adding the Select → Link menu items - next to the navigation buttons, edit P:\dev\apps\editor\xmlmind\doc\gui\gui\DesktopApp.xxe_gui:
To activate this customization press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add a New system variable XXE_GUI pointing to file:///P:/ dev/apps/editor/xmlmind/doc/gui/gui/DesktopApp.xxe_gui.
31 Chapter 5.
Note
The original DesktopApp.xxe_gui can be restored from P:\dev\apps\editor\xml- mind\bin\xxe.jar.
32 Chapter 6. Chapter 6. Mail Server 6.1. Thunderbird
Reclaim your inbox with Mozilla's email application Thunderbird. 6.1.1. Resources
• Download a fully localized version. • Thunderbird features many enhancements to help you better manage your unruly inbox, and stay informed. • Add-ons extend Thunderbird, letting you personalize your email client: • Lightning Calendar to organize your life — it's about time! Integrated by default with your Thunderbird. • TT DeepDark is a smooth dark theme for Thunderbird. 6.1.2. Thunderbird installation guide
Download the Win32 binary: Thunderbird Setup 52.8.0.exe [version 52.8.0].
Run this .exe file to install Thunderbird in P:\dev\apps\mail\thunderbird.
Optionally install useful extensions and pimp the email client theme for example with the the above mentioned addons. 6.1.2.1. Custom profile folder
Mozilla Thunderbird stores all your personal settings, such as your mail, passwords and extensions, in a profile folder.
To move it to a new location, all you have to do is close Thunderbird and move the default profile folder and set the absolute path for the new location in %APPDATA%\Thunderbird\profiles.ini:
The Apache Java Enterprise Mail Server (a.k.a. Apache James) is a 100% pure Java SMTP and POP3 Mail server and NNTP News server. 6.2.1. Resources
• Download the Apache James Mail Server. • Upgrade checklist: • Related products: -. • References: FishEye - Global Settings SMTP Host name, JIRA - Required settings Host Name, Nexus - Server Administration Hostname, Jenkins - System SMTP server and SonarQube - Settings SMTP host. • Integration configuration changes: Apache James - configuration Global server, Apache James - configura- tion DNS, Apache James - configuration Remote Manager, Apache James - configuration SMTP, Apache James - configuration POP3, Apache James - configuration NNTP, Apache James - configuration Trans- port processor, Apache James - configuration Data storage and Apache James - Windows Service wrapper.conf. • Introduction to the concepts behind the James implementation, as well as a guide to installing, configuring, (and for developers) building the James server. 6.2.2. Apache James installation guide
Download the archive: apache-james-2.3.2.zip [version 2.3.2].
Extract this .zip file to P:\dev\apps\mail.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add a New system variable JAMES_HOME pointing to P:\dev\apps\mail\james-2.3.2. 6.2.3. Apache James configuration
To get the configuration files extracted to P:\dev\apps\mail\james-2.3.2\apps\james\conf and P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF start the server with the following command %JAMES_HOME [33]%\bin\run.bat. Terminate the server with Ctrl+C.
33 Chapter 6.
6.2.3.1. Host configuration
In C:\WINDOWS\system32\drivers\etc\hosts file on the Mail Server computer add the following host map- ping :
# Copyright (c) 1993-2009 Microsoft Corp. # # This is a sample HOSTS file used by Microsoft TCP/IP for Windows. # # This file contains the mappings of IP addresses to host names. Each # entry should be kept on an individual line. The IP address should # be placed in the first column followed by the corresponding host name. # The IP address and the host name should be separated by at least one # space. # # Additionally, comments (such as these) may be inserted on individual # lines or following the machine name denoted by a '#' symbol.
# localhost name resolution is handled within DNS itself. # 127.0.0.1 localhost # ::1 localhost
127.0.0.1 mail.freedumbytes.dev.net
6.2.3.2. Global server configuration
Edit the following settings - in P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF \config.xml:
After the Mail Server is restarted access to the Remote Administration Tool should now only possible with telnet 127.0.0.1 4555 and the new username and password. Type shutdown to kill the current JVM (convenient when James is run as a daemon) or type quit to close the connection.
6.2.3.4.1. PuTTY configuration
Run PuTTY and configure the following custom Session:
In Session set Saved Sessions: mail.freedumbytes.dev.net remote manager and click Save and Open. Login with admin. Type quit to exit. 6.2.3.5. SMTP configuration
Edit the following settings - in P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF \config.xml:
After the Mail Server is restarted verify the connection with telnet 127.0.0.1 25. When all goes well the re- sponse will be: 220 mail.freedumbytes.dev.net SMTP Server (JAMES SMTP Server 2.3.1) ready…. Type quit to close the connection.
Tip
When telnet is Connecting To 127.0.0.1… and reports the following error “Could not open connection to the host, on port 25: Connect failed” use for example the PortQueryUI tool (see also features and functionality) to get additional information with the command portqry.exe -n 127.0.0.1 -e 25 -p TCP. In case of “Error opening socket: 10053 A Winsock error has been encountered”. A firewall or a virus scanner is probably blocking the port.
Important
In case the INFO message “James.Mailet: RemoteDelivery: Could not connect to SMTP host: …, port: 25, response: 421” and the virus scanner is Avast select Open Avast user interface → REAL-TIME SHIELDS → Mail Shield to disable Scan outbound messages.
35 Chapter 6.
6.2.3.5.1. PuTTY configuration
Run PuTTY and configure the following custom Session:
In Session set Saved Sessions: mail.freedumbytes.dev.net telnet and click Save and Open. Type quit to exit. 6.2.3.6. POP3 configuration
Disable the following setting in P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF \config.xml:
110
mail.freedumbytes.dev.net 120000
6.2.3.7. NNTP configuration
Disable the following settings and in P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF \config.xml:
119
mail.freedumbytes.dev.net 120000false
…
file://var/nntp/spool1 060000
6.2.3.8. Transport processor configuration
In case of a Microsoft Exhange Server edit the following settings - in P:\dev\apps\mail \james-2.3.2\apps\james\SAR-INF\config.xml:
…
…
IP address of Microsoft Exhange Server
36 Mail Server
25
6.2.3.9. Data storage
For the ease of future upgrades replace file://var/ with file:///P:/dev/data/mail/james/ for the fol- lowing file locations in P:\dev\apps\mail\james-2.3.2\apps\james\SAR-INF\config.xml:
# Wrapper Windows Properties wrapper.console.title=James Mail Server 2.3.2freedumbytes.dev.net Apache James
…
# Wrapper Windows NT/2000/XP Service Properties wrapper.ntservice.name=James 2.3.2james wrapper.ntservice.displayname=freedumbytes.dev.net Apache James wrapper.ntservice.description=The Java Apache Mail Enterprise Server.
Register the Apache James service with the command in an elevated command processor: %JAMES_HOME%\bin \Wrapper.exe -i %JAMES_HOME%\conf\wrapper.conf. Start Apache James with net start james.
Important
In case of a new jdk installation use an elevated command prompt just run net stop james and net start james.
6.2.4.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Apache James and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service.
37 38 Chapter 7. Chapter 7. HTTP Server 7.1. Apache HTTP Server
The Apache HTTP Server Project is an effort to develop and maintain an open-source HTTP server for modern operating systems including UNIX and Windows NT. The goal of this project is to provide a secure, efficient and extensible server that provides HTTP services in sync with the current HTTP standards. 7.1.1. Resources
• Download the Apache HTTP Server. • Apache Lounge is all about the Apache Web Server provided by the Apache Software Foundation (ASF) HTTPD Server Project. Apache Lounge has provided up-to-date Windows binaries and popular third-party modules for more than 10 years. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: Apache HTTP Server - installation guide ServerRoot / Document- Root / Directory / cgi-bin, Apache HTTP Server - access httpd-lan-access.conf, Apache HTTP Server - Realm httpd-freedumbytes-realm.conf, Apache HTTP Server - Home page Document- Root / Directory, Apache HTTP Server - Manual activation httpd-manual.conf / AliasMatch / Di- rectory / RequireAll, Apache HTTP Server - Virtual hosting Reusable config for upgrades, Apache HTTP Server - Module configuration mod_jk, Apache HTTP Server - Html rewrite configuration mod_proxy_html / mod_xml2enc / mod_headers in copy apache-conf\httpd-proxy-html.conf of conf/extra/proxy-html.conf, Apache HTTP Server - Module configuration mod_wsgi, Apache HTTP Server - Module configuration mod_perl, Apache HTTP Server - Module configuration mod_php5, Apache HTTP Server - Module configuration mod_dav / mod_dav_fs / mod_dav_lock, Apache HTTP Server - Win- dows Service httpd -k (un)install and Apache HTTP Server - Firewall Rules.
Note
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe p: \dev\apps\httpserver\apache-2.4.16\conf p:\dev\apps\httpserv- er\apache-2.4.10\conf to copy the changes in for example httpd.conf and ex- tras\httpd-manual.conf.
• Apache HTTP Server version 2.4 Documentation and Upgrading to 2.4 from 2.2. • Using Apache with Microsoft Windows. • Running a Reverse Proxy in Apache. 7.1.2. Apache HTTP Server installation guide
Download the archive: httpd-2.4.16-win64-VC14.zip [version 2.4.16].
Extract this .zip file to P:\dev\apps\httpserver and rename P:\dev\apps\httpserver\Apache24 to P:\dev\apps\httpserver\apache-2.4.16.
The VC11 build requires to have the Microsoft Visual C++ 2015 Redistributable Package (x64) [1] installed.
To use the custom installation location in P:\dev\apps\httpserver\apache-2.4.16 edit the configuration file P:\dev\apps\httpserver\apache-2.4.16\conf\httpd.conf:
ServerRoot "c:/Apache24P:/dev/apps/httpserver/apache-2.4.16" … DocumentRoot "c:/Apache24P:/dev/apps/httpserver/apache-2.4.16/htdocs" … ScriptAlias /cgi-bin/ "c:/Apache24 P:/dev/apps/httpserver/apache-2.4.16/cgi-bin/" … # # "c:/Apache24P:/dev/apps/httpserver/apache-2.4.16/cgi-bin" should be changed # to whatever your ScriptAliased # CGI directory exists, if you have that configured. #
Configure and uncomment the following general information P:\dev\apps\httpserv- er\apache-2.4.16\conf\httpd.conf:
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %APACHE_HOME%\bin;. Also add a New system variable APACHE_HOME pointing to P:\dev\apps\httpserver\apache-2.4.16 .
Verify the installation with httpd -v.
Tip
From the command line run httpd -t to run a syntax check on the changed config files.
7.1.3. Windows service
To restart automatically on Microsoft Windows listening on port 80 install the service with the command in an elevated command processor: httpd -k install -n "freedumbytes.dev.net Apache HTTP" (which will result in a service named freedumbytes.dev.netApacheHTTP) and start it with httpd -k start -n "freedumbytes.dev.net Apache HTTP".
Note
In case of an upgrade uninstall the existing service, which is referencing the prior version, with httpd -k uninstall -n "freedumbytes.dev.net Apache HTTP" first (see also Sec- tion 1.1.5, “Task Manager replacement” in case of “The specified service has been marked for deletion.”).
After the service is started verify the installation in your browser at http://localhost.
When the Apache HTTP Service is running, you can see the Apache Service Monitor icon in the system tray. Right click on this icon and select Open Apache Monitor to activate the console in which the Apache HTTP Server can be (re)started and stopped.
7.1.3.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Apache HTTP and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service. 7.1.4. Apache HTTP Server access
Require tests whether an authenticated user is authorized according to a particular authorization provider and the specified restrictions. Block Internet access based on ips in the two non default Directory elements of P:\dev\apps\httpserver\apache-2.4.16\conf\httpd.conf:
# # DocumentRoot: The directory out of which you will serve your # documents. By default, all requests are taken from this directory, but # symbolic links and aliases may be used to point to other locations. # DocumentRoot "P:/dev/apps/httpserver/apache-2.4.16/htdocs" …
# # Controls who can get stuff from this server. # Require all granted Include ../apache-conf/httpd-lan-access.conf
…
40 HTTP Server
# # "P:/dev/apps/httpserver/apache-2.4.16/cgi-bin" should be changed to whatever # your ScriptAliased CGI directory exists, if you have that configured. # AllowOverride None Options None Require all granted Include ../apache-conf/httpd-lan-access.conf
Create P:\dev\apps\httpserver\apache-conf\httpd-server-access.conf to allow access from the local area network Servers between themselves (see also JIRA Health Check in Section 12.1.4, “Apache con- figuration”):
# HTTP Server IP # Git Server IP # Fisheye Server IP # JIRA Server IP # Nexus Server IP # Tomcat Server IP # Jenkins Server IP # GlassFish Server IP # Payara Server IP # SonarQube Server IP Require ip 192.168.0.21
Create P:\dev\apps\httpserver\apache-conf\httpd-lan-access.conf to allow access from the local area network and localhost :
# Local area network partial IP range Require ip 192.168.0
Include ../apache-conf/httpd-localhost-access.conf
Create P:\dev\apps\httpserver\apache-conf\httpd-localhost-access.conf to allow access from localhost:
# Localhost Require ip 127.0.0.1 Require ip ::1
7.1.4.1. Authentication, authorization and access control
Authentication is any process by which you verify that someone is who they claim they are. Authorization is any process by which someone is allowed to be where they want to go, or to have information that they want to have. Access Control refers to any means of controlling access to any resource. Access control can be done by several different modules. The most important of these is mod_authz_host.
Create a batch file P:\dev\apps\windows\batch\http-add-user.bat:
if exist P:\dev\data\httpserver\security\http-users-freedumbytes.dev.net goto addUser
Create the first user and the password file P:\dev\data\httpserver\security\http-users- freedumbytes.dev.net with the command: http-add-user jjasper. And all consecutive users with the commands: http-add-user maven, http-add-user fisheye, http-add-user jira, http-add-user jenkins and http-add-user sonar.
Setup the user's level of access to the repository path: either r (read-only) or rw (read-write) in the file P:\dev \data\httpserver\security\http-access-freedumbytes.dev.net:
[/] jjasper = rw maven = r fisheye = r jira = r jenkins = r sonar = r
7.1.4.2. Realm
Create the realm configuration file P:\dev\apps\httpserver\apache-conf\httpd-freedum- bytes-realm.conf:
Add as an extra requirement the freedumbytes realm with basic authentication to P:\dev\apps \httpserver\apache-2.4.16\conf\httpd.conf:
# # DocumentRoot: The directory out of which you will serve your # documents. By default, all requests are taken from this directory, but # symbolic links and aliases may be used to point to other locations. # DocumentRoot "P:/dev/apps/httpserver/apache-2.4.16/htdocs" …
# # Controls who can get stuff from this server. # Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
…
# # "P:/dev/apps/httpserver/apache-2.4.16/cgi-bin" should be changed to whatever # your ScriptAliased CGI directory exists, if you have that configured. # AllowOverride None Options None
Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
42 HTTP Server
7.1.5. Home page
Make the freedumbytes.dev.net Home page content upgrade safe by changing its location and in P:\dev \apps\httpserver\apache-2.4.16\conf\httpd.conf:
# # DocumentRoot: The directory out of which you will serve your # documents. By default, all requests are taken from this directory, but # symbolic links and aliases may be used to point to other locations. # DocumentRoot "P:/dev/apps/httpserver/apache-2.4.16/htdocs P:/dev/data/httpserver/home " …
# # Controls who can get stuff from this server. # Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
Initially copy the content of P:\dev\apps\httpserver\apache-2.4.16\htdocs to P:\dev\da- ta\httpserver\home as a starting point or create your own index.html which redirects to http:// freedumbytes.dev.net/mvn-sites/maven-setup (see also Section 16.1.2, “Project information”).
After the Apache HTTP Server is restarted verify the installation in your browser at http://localhost. 7.1.6. Manual activation
To activate the local copy of the Apache HTTP Server Manual just uncomment the following line in P:\dev \apps\httpserver\apache-2.4.16\conf\httpd.conf:
# Local access to the Apache HTTP Server Manual Include conf/extra/httpd-manual.conf
Also block Internet access based on ips with basic authentication for the manual and take into account the custom installation in P:\dev\apps\httpserver\apache-2.4.16\conf\extra\httpd- manual.conf:
Make sure the directory P:\dev\logs\httpserver exists. Also create P:\dev\apps\httpserver\apache- conf\modules for storing third-party modules.
Note
It is also possible to create subdomains with VirtualHost (see also Section 7.1.9.1.1.1, “Subdo- main sample”).
In C:\WINDOWS\system32\drivers\etc\hosts file add the following mapping (on all developer machines that want to access http://freedumbytes.dev.net and subdomains on the above configured host machine - ):
# Copyright (c) 1993-2009 Microsoft Corp. # # This is a sample HOSTS file used by Microsoft TCP/IP for Windows. # # This file contains the mappings of IP addresses to host names. Each # entry should be kept on an individual line. The IP address should # be placed in the first column followed by the corresponding host name. # The IP address and the host name should be separated by at least one # space. # # Additionally, comments (such as these) may be inserted on individual # lines or following the machine name denoted by a '#' symbol. #
# localhost name resolution is handled within DNS itself. #127.0.0.1 localhost #::1 localhost
Check if your browser is using a proxy. In Firefox select Tools → Options… → Advanced → Network → Settings…. If this is the case add the following entries to No proxy for: local- host, 127.0.0.1, freedumbytes.dev.net, 192.168.0.0/16.
After the Apache HTTP Server is restarted verify the installation of the virtual host in your browser at http:// freedumbytes.dev.net. You should be able to see the freedumbytes.dev.net Home page index.html. 7.1.7.1. Firewall configuration
To allow requests from other machines in the local area network select Start Menu → Control Panel → System and Security → Windows Firewall → Advanced settings and under Windows Firewall with Advanced Security on Local Computer → Inbound Rules create a New Rule…:
• Rule Type: Port and click Next. • Protocol and Ports: TCP for Specific local ports 80 and click Next. • Action: Allow the connection and click Next. • Profile: Private only and click Next. • Name: Apache HTTP Server freedumbytes.dev.net and click Finished.
Note
It is possible that there are already two TCP and UDP rules for Apache HTTP Server when the Firewall notification has popped up. Optionally disable those.
Further restrict the Apache HTTP Server freedumbytes.dev.net rule by right clicking and set Properties → Pro- grams and Services to This Program P:\dev\apps\httpserver\apache-2.4.16\bin\httpd.exe. 7.1.8. Module mod_macro configuration
Mod_macro allows the definition and use of macros within apache runtime configuration files. It is included in Apache Lounge httpd-x.x.x.zip.
Include the Macro module configuration at the top in P:\dev\apps\httpserver\apache-conf \httpd.conf:
# Apache Macro Include ../apache-conf/httpd-macro.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-macro.conf:
LoadModule macro_module modules/mod_macro.so
For a sample use of mod_macro within a configuration file see Section 11.1.4, “Apache configuration” of the Git repositories. 7.1.9. Proxy configuration
A proxy server sits between a client application, such as a Web browser, and a real server. It intercepts all requests to the real server to see if it can fulfill the requests itself. If not, it forwards the request to the real server.
In the P:\dev\apps\httpserver\apache-conf\httpd-vhosts.conf file before the VirtualHost settings from Section 7.1.7, “Virtual hosting configuration” insert the following two modules , and at the end of the VirtualHost settings add the following proxy configuration - :
Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
The actual proxied servers includes will be supplied in the following chapters (for an example see Section 7.1.9.1.1, “Module mod_proxy_ajp configuration”). 7.1.9.1. Apache JServ Protocol
Apache JServ Protocol version 1.3 (hereafter ajp13) is packet-oriented. A binary format was presumably chosen over the more readable plain text for reasons of performance. The web server communicates with the servlet container over TCP connections. To cut down on the expensive process of socket creation, the web server will attempt to maintain persistent TCP connections to the servlet container, and to reuse a connection for multiple request/response cycles.
Once a connection is assigned to a particular request, it will not be used for any others until the request-handling cycle has terminated. In other words, requests are not multiplexed over connections. This makes for much simpler code at either end of the connection, although it does cause more connections to be open at once.
If AJP is to be used, the then mod_proxy_ajp module is preferred over mod_jk.
7.1.9.1.1. Module mod_proxy_ajp configuration
Mod_proxy_ajp is the AJP support module for mod_proxy.
In the P:\dev\apps\httpserver\apache-conf\httpd-vhosts.conf file before the VirtualHost settings from Section 7.1.7, “Virtual hosting configuration” append the following module :
And then for example instruct Apache to proxy all URLs whose path portions begin with /http-sample/ and /ajp-sample/ in subdomain sample.freedumbytes.dev.net using the following P:\dev\apps\httpserv- er\apache-conf\httpd-vhosts.conf include :
Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
Prepackaged sample applications http-sample.war and ajp-sample.war can be downloaded for deployment in the installed Tomcat.
After the Apache HTTP Server is restarted verify the installation of the sample proxies in your browser at http://sample.freedumbytes.dev.net/http-sample, http://freedumbytes.dev.net/sample/http-sample and http:// sample.freedumbytes.dev.net/ajp-sample.
Important
For this to work all URL references need to be relative (see also Section 7.1.9.2, “Html rewrite configuration”).
7.1.9.1.2. Module mod_jk configuration
Mod_jk is the Apache Tomcat Connector based on the ajp1.3.
Download the Apache Tomcat Connector or Apache Lounge modules archive: mod_jk-1.2.41-win64- VC14.zip [version 1.2.41].
Extract the module to mod_jk.so from the downloaded archive and place it in P:\dev\apps\httpserv- er\apache-conf\modules.
Include the Apache Tomcat Connector module configuration just before httpd-vhosts.conf in P:\dev \apps\httpserver\apache-conf\httpd.conf:
# Apache Tomcat Connector Include ../apache-conf/httpd-jk.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-jk.conf:
Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
A prepackaged sample application jk-sample.war can be downloaded for deployment in the installed Tomcat.
After the Apache HTTP Server is restarted verify the installation of the sample proxy in your browser at http:// sample.freedumbytes.dev.net/jk-sample.
Important
For this to work all URL references need to be relative (see also Section 7.1.9.2, “Html rewrite configuration”).
7.1.9.2. Html rewrite configuration
For some proxies it might be necessary to additionally configure the mod_proxy_html output filter to rewrite HTML links to ensure that links work for users outside the proxy. It is included in Apache Lounge httpd-x.x.x.zip.
To make the upcoming changes upgrade safe copy and rename the initial P:\dev\apps \httpserver\apache-2.4.16\conf\extra\proxy-html.conf to P:\dev\apps\httpserver\apache- conf\httpd-proxy-html.conf. Thus when upgrading just use WinMerge to look for changes between the proxy-html.conf files in both unzipped httpd-x.x.x.zip versions with fc p: \dev\apps\httpserver\apache-2.4.10\conf\extra\proxy-html.conf p:\dev\apps\httpserv-
48 HTTP Server
er\apache-2.4.16\conf\extra\proxy-html.conf. Apply possible changes in P:\dev\apps\httpserv- er\apache-conf\httpd-proxy-html.conf.
In the initial copy P:\dev\apps\httpserver\apache-conf\httpd-proxy-html.conf load the modules - :
# Configuration example. … # For Windows (I don't know if there's a standard path for the libraries) # LoadFile C:/path/zlib.dll # or LoadFile "P:/dev/apps/httpserver/apache-2.4.16/bin/zlib1.dll" # LoadFile C:/path/iconv.dll # or LoadFile "P:/dev/apps/httpserver/apache-2.4.16/bin/libapriconv-1.dll" # LoadFile C:/path/libxml2.dll # or LoadFile "P:/dev/apps/httpserver/apache-2.4.16/bin/libxml2.dll" LoadModule proxy_html_module modules/mod_proxy_html.so LoadModule xml2enc_module modules/mod_xml2enc.so
…
# To support scripting events (with ProxyHTMLExtended On), # you'll need to declare them too.
And include the html rewriting configuration just before httpd-vhosts.conf in P:\dev\apps\httpserv- er\apache-conf\httpd.conf:
# Html rewriting Include ../apache-conf/httpd-proxy-html.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf
Finally remove the now superfluous include in P:\dev\apps\httpserver\apache-2.4.16\conf \httpd.conf:
# Configure mod_proxy_html to understand HTML4/XHTML1 # #Include conf/extra/proxy-html.conf #
7.1.9.2.1. Sample
For sample usage take a look at Section 8.1.5, “Apache configuration”.
Important
Notice the difference between http://sample.freedumbytes.dev.net/http-sample and http:// freedumbytes.dev.net/tomcat/http-sample/ for the absolute URL reference due to Proxy- HTMLURLMap http://localhost:8068 /tomcat.
49 Chapter 7.
7.1.10. Module mod_wsgi configuration
Mod_wsgi aims to implement a simple to use Apache module which can host any Python application which sup- ports the Python WSGI interface. The module would be suitable for use in hosting high performance production web sites, as well as your average self managed personal sites running on web hosting services.
Download the archive mod_wsgi-windows-4.4.12.tar.gz. After extracting this archive move mod_wsgi-windows-4.4.12\Apache24-win64-VC10\modules\mod_wsgi-py34-VC10.so to P:\dev \apps\httpserver\apache-conf\modules\mod_wsgi.so [version 4.4.12].
Or download the alternative archive mod_wsgi-4.4.13+ap24vc10-cp34-none-win_amd64.whl (just rename to .zip). After extracting this archive move mod_wsgi-4.4.13+ap24vc10-cp34- none-win_amd64\mod_wsgi-4.4.13+ap24vc10.data\data\mod_wsgi.so to P:\dev\apps\httpserv- er\apache-conf\modules\mod_wsgi.so [version 4.4.13].
Include this Python module just before httpd-vhosts.conf in P:\dev\apps\httpserver\apache- conf\httpd.conf:
# Python Include ../apache-conf/httpd-python.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-python.conf:
# Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-localhost-access.conf
# Python Sample Include ../apache-conf/httpd-python-sample.conf
Create the following file P:\dev\apps\httpserver\sample\python\status.wsgi:
import pprint
def application(environ, start_response): """ Display the contents of the environ dictionary.""" # produce some content output = b'Hello World\r\r\r' + pprint.pformat(environ).encode('utf8')
# send first header and status status = '200 OK' headers = [('Content-type', 'text/plain'),('Content-Length', str(len(output)))] start_response(status, headers)
return [output]
Map this WSGI script in P:\dev\apps\httpserver\apache-conf\httpd-python-sample.conf:
After the Apache HTTP Server is restarted test the mod_wsgi by taking a look at the Python Status. 7.1.11. Module mod_perl configuration
Mod_perl brings together the full power of the Perl programming language and the Apache HTTP server. You can use Perl to manage Apache, respond to requests for web pages and much more. Mod_perl is more than CGI scripting on steroids. It is a whole new way to create dynamic content by utilizing the full power of the Apache web server to create stateful sessions, customized user authentication systems, smart proxies and much more. Yet, magically, your old CGI scripts will continue to work and work very fast indeed. With mod_perl you give up nothing and gain so much!
50 HTTP Server
Important
Alas haven't found a x64 Apache HTTP Server 2.4.x Perl 5.18 binary yet. Thus mod_perl is no longer available after current upgrades.
When ppm install mod_perl is run it will also fetch the Apache2 module mod_perl.so. When asked Where should mod_perl.so be placed? just answer P:\dev\apps\httpserver\apache-conf\modules [ver- sion 2.000004].
Note
In case of an upgrade just ppm uninstall mod_perl first.
Include this Perl module just before httpd-vhosts.conf in P:\dev\apps\httpserver\apache-conf \httpd.conf:
# Perl Include ../apache-conf/httpd-perl.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf with the following content in P:\dev\apps\httpserver\apache-conf\httpd-perl.conf:
SetHandler perl-script PerlHandler Apache2::Status PerlSetVar StatusOptionsAll On
Include ../apache-conf/httpd-localhost-access.conf
# Perl Sample Include ../apache-conf/httpd-perl-sample.conf
It is possible to verify the installation using the instructions of the mod_perl Manual. Or just create the following file P:\dev\apps\httpserver\sample\perl\test.pm:
package sample::perl::test; use strict;
use Apache2::RequestRec (); # for $r->content_type use Apache2::RequestIO (); # for $r->puts use Apache2::Const -compile => ':common';
sub handler { my $r = shift; my $time = scalar localtime(); my $package = __PACKAGE__; $r->content_type('text/html'); $r->puts(<<"END");
Test
Hello from $package! The time is $time. END
return Apache2::Const::OK; }
1;
51 Chapter 7.
Register this perl handler in P:\dev\apps\httpserver\apache-conf\httpd-perl-sample.conf:
Include ../apache-conf/httpd-localhost-access.conf
After the Apache HTTP Server is restarted test the mod_perl module at http://freedumbytes.dev.net/perl-sample and take a look at the Perl Status. In case of memory problems read up on Apache2::SizeLimit - Because size does matter. 7.1.12. Module mod_php5 configuration
php5apache2_4.dll is needed to allow Apache to use PHP.
Include this PHP module just before httpd-vhosts.conf in P:\dev\apps\httpserver\apache-conf \httpd.conf:
# Python Include ../apache-conf/httpd-php.conf
# Virtual hosts Include ../apache-conf/httpd-vhosts.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-php.conf:
Include ../apache-conf/httpd-localhost-access.conf
Include ../apache-conf/httpd-localhost-access.conf
Create the following file P:\dev\data\httpserver\home\status.php and/or P:\dev\data\httpserv- er\home\status.php5:
Edit t- in the file P:\dev\apps\prg\php-5.6.13\php.ini:
; The root of the PHP pages, used only if nonempty. ; if PHP was not compiled with FORCE_REDIRECT, you SHOULD set doc_root ; if you are running php as a CGI under any web server (other than IIS) ; see documentation for security issues. The alternate is to use the ; cgi.force_redirect configuration below ; http://php.net/doc-root doc_root = P:\dev\data\httpserver\home
…
; Directory in which the loadable extensions (modules) reside. ; http://php.net/extension-dir ; extension_dir = "./"
52 HTTP Server
; On windows: ; extension_dir = "ext" extension_dir = P:\dev\apps\prg\php-5.6.13\ext
After the Apache HTTP Server is restarted test the mod_php5 by taking a look at the PHP Status. 7.1.13. Module mod_dav configuration
Mod_dav provides class 1 and class 2 WebDAV (“Web-based Distributed Authoring and Versioning”) function- ality for Apache. This extension to the HTTP protocol allows creating, moving, copying, and deleting resources and collections on a remote web server.
Enable the following modules - in P:\dev\apps\httpserver\apache-2.4.16\conf\httpd.conf:
For example include the Maven Sites WebDAV configuration just before httpd-vhosts.conf in P:\dev \apps\httpserver\apache-conf\httpd.conf:
# Maven Sites Include ../apache-conf/httpd-mvn.conf
# Virtual hosts Include conf/extra/httpd-vhosts.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-mvn.conf:
DavLockDB "P:/dev/data/httpserver/webdav/DavLock"
Alias /mvn-sites "P:/dev/data/httpserver/mvn/sites"
Dav On DavMinTimeout 120
Options Indexes
Include ../apache-conf/httpd-lan-access.conf Include ../apache-conf/httpd-freedumbytes-realm.conf
Don't forget to create the specified directories: P:\dev\data\httpserver\webdav and P:\dev\da- ta\httpserver\mvn\sites. After the Apache HTTP Server is restarted take a look at http:// freedumbytes.dev.net/mvn-sites/. 7.1.14. Server info
The mod_info module provides a comprehensive overview of the server configuration including all installed mod- ules and directives in the configuration files. And the mod_status module allows a server administrator to find out how well their server is performing.
To make the upcoming changes upgrade safe initially copy the P:\dev\apps\httpserv- er\apache-2.4.16\conf\extra\httpd-info.conf to P:\dev\apps\httpserver\apache-conf \httpd-info.conf. Thus when upgrading this functionality doesn't have to be reconfigured again.
Include the real-time info configuration at the top in P:\dev\apps\httpserver\apache-conf \httpd.conf:
# Real-time info on requests and configuration
53 Chapter 7.
Include ../apache-conf/httpd-info.conf
# Apache Macro Include ../apache-conf/httpd-macro.conf
In the copy P:\dev\apps\httpserver\apache-conf\httpd-info.conf load the modules - and re- strict access to localhost:
# # Allow server status reports generated by mod_status, # with the URL of http://servername/server-status # Change the ".example.com" to match your domain to enable.
SetHandler server-status Include ../apache-conf/httpd-localhost-access.conf
# # ExtendedStatus controls whether Apache will generate "full" status # information (ExtendedStatus On) or just basic information (ExtendedStatus # Off) when the "server-status" handler is called. The default is Off. # #ExtendedStatus On
# # Allow remote server configuration reports, with the URL of # http://servername/server-info (requires that mod_info.c be loaded). # Change the ".example.com" to match your domain to enable. # SetHandler server-info Include ../apache-conf/httpd-localhost-access.conf
Note
Keep the full status settings disabled, performance wise. After the Apache HTTP Server is restarted take a look at the Apache Server Status and the Apache Server Information.
54 Chapter 8. Chapter 8. Web Server 8.1. Tomcat
Apache Tomcat is an implementation of the Java Servlet and JavaServer Pages technologies. 8.1.1. Resources
• Download Tomcat. • Upgrade checklist: • Related products: -. • References: Tomcat - Manager tomcat8w.exe and Jenkins - installation guide webapps. • Integration configuration changes: Tomcat - HTTP Binding address, Tomcat - access tomcat-users.xml, Tomcat - Window Service tomcat8.exe //US//tomcat, Tomcat - Account Log On, Tomcat - Logging juli / log4j / slf4j and Tomcat - Remote authentication tomcatAuthentication.
Note
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe p:\dev \apps\webserver\apache-tomcat-8.0.26\conf p:\dev\apps\webserv- er\apache-tomcat-8.0.17\conf to copy the changes in for example server.xml and tomcat-users.xml.
• Tomcat Documentation such as Migration 8.0.x, HTTP and AJP Connectors. 8.1.2. Tomcat installation guide
Download the archive: apache-tomcat-8.0.26-windows-x64.zip [version 8.0.26].
Extract this .zip file to P:\dev\apps\webserver. 8.1.2.1. HTTP Binding
Configure Tomcat settings - in the file P:\dev\apps\webserver\apache-tomcat-8.0.26\conf \server.xml:
… …
8.1.2.2. Tomcat access
Configure Tomcat access in the file P:\dev\apps\webserver\apache-tomcat-8.0.26\conf\tom- cat-users.xml as follows:
55 Chapter 8.
These roles are used by the following two applications respectively:
1. Tomcat Web Application Manager: • HTML Manager interface (for humans) at http://localhost:8068/manager/html. • Text Manager interface (for scripts) at http://localhost:8068/manager/text. • JMX Proxy interface at http://localhost:8068/manager/jmxproxy.
Status interface for all four users at http://localhost:8068/manager/status 2. Tomcat Virtual Host Manager: • HTMLHostManager commands at http://localhost:8068/host-manager/html. • HostManager commands at http://localhost:8068/host-manager/text. 8.1.3. Windows service
To restart automatically on Microsoft Windows create a service using an elevated command prompt as follows:
cd /d P:\dev\apps\webserver\apache-tomcat-8.0.26
set JAVA_HOME=P:\dev\apps\prg\java-x64\jdk1.8.0_162 set CATALINA_HOME=P:\dev\apps\webserver\apache-tomcat-8.0.26 bin\service.bat install tomcat
bin\tomcat8.exe //US//tomcat --DisplayName="freedumbytes.dev.net Tomcat" ^ --Description="Tomcat is an open source web server." ^ --Startup=auto
In case of an upgrade sc delete tomcat first (see also Section 1.1.5, “Task Manager replace- ment” in case of “The specified service has been marked for deletion.”).
Important
In case of a new jdk installation just run P:\dev\apps\webserver\apache- tomcat-8.0.26\bin\tomcat8.exe //US//tomcat --Jvm auto (when JAVA_HOME is still pointing to the 64-bit version; see also Section 8.1.3.2, “Tomcat Manager”).
Verify the installation and the JVM memory usage at http://localhost:8068/manager/status. 8.1.3.1. Log On Windows user
For easier management of user specific files such as for example the Maven C:\Users\tom- cat\.m2\settings-security.xml select Start Menu → Control Panel → User Accounts and Family Safety → User Accounts → Add or remove user accounts → Create a new account Tomcat as Standard user and click Create Account. Select this new user and Create a password. Also Change the picture with Browse for more pic- tures… P:\dev\apps\webserver\apache-tomcat-8.0.26\webapps\ROOT\tomcat.png and click Open.
Launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Tomcat and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service. 8.1.3.1.1. netrc
The _netrc file contains login and initialization information used by the auto-login process.
When logged in as tomcat user type the following commands:
This way the Jenkins build jobs will be able to auto-login to the git-repo on freedumbytes.dev.net.
56 Web Server
Important
The usage of Section 14.1.4.5, “Git authentication” is a better alternative.
Restart tomcat service to pick up the HOME setting.
8.1.3.2. Tomcat Manager
Tomcat8w is a GUI application for monitoring and configuring Tomcat services.
To monitor the tomcat service run: P:\dev\apps\webserver\apache-tomcat-8.0.26\bin \tomcat8w.exe //MS//tomcat or create a shortcut Tomcat Manager for it and edit Properties → Target to include the correct service name P:\dev\apps\webserver\apache-tomcat-8.0.26\bin \tomcat8w.exe //MS//tomcat.
When the Tomcat Manager is running, you can see the freedumbytes.dev.net Tomcat icon in the system tray. Right click on this icon and select Configure to activate the console in which the Apache Tomcat Server can be (re)started and stopped.
Note
When the monitor application was already running when the Log On account was changed (as described in the prior Section 8.1.3.1, “Log On Windows user”) right click on its icon and select Exit and then restart it. Because this change wont show up otherwise.
Tip
As an alternative to change the additional settings of the Tomcat service run: P:\dev\apps \webserver\apache-tomcat-8.0.26\bin\tomcat8w.exe //MS//tomcat such as Java → Java Virtual Machine P:\dev\apps\prg\java-x64\jdk1.8.0_162\jre\bin\serv- er\jvm.dll.
8.1.4. Logging
Tomcat 7.0 uses Commons Logging throughout its internal code allowing the developer to choose a logging con- figuration that suits their needs such as Log4J.
To switch to Log4J apply the following steps after shutting down tomcat with net stop tomcat:
1. Tomcat replacements: • Download: tomcat-juli-adapters.jar and tomcat-juli.jar [version 8.0.26]. • Replace the tomcat-juli.jar in P:\dev\apps\webserver\apache-tomcat-8.0.26\bin. • Install the tomcat-juli-adapters.jar in the P:\dev\apps\webserver\apache- tomcat-8.0.26\lib directory. 2. Log4J integration: • Download: apache-log4j-1.2.17.zip [version 1.2.17]. • Install the log4j-1.2.17.jar in the P:\dev\apps\webserver\apache-tomcat-8.0.26\lib direc- tory. 3. SLF4J integration: • Download: slf4j-1.7.12.zip used by frameworks such as Hibernate [version 1.7.12]. • Install the slf4j-api-1.7.12.jar in the P:\dev\apps\webserver\apache-tomcat-8.0.26\lib directory. • And bind SLF4J to Log4J by installing the slf4j-log4j12-1.7.12.jar in the P:\dev\apps\web- server\apache-tomcat-8.0.26\lib directory. 4. Logging configuration: • Create a file called log4j.xml in P:\dev\logs\webserver\tomcat with the following content:
57 Chapter 8.
• Delete the obsolete P:\dev\apps\webserver\apache-tomcat-8.0.26\conf \logging.properties or rename to logging.properties.obsolete. • Configure the tomcat service to use the log4j.xml instead as follows:
Open the Tomcat Manager in the system tray by right clicking on this icon and selecting Configure. In the Java tab remove the now obsolete Java Options parameter -Djava.util.logging.config.file=P: \dev\apps\webserver\apache-tomcat-8.0.26\conf\logging.properties. 8.1.5. Apache configuration
Instruct Apache to proxy all URLs whose path portions begin with /tomcat/ using the following P:\dev\apps \httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# Tomcat Include ../apache-conf/httpd-tomcat.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-tomcat.conf (see also Sec- tion 7.1.9.2, “Html rewrite configuration”):
Notice that only local access is activated and not the freedumbytes.dev.net realm because this clashes with the basic authentication of Tomcat itself resulting in the fol- lowing error “AH01618: user admin not found: /tomcat/manager/status, referer: http:// freedumbytes.dev.net/tomcat/” and hanging HTTP requests.
8.1.5.1. Remote authentication
For the AJP protocol connector tell Tomcat not to use its internal (primitive) authentication mechanism, but in- stead to use remote authentication provided by the front-end web server with the following setting in the file P:\dev\apps\webserver\apache-tomcat-8.0.26\conf\server.xml:
This can be useful for passing on authenticated users to for example applications such as Jenkins.
When the Tomcat web server and the Apache HTTP Server are restarted you should be able to browse Tomcat at http://freedumbytes.dev.net/tomcat/.
59 60 Chapter 9. Chapter 9. Application Server
9.1. Payara
Payara Server is a drop in replacement for GlassFish Server Open Source Edition, with the peace of mind of quarterly releases containing enhancements, bug fixes and patches (see also the Data Sheet). Payara Server is derived from GlassFish Server Open Source Edition and 100% open source at GitHub.
There is also Payara Micro, which enables you to run war files from the command line without any application server installation. It is small, < 70MB in size and incredibly simple to use. With its automatic and elastic clustering, Payara Micro is designed for running Java EE applications in a modern containerized/virtualized infrastructure, using automated provisioning tools like Chef, Ansible or Puppet (see also the Data Sheet).
Just run p:\dev\apps\editor\winmerge\WinMergeU.exe p:\dev\apps \appserver\payara-4.1.1.163\glassfish\domains\payaradomain\config p:\dev\apps\appserver\payara-4.1.1.152\glassfish \domains\payaradomain\config to copy the changes in for example domain.xml,
Note
And run p:\dev\apps\editor\winmerge\WinMergeU.exe p:\dev\apps \appserver\payara-4.1.1.163\glassfish\config p:\dev\apps\appserv- er\payara-4.1.1.152\glassfish\config for asenv.bat.
• GlassFish vs Payara Server - Comparison. • Payara Server Documentation such as Basics and How-to from the Blog & News Archive or GitBook. • Getting Started with Payara Server (video tutorial). • Payara Server in Production. • Automating Production in Payara Server. • Troubleshooting your Java EE Applications. • New features for Ops Teams in Payara Server. • 6 Production Features of Payara Server (& GlassFish) You May Not Know About. • Advanced Administration & Maintenance of Payara Server. • Monitoring Payara Server in Production. • An introduction to connection pools in Payara Server. • GlassFish performance tuning tips from the field. • Introducing Payara Micro - A New Way of Running Java EE Applications. • How Micro is Payara Micro? • Using Payara Embedded & Payara Micro with Maven. • Building RESTful Java EE Microservices with Payara Server Embedded. • Testing applications with Payara Server & Arquillian. • Latest versions Payara and WildFly improve Java EE 7 authentication compliance. • Payara's Enhanced GlassFish Server Gets Hazelcast Enterprise Add-on. • Ondrej Mihályi @OMihalyi #JavaEE Trainer, consultant and evangelist. Service engineer at @Payara_Fish. De- ploying Multiple Application Versions on Payara Server. • GlassFish Became The Killer AppServer And Then Changed The Name.
9.1.2. Payara installation guide
Download the archive: payara-4.1.1.163.zip [version 4.1.1.163].
Extract the .zip file to P:\dev\apps\appserver and rename P:\dev\apps\appserver\payara41 to P: \dev\apps\appserver\payara-4.1.1.163. 61 Chapter 9.
Important
New since release 152 is the extra payaradomain template.
Change the listener addresses to localhost (to restrict access), host IP (to allow local area net- work access; see also 192.168.0.xx host [44]) and optionally disable the ssl connections by editing the configuration file p:\dev\apps\appserver\payara-4.1.1.163\glassfish\domains\payarado- main\config\domain.xml:
… … … … … … … … …
… … … … … … …
62 Application Server
… …
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %PAYARA_HOME%\bin;. Also add a New system variable PAYARA_HOME pointing to P:\dev\apps\appserver\payara-4.1.1.163.
For administration tasks, the Payara server software provides the following tools, which enable administrators to manage server instances:
• The Admin Console, a browser-based graphical user interface (GUI). • The asadmin utility, a command-line tool.
Start the server by entering the command asadmin start-domain payaradomain.
Show domains running on the server by entering the command asadmin list-domains.
Verify that the server is running in your browser at http://localhost:8071.
Stop the server by entering the command asadmin stop-domain payaradomain. 9.1.3. Changing JDK version
Updating Payara to run with a JDK version that is different to the one used during installation will affect all local domains. To do this, update the following in the file P:\dev\apps\appserv- er\payara-4.1.1.163\glassfish\config\asenv.bat:
REM Yet, this file is also where users of earlier versions have sometimes added REM a definition of AS_JAVA to control which version of Java Payara should use. REM As a result, in order to run a user-specified version of Java, the asadmin REM and appclient scripts do indeed invoke this file as a script - but ONLY to REM define AS_JAVA. Any calling script should not rely on the other settings REM because the relative paths will be resolved against the current directory REM when the calling script is run, not the installation directory of Payara, REM and such resolution will not work correctly unless the script happens to be REM run from the Payara installation directory. … set AS_JAVA=P:\dev\apps\prg\java-x64\jdk1.8.0_162
Verify the installation and the JVM memory usage at http://freedumbytes.dev.net:4848/common/appServ- er/jvmReport.jsf?reportType=summary&instanceName=server which is hidden at Home → server (Admin Serv- er) → General → JVM → JVM Report. 9.1.4. Autodeploy
A prepackaged application that says “Hello” can be downloaded from http://glassfish.java.net/downloads/quick- start/hello.war.
Just copy the application, which is packaged as Java ARchive (JAR), Web ARchive (WAR), or Enterprise ARchive (EAR) file, into p:\dev\apps\appserver\payara-4.1.1.163\glassfish\domains\payaradomain\au- todeploy\ or run for example: asadmin -H freedumbytes.dev.net deploy --force p:\dev\tools \appserver\hello.war (use --force for redeployment). 9.1.5. Admin console
Access http://freedumbytes.dev.net:4848. Enter the admin user name and password. Change/Set the admin pass- word at Domain → Administrator Password.
In the left pane, click the Applications node. If you already have the hello.war application deployed, undeploy it by selecting the checkbox next to it and clicking Undeploy. To deploy a newly assembled application click De- ploy….
To verify that it was deployed properly, click Launch.
63 Chapter 9.
9.1.6. Apache configuration
Instruct Apache to proxy all URLs whose path portions begin with /payara/ using the following P:\dev\apps \httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# Payara Include ../apache-conf/httpd-payara.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-payara.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf
When the domain and the Apache HTTP Server are restarted you should be able to browse Payara at http:// freedumbytes.dev.net/payara. 9.1.7. Windows service
To restart automatically on Microsoft Windows create a service using an elevated command prompt as follows:
cd /d P:\dev\apps\appserver\payara-4.1.1.163
set JAVA_HOME=P:\dev\apps\prg\java-x64\jdk1.8.0_162 set PAYARA_HOME=P:\dev\apps\appserver\payara-4.1.1.163 asadmin create-service payaradomain
dir payaradomainService.xml /s
Note
In case of an upgrade sc delete payara first (see also Section 1.1.5, “Task Manager replace- ment” in case of “The specified service has been marked for deletion.”).
Edit the following entries - in the newly generated service configuration file p:\dev\apps\appserv- er\payara-4.1.1.163\glassfish\domains\payaradomain\bin\payaradomainService.xml:
64 Application Server
payaradomainpayara payaradomain Payara Serverfreedumbytes.dev.net Payara Payara ServerPayara Application Server P:/dev/apps/appserver/payara-4.1.1.163/glassfish/lib/nadmin.bat P:\\dev\\apps\\appserver\\payara-4.1.1.163\\glassfish\\domains/payaradomain/ bin resettcpipstart-domain--watchdog--domaindir P:\\dev\\apps\\appserver\\payara-4.1.1.163\\glassfish\\domains payaradomainstop-domain--domaindir P:\\dev\\apps\\appserver\\payara-4.1.1.163\\glassfish\\domains payaradomain
The payaradomainService.exe install must be run in its directory glassfish\do- mains\payaradomain\bin\ otherwise the following error “System error 2 has occurred. The system cannot find the file specified.” will occur during net start payara.
Start it with net start payara.
Verify the installation at http://freedumbytes.dev.net/payara. 9.1.7.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Payara and select Properties → Log On to use This ac- count .\Tomcat. After password conformation click Apply and restart the service.
65 66 Chapter 10. Chapter 10. Database 10.1. MySQL
MySQL is the world's most popular open source database. 10.1.1. Resources
• Download and Upgrading MySQL (from previous series). • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: MySQL - Upgrading backup xcopy / sc delete mysqlmaster / unzip lastest / MYSQL_HOME / basedir / mysql-create-service master / mysql_upgrade / mysql_upgrade_info. • MySQL Reference Manual and Documentation. • Download MySQL Workbench provides DBAs and developers an integrated tools environment for: Database Design & Modeling, SQL Development (replacing MySQL Query Browser) and Database Administration (replac- ing MySQL Administrator). • Download the official JDBC driver Connector/J for MySQL. • Download MySQL Utilities a collection of command-line utilities that are used for maintaining and administer- ing MySQL servers. • Download MySQL Router a lightweight middleware that provides transparent routing between your applica- tion and any backend MySQL Servers. It can be used for a wide variety of use cases, such as providing high availability and scalability by effectively routing database traffic to appropriate backend MySQL Servers. • Download MySQL Shell an interactive Javascript, Python, or SQL interface supporting development and ad- ministration for the MySQL Server and is a component of the MySQL Server. You can use the MySQL Shell to perform data queries and updates as well as various administration operations. • Download MySQL Notifier a small utility that can keep watch over your Windows and MySQL services, notifying you of changes in their operation. • Time zone description tables. • MySQL Benchmark Tool. • MySQL Performance Blog: Recovery beyond data restore. • Xaprb Blog: How to make replication reliable, How replication got out of sync, How to check replication integrity continually and Oracle is not screwing MySQL. • How to support full Unicode in MySQL databases. 10.1.2. MySQL installation guide
Download the archive: mysql-5.7.20-winx64.zip [version 5.7.20].
Extract this .zip file to P:\dev\apps\db.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %MYSQL_HOME%\bin;. Also add a New system variable MYSQL_HOME pointing to P:\dev\apps\db\mysql-5.7.20-winx64.
Verify the installation with mysql --version.
When upgrading now return to Section 10.1.5, “Upgrading”.
10.1.2.1. Custom data directory
Initialize a custom MySQL data directory with a copy of the installation data folder P:\dev\apps \db\mysql-5.7.20-winx64\data from the command line with xcopy /E /I P:\dev\apps\db \mysql-5.7.20-winx64\data K:\dev\data\db\mysql\data.
Create a MySQL Server configuration file K:\dev\data\db\mysql\conf\mysql-master.ini:
To avoid the error “InnoDB: Error: unable to create temporary file; errno: 2” create the tmpdir with mkdir K:\dev\data\db\mysql\tmp.
Note
Set max_allowed_packet in case of “com.mysql.jdbc.PacketTooBigException: Packet for query is too large.” issues.
From the command line start MySQL Server with mysqld --defaults-file=K:\dev\data\db\mysql\conf \mysql-master.ini.
Verify the installation with mysqlshow mysql -u root and/or mysqladmin version status proc -u root. To see a list of options provided by mysql, invoke it with the --help option.
To shutdown the SQL Server use mysqladmin -u root --port=3306 shutdown. 10.1.2.2. MySQL character set configuration
Change the default character set and collation in K:\dev\data\db\mysql\conf\mysql-master.ini:
Verify those settings before and after restarting service mysql-1 with:
mysql -u root mysql> show variables where variable_name -> like 'character\_set\_%' or variable_name LIKE 'collation%'; mysql> quit
10.1.2.3. MySQL root password
If you have never set a root password for MySQL server, the server does not require a password at all for connect- ing as root. To setup root password for first time, use mysqladmin command at shell prompt as follows and enter and confirm the new password:
mysqladmin -u root password
However, if you want to change a root password, then you need to use the following command and enter the old password followed by the new password:
mysqladmin -u root -p password
10.1.3. Windows service
With the following batch file P:\dev\apps\windows\batch\mysql-create-service.bat:
@echo off
if not "%1"=="" goto createService
68 Database
echo Usage: mysql-create-service ^ goto end
:createService setlocal
set SERVICE_ID=mysql%1 set SERVICE_DISPLAY_NAME="freedumbytes.dev.net MySQL %1" set SERVICE_EXE=%MYSQL_HOME%\bin\mysqld.exe set SERVICE_START_CMD=--defaults-file=\"K:\dev\data\db\mysql\conf\mysql-%1.ini\" set SERVICE_STOP_CMD=
sc description %SERVICE_ID% "MySQL is a SQL database management system."
endlocal
:end
Define the service for the master from an elevated command prompt with: mysql-create-service master. Start the server with sc start mysqlmaster or net start mysqlmaster. 10.1.3.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net MySQL master and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service. 10.1.4. Backup options
1. The mysqldump client is a backup program. It can be used to dump a database or a collection of databases for backup or transfer to another SQL server (not necessarily a MySQL server).
Create a batch file P:\dev\apps\windows\batch\mysql-backup-all.bat:
@echo off
if not "%1"=="" goto backupDatabases
echo Usage: mysql-backup-all ^ goto end
:backupDatabases if not exist S:\dev\backup\db\mysqldump mkdir S:\dev\backup\db\mysqldump cd /d S:\dev\backup\db\mysqldump mysqldump --single-transaction --routines --triggers -q --all-databases ^ -u root -p > S:\dev\backup\db\mysqldump\all.%1.sql
:end
To dump all databases use the following statement: mysql-backup-all 20091205.
Create a batch file P:\dev\apps\windows\batch\mysql-backup-db.bat:
@echo off
if not "%2"=="" goto backupDatabase
echo Usage: mysql-backup-db ^ ^ goto end
:backupDatabase if not exist S:\dev\backup\db\mysqldump mkdir S:\dev\backup\db\mysqldump cd /d S:\dev\backup\db\mysqldump mysqldump --single-transaction --routines --triggers -q %1 ^ -u root -p > S:\dev\backup\db\mysqldump\%1.%2.sql
:end
69 Chapter 10.
To dump a specific database (for example mysql) use the following statement: mysql-backup-db mysql 20091205. And import will look like mysql -u root -p mysql < S:\dev\backup\db\mysql- dump\mysql.20091205.sql.
2. You can also create a binary backup by simply archiving the complete K:\dev\data\db\mysql directory, when the MySQL service is shut down. 10.1.5. Upgrading
Backup the mysql database separately, for example:
mysql-backup-db mysql 20150917 mysql-backup-db fisheye 20150917 mysql-backup-db jira 20150917 mysql-backup-db sonarqube 20150917 mysql-backup-db test 20150917
Or stop the master service (after stopping services that might rely on a database such as payara, tomcat, fisheye, jira and sonarqube):
net stop mysqlmaster
And create a binary backup of the MySQL configuration and the data directory:
After extracting the latest zip as described in installation guide and updating MYSQL_HOME also update the basedir in K:\dev\data\db\mysql\conf\mysql-master.ini:
In case of an upgrade sc delete mysqlmaster first (see also Section 1.1.5, “Task Manager replacement” in case of “The specified service has been marked for deletion.”).
Redefine the Windows service for the master: mysql-create-service master.
Restart the master with net start mysqlmaster.
Mysql_upgrade examines all tables in all databases for incompatibilities with the current version of MySQL Server. Mysql_upgrade also upgrades the system tables so that you can take advantage of new privileges or capabilities that might have been added.
Finally upgrade the master tables with mysql_upgrade -P 3306 -u root -p. After successfully running this command, type K:\dev\data\db\mysql\data\mysql_upgrade_info should display: 5.7.20.
After running mysql_upgrade, once again stop the server and restart it so that any changes made to the system tables take effect. 10.1.6. MySQL commands and connection settings
Some useful MySQL commands are:
• SHOW [FULL] PROCESSLIST shows you which threads are running and is useful to display a currently long running query.
• SHOW DATABASES; lists the databases on the MySQL server host.
• USE to use the database as the default (current) database for subsequent statements.
• DESC[RIBE]
; to get a MySQL table's structure.
• SHOW INDEX FROM
; to see all indexes of a table.
Your connection URL should look something like: jdbc:mysql://localhost:3306/data- base?useSSL=[false|true][&autoReconnect=true][&useUnicode=true] [&characterEncoding=UTF8][&connectionCollation=UTF8_bin].
70 Database
This will also enable/disable SSL and suppress the SSL logging “WARN: Establishing SSL connection without server's identity verification is not recommended.”. 10.1.7. SQL Manager installation guide
EMS SQL Manager for MySQL Freeware is an excellent freeware graphical tool for MySQL Server administration. It has minimal required set of instruments for those users who are new to MySQL server and need only its basic functionality.
Download the SQLManager for MySQL archive: mymanager_lite.zip [version 5.5.3.46192].
Extract this .zip file to C:\tmp and run the C:\tmp\MyManagerLite.msi to install SQL Manager in P:\dev \apps\db\sqlmanager.
Run the SQLManager and select Database → Register Database… and enter the following settings:
• Host name: localhost on Port: 3306. • User name: root. • Password: password and select Next. • Database name: mysql and click Finish. 10.1.8. Navicat installation guide
Navicat is a fast, reliable and affordable Database Administration tool purpose-built for simplifying database management and reducing administrative costs. Designed to meet the needs of database administrators, devel- opers, and small and medium businesses, Navicat is built with an intuitive GUI which lets you create, organize, access and share information in a secure and easy way.
Download the Win32 binary: navicat111_mysql_en_x64.exe [version 11.1.13].
Note
At the time of writing there isn't a lite version available anymore.
Run this .exe file to install Navicat Lite in P:\dev\apps\db\navicat.
Run the Navicat and select File → New Connection… → MySQL… and enter the following settings:
• Connection name: MySQL. • Host name/IP address: localhost. • Port: 3306. • User name: root. • Password: password and click Ok. 10.2. Oracle
Oracle Database. 10.2.1. Resources
• Oracle Database Software Downloads. • Get Oracle JDBC drivers from the Oracle Maven Repository. • Oracle SQL Developer is a free integrated development environment that simplifies the development and management of Oracle Database in both traditional and Cloud deployments. SQL Developer offers complete end-to-end development of your PL/SQL applications, a worksheet for running queries and scripts, a DBA con- sole for managing the database, a reports interface, a complete data modeling solution, and a migration plat- form for moving your 3rd party databases to Oracle.
71 72 Chapter 11. Chapter 11. Version Control System 11.1. Git
Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Every Git clone is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server. Branching and merging are fast and easy to do.
GitHub is a web-based hosting service for software development projects that use the Git revision control system. 11.1.1. Resources
• Download prepackaged binaries from Git for Windows. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: GitWeb - Perl implementation / gitweb.cgi and GitWeb - Repository styling git pull gitweb-theme. • A few git tips you didn't know about. • The Pro Git Book (see for example: Git Tools - Rewriting History). • github:help - Set Up Git for Windows and gitcheatsheet. • Changing history, How to Move Tags, Changing a commit message, How do I edit an incorrect commit message in git (I've pushed)?. • Git ready. • My quick and dirty dive into git theory and tricks. • Git Credential Manager for Windows. 11.1.2. Git installation guide
Download the binary: Git-2.13.0-64-bit.exe [version 2.13.0].
Run this .exe file to install Git in P:\dev\apps\vcs\git with the default settings.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %GIT_HOME%\bin;. Also add a New system variable GIT_HOME pointing to P:\dev\apps\vcs\git.
Verify the installation with git --version.
Now mkdir P:\dev\data\repo\git where the new project repositories will be residing. Also mkdir P:\dev \apps\vcs\git\tmp to prevent the following message “bash.exe: warning: could not find /tmp, please create”. 11.1.2.1. Git bash home directory
Right-click on the shortcut for Git Bash and change Properties → Start in from %HOMEDRIVE%%HOMEPATH% to C:\dev\projects and remove -cd option from the Target.
Tip
Auto-completion should be preconfigured during installation. When typing a git command press TAB for auto-completion when it should be obvious what the statement will be. Type TAB twice to get the suggestions when it isn't. For example git clo versus git cl (with suggestions: clean clone).
11.1.2.2. Global settings
Start Git Bash and setup user specific global settings with (see also C:\Users\\.gitconfig):
Alternatively use KDiff3 instead of WinMerge as the diff tool overwrite the following settings:
git config --global diff.tool kdiff3
Warning
When say you are working on a branch, that started just as the master with parent version 4.2, but now is it updated to 4.3.0. While the master got patched to parent version 4.3.4. You will no- tice a difference in kdiff3 between the content of the Local and Remote pom.xml when re- basing vs merging of the master onto the branch. The Local commit will always be the commit before the Remote commit in the newly created git log. Thus the current state of pom.xml in the branch could show up as Remote or as Local respectively (see also the screenshots below).
Figure 11.1. Git Rebase Master
Figure 11.2. Git Merge Master
74 Version Control System
11.1.2.5. Ignores template
Create the following ignores template P:\dev\data\repo\.gitignore based on Java.gitignore and Maven.gitignore of .gitignore templates:
Create the following file C:\dev\projects\test-drive\hello.txt:
Hello world! Second line. Third Line.
Stage hello.txt also and commit both files with:
git status git add hello.txt git status git commit -m 'initial commit.' git status git log --pretty=oneline
Create a new branch and check it out in one go:
git checkout -b hello-git git status
Change the branched C:\dev\projects\test-drive\hello.txt as follows:
Hello Git! Second line. 3rd Line.
Skip the staging command by committing the change immediately and switch back to the master branch:
git status git commit -a -m 'branch commit.' git log --pretty=oneline git status git checkout master git status
Append text to the head revision first line and change third line uppercase 'L' in C:\dev\projects\test-dri- ve\hello.txt as follows:
Hello world! Hello indeed. Second line. Third line.
Use WinMerge to view all/single diffs:
git difftool git difftool hello.txt
Commit this change to the master branch and try to merge both branches:
76 Version Control System
git status git commit -a -m 'head commit.' git status git merge hello-git git status git diff
Use KDiff3 to fix the merge conflicts:
git mergetool
For the first conflict select the content of both the file C and the file B with Ctrl+3 followed by Ctrl+2. Move to the next conflict with Ctrl+Down and select the content of the file B with Ctrl+2. Finally combine the first line copies of B and C in the output file to look like:
Hello Git! Hello indeed. Second line. Third line.
Quit KDiff3 with Ctrl+Q and verify the resulting patch or diff using WinMerge:
Git-http-backend is a simple CGI program to serve the contents of a Git repository to Git clients accessing the repository over http and https protocols. The program supports clients fetching using both the smart HTTP proto- col and the backwards-compatible dumb HTTP protocol, as well as clients pushing using the smart HTTP protocol.
Define a macro to delegate the handling of all URLs whose path portions begin with /git-repo/ to use this Smart HTTP Transport with the following include just after httpd-macro.conf in P:\dev\apps\httpserv- er\apache-conf\httpd.conf:
# Apache Macro Include ../apache-conf/httpd-macro.conf Include ../apache-conf/httpd-git-macro.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-git-macro.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf
77 Chapter 11.
Use this macro in P:\dev\apps\httpserver\apache-conf\httpd-vhosts.conf:
…
…
…
# Git Use GitRepos /git-repo P:/dev/data/repo/git
Restart the Apache HTTP Server.
You may need to reboot the server for the Path setting for GIT_HOME [73] to take effect. 11.1.4.1. Git repository
Also define in this macro to delegate the handling of all URLs whose path portions begin with /git-repo/ to the CGI (Common Gateway Interface) of GitWeb which allows browsing a Git repository (or a set of Git reposi- tories) using a web browser in the above created file P:\dev\apps\httpserver\apache-conf\httpd-git- macro.conf:
… Include ../apache-conf/httpd-freedumbytes-realm.conf
Supply the extra parameter for $gitweb_conf filename in P:\dev\apps\httpserver\apache-conf\httpd- vhosts.conf:
# Git Use GitRepos /git-repo P:/dev/data/repo/git gitweb.conf
78 Version Control System
11.1.4.2. GitWeb Perl implementation
Because GitWeb isn't working with ActivePerl configure it to use the supplied Perl implementation of mysysGit in P:\dev\apps\vcs\git\mingw64\share\gitweb\gitweb.cgi:
When the Apache HTTP Server is restarted you should be able to browse the Git Repository at http:// freedumbytes.dev.net/git-repo. 11.1.4.3. Git private repository
With the above macro setup, it is possible to simply add a private Git repository that is only available at for example http://localhost/git-private with the following changes in P:\dev\apps\httpserver\apache-conf\httpd- vhosts.conf:
…
Use GitRepos /git-private P:/dev/data/repo/private/git gitweb-private.conf
with its own GitWeb configuration file P:\dev\data\httpserver\home\git\gitweb-private.conf:
tmp=${TMPDIR-/tmp} tmp=$tmp/git-clone.$RANDOM.$RANDOM.$RANDOM.$$ echo "Create temporary clone of project $project_name in $tmp" mkdir $tmp || exit 3 cd $tmp git clone $git_repo/$project_name $tmp/$project_name cd $project_name
Note
See Safely Creating Temporary Files in Shell Scripts.
echo "Add .gitignore to project" cp P:/dev/data/repo/.gitignore $tmp/$project_name git config user.name "freedumbytes.dev.net GitWeb" git config user.email [email protected] git config core.autocrlf false (see also note below about line endings) git add .gitignore git commit -am "initial commit." git push origin master
echo "Cleanup temporary clone of project $project_name in $tmp" cd ../.. rm -rf $tmp || exit 4 fi fi
Note
Line endings… the scourge of every Windows-based developer that tries to mingle with linux- or mac-based developers. Though most modern text editors can handle both newline types without issue, git is not as graceful. Combined with Eclipse issue 301775: “EGit/JGit ignores the core.autocrlf flag which may be quite inconvenient for Windows users” the command git config core.autocrlf false is added to the above script. The article linked at the beginning of this note also supplies information to fix this setting issue and how to reset your repos at a later stage. If you're using Git to collaborate with others on GitHub, GitLab and Bitbucket, ensure that Git is properly configured to handle line endings. Speaking about those repository management services take a look at Appendix B, Hosting a group Website in the Cloud and Section B.5, “Multiple remote git locations”.
Create a new bare repository with the following command git-create-repo.sh P:/dev/data/repo/git dpl-manual.git "Production Development Line manual" -jenkinsHook. 11.1.5.1. Anonymous Jenkins Notify Hook
To enable anonymous access from the local area network to the Jenkins hook edit P:\dev\apps \httpserver\apache-conf\httpd-jenkins.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf
81 Chapter 11.
Include ../apache-conf/httpd-lan-access.conf
11.1.6. Working on a project
Ask Git to clone a copy of a repository with the following batch file P:\dev\apps\windows\batch\git- clone.bat:
@echo off
if not "%2"=="" goto cloneRepo
echo Usage: git-clone ^ ^ goto end
:cloneRepo cd /d C:\dev\projects if exist \projects\%2 ren "%2" "%2.old" git clone http://%[email protected]/git-repo/%2 cd %2 git config core.autocrlf false (see also note about line endings)
:end
Clone the remote repository with the following command git-clone jjasper dpl-manual.
Now you have a personal copy of the remote repository in a new directory named C:\dev\projects\dpl-man- ual. You can edit the files in your working copy and then commit and push those changes back into the remote repository. 11.1.7. Git commands
Other useful Git command options are:
• git help for more information on a specific command.
• git branch to list, create, or delete branches (see also Using branch): • git branch [-r] [--all] to list ([remote-tracking] or [all]) branches. • git branch -[d|D] to (force) delete a local branch. • git push origin : to remove a remote branch. • git branch -f master to replace local master branch and git push remote +:master for the remote branch. • git branch -M master to rename the branch to master. • git checkout -B origin/ to reset a local branch (or git reset --hard origin/; see also Undo-ing ‘git reset hard’ about git fsck [--lost-found]. The dangling objects are written into .git/lost-found/commit/ or .git/lost-found/other/, depend- ing on type. If the object is a blob, the contents are written into the file, rather than its object name. But all uncommitted/unadded modified files will be gone. See also How to recover a git branch you accidentally deleted with git reflog). • git fetch -p to prune local references to remote deleted branches. • git checkout --orphan if you want a branch that is empty and has no history, this is the way to go.
git rm -rf . to remove all the files you'll have in the staging area (so that they don't get committed).
git commit --allow-empty -m "root commit" before you can push to GitHub (or any other Git repository), you will need at least one commit, even if it does not have any content on it (i.e. empty commit), as you cannot push an empty branch.
git push origin to finally, push it to the remote, and crack open a beer or SPA Intense for that matter.
• git tag to list, create, or delete tags: • git tag [-l] to list tags. • git tag -a and git push --tags to create an unsigned, annotated tag object. • git tag -d to delete a tag and also on remote push origin :refs/tags/. • git fetch --tags to get all the tags and not just ones referencing commits reachable from the branch heads. Does git fetch --tags include git fetch?
82 Version Control System
• git tag -l | xargs git tag -d followed by the above mentioned command git fetch -t to prune local git tags that don't exist on remote and thus keeping others from pushing them back.
• git status to list files changed in working directory.
• git add file contents to the index aka staging area: • git add . to stage new and modified, without deleted. • git add . -u to stage modified and deleted, without new. • git add (-A|--all) to stage all.
• git stash [pop] temporarily shelves (or stashes) changes you've made to your working copy so you can work on something else, and then come back and re-apply them later on.
• git checkout a branch or paths to the working directory (see also Using checkout): • git checkout [-b] to checkout [and create new] branch. • git ls-files -d | xargs git checkout -- to recover all unstaged deletions at once, automatically, without specifying each single path. • git status | grep 'deleted:' | awk '{print $2}' | xargs git checkout -- to recover all staged deletions at once, automatically, without specifying each single path.
• git commit to record changes to the local repository: • git commit -[a]m "use -a to skip add of modified file to staging area first.". • git commit --amend is a convenient way to modify the most recent commit. It lets you combine staged changes with the previous commit instead of creating an entirely new commit. It can also be used to simply edit the previous commit message without changing its snapshot. But, amending does not just alter the most recent commit, it replaces it entirely, meaning the amended commit will be a new entity with its own ref (see also Rewriting history). • git ls-files --eol | grep auto to identify files that aren't covered by .gitattributes and thus is marked for automatic end-of-line conversion.
• git fetch latest changes from origin to local repository: • git diff master origin/master to see what the differences are between your branch and the remote one.
git fetch origin master:master to Git: Fetch and Merge, Don't Pull.
git rebase [--preserve-merges] master
The arguments after the remote origin in fetch are refspecs. Using master:master will overwrite your mas- ter branch: • Differences between git fetch and git fetch origin master. • What is the difference between these `git fetch` syntaxes?
Important
--preserve-merges or -p recreate merge commits instead of flattening the history by replaying commits a merge commit introduces. Merge conflict resolutions or manual amendments to merge commits are not preserved.
This uses the --interactive or -i machinery internally, but combining it with the -- interactive option explicitly is generally not a good idea unless you know what you are doing (see BUGS).
• git pull latest changes from origin and then merge into the current branch.
• git push your work back up to the origin.
• git diff displays changes between commits, commit and working directory, etc: • git diff shows what has changed but hasn't been added to the index yet via git add. • git diff (--cached|--staged)shows what has been added to the index via git add but not yet committed. Useful to see trailing spaces in new files, that weren't listed in git diff, when those files were untracked before git add .. • git diff (HEAD shows what has changed since the last commit. It is the combination of git diff and git diff (--cached|--staged).
Tip
As of Git 1.8.5, @ is an alias for HEAD.
83 Chapter 11.
• git diff (HEAD^|HEAD~)shows what has changed since the commit before the latest commit. Useful to check diff before usage of git commit --amend.
Tip
git difftool HEAD~1 HEAD~0.
Note
Ancestry References by placing a ^ (caret) or ~ (tilde) at the end of a reference (see also examples for git show).
• git diff -- module/pom.xml compares a single file between two branches/tags.
• git mv is a convenience function for renaming files.
• git remote to manage the remote repositories (see also Changing a remote's URL): • git remote -v to list them all. • git clone creates the origin. • git remote add to add a new remote repository as a shortname (see also TODO push url xref). • git remote show [] to inspecting a remote repository.
• git submodules allow you to keep a Git repository as a subdirectory of another Git repository (see also Git Tools - Submodules): • git clone --recurse-submodules … will automatically initialize and update each submodule in the repository. • git submodule update --recursive --init initialize each submodule for already cloned reposi- tories. • git submodule update --recursive --remote update each submodule for already cloned repos- itories.
• git revert some existing commits.
• git reset undo changes (see also note for git rebase): • git reset [(HEAD~|) --hard] and optionally git push origin (HEAD|) -f to remove a commit(, that is already pushed). • git push origin +: to remove a commit that is already pushed. • git reset --soft HEAD~1 to undo the last commit, but the file changes will stay in your working tree. • git reset HEAD~1 to undo the last commit, but the file changes will stay in your working tree, but not on the index. • git clean [-f] to remove untracked files from the working tree. • git clean -[f]d to remove all untracked files and directories from the working tree.
Tip
Add -n to clean to see what files/directories will be removed without actually removing them.
• git show one or more objects such as blobs, trees, tags and commits (see also Revision Selection): • git show [(HEAD~|)] [--name-only] to show commit diff and/or files. • git show [(HEAD|HEAD~0|HEAD^0)] shows the latest commit. • git show (HEAD~|HEAD~1|HEAD^|HEAD^1) shows the commit before the last commit or the parent commit. • git show (HEAD~~|HEAD~2|HEAD^^) shows the grandparent commit.
Note
git show HEAD^2 syntax is only useful for merge commits, which have more than one parent. The first parent is the branch you were on when you merged, and the second is the commit on the branch that you merged in.
84 Version Control System
• git show HEAD@{2017-10-20} means “where HEAD was on 2017-10-20” in this repository (using the reflog). • git reflog show [--all] to get a complete list of reference logs. Reference logs, or reflogs, record when the tips of branches and other references were updated in the local repository. • git rev-list --all [--remotes] --pretty=oneline [-- <(file)path>] to list commit objects in reverse chronological order and query git reflog for all commits to a specific file.
• git merge joins two or more development histories together (see also Using merge): • git checkout master to switch back to master for an update from remote git pull origin master and after return to git checkout to git merge master (see also Merging vs. Rebasing). • git merge and use git mergetool and git rebase --continue to fix merge conflicts (see also Section 11.1.2.4, “External merge tool”).
• git rebase takes all the changes that were committed on one branch and replays them on another one (see also What is git rebase?, Don't Be Scared of git rebase and Git rebase is not a lion, it's a cat - Part One and Two): • git rebase [(HEAD~|)] -i to pick, reword,edit, squash, fixup, exec, drop commits. • git rebase -i --root to rebase to the root commit of the branch.
Note
Do not rebase commits that you have pushed to a public repository. Also take a look at Why you should stop using Git rebase.
Tip
Split a commit in two with Git:
# git rebase/reset starting with hash of commit just before combined commit, that was actually made of two. git rebase -i # mark the combined commit as 'edit' git reset HEAD^ git add … git commit -m "First commit." git add … git commit -m "Second commit." git rebase --continue
Important
git add -p … is the most powerful git feature you're not using yet. Patch mode allows you to stage parts of a changed file, instead of the entire file. git add -i … is the big brother of -p. --patch only lets you decide about the individual hunks in files. --interactive enters the interactive mode, and is a bit more powerful.
• git reflog keeps track of updates to the tip of branches using a mechanism called reference logs, or reflogs. Many Git commands accept a parameter for specifying a reference or ref, which is a pointer to a commit.
• Git manages file permissions for each file in the repository. It may be appropriate to have the executable bit set for shell/bash files to make them easier to execute in Linux systems.
For example branch executable-bit has the following committed, added and new untracked executable files when compared with the master branch:
Git does not track an empty directory. In order to track a directory, a file must exist within it. A trick that many people use is to create a hidden file usually called .gitkeep. The name of the file doesn't matter.
A way to place an empty copy of the .gitkeep file in every empty folder would be to use the following command:
Subversion is a version control system that is a compelling replacement for CVS in the open source community. 11.2.1. Resources
• Download prepackaged binaries. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: -. • The Subversion Book: Version Control with Subversion. • Subversion Quick Start, Complete Reference and CVS to SVN Crossover Guide. • SVNKit is a pure Java Subversion library. You would like to use SVNKit when you need to access or modify Subversion repository from your Java application, be it a standalone program, plugin or web application. 11.2.2. Subversion installation guide
Download the archive (alternative location): svn-win32-1.8.14-ap24.zip for Apache 2.4.x [version 1.8.14].
Extract this .zip file to P:\dev\apps\vcs.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %SVN_HOME%\bin;. Also add a New system variable SVN_HOME pointing to P:\dev\apps\vcs\svn-win32-1.8.14. Optionally add a New system variable SVN_EDITOR pointing to for example P:\dev\apps\editor\ultraedit\Uedit32.exe.
Verify the installation with svn --version. 11.2.3. Global ignores
Edit the %APPDATA%\subversion\config file to ignore Eclipse, Maven and backup files as follows:
To immediately delete a directory with its subdirectories use svn delete -m "Delete resources from project X." http://freedumbytes.dev.net/svn-repo/project-x/trunk/src/main/resources/.
This is useful when svn delete src\main\resources followed by svn commit -m "Delete resources from project X." results in “svn: Aborting commit: 'C:\dev\projects\project-x\src\main\resources' remains in conflict”.
87 Chapter 11.
11.2.5. Credentials
Sometimes users will want to remove specific credentials from the disk cache. To do this, you need to navigate into the %APPDATA%\Subversion\auth\svn.simple area and manually delete the appropriate cache file. Creden- tials are cached in individual files; if you look inside each file, you will see keys and values. The svn:realmstring key describes the particular server realm that the file is associated with, for example:
K 8 passtype … K 15 svn:realmstring V 61 CollabNet Subversion Repository K 8 username … END
11.3. FishEye
Your source code repository contains an abundance of useful information that is not easy to extract, compre- hend or keep up to date. FishEye painlessly opens up your repository to help you better understand your changing source. 11.3.1. Resources
• FishEye Documentation. • Download FishEye and Plugins Marketplace such as Fisheye Release Report. • Migrating to MySQL. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: FishEye - Apache configuration Alias /fisheye/static, FishEye - MySQL Connector/J mysql-connector-java-x.y.z-bin.jar and FishEye - Windows service Fish- eye-Control.bat / wrapper folder / Fisheye-Install-NTService.bat. 11.3.2. FishEye installation guide
Download the archive: fisheye-4.5.1.zip [version 4.5.1].
Extract this .zip file to P:\dev\apps\vcs.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Envi- ronment Variables and Edit the Path to append %FISHEYE_HOME%\bin;. Also add a New system variable FISHEYE_HOME pointing to P:\dev\apps\vcs\fecru-4.5.1.
Verify the installation with: .
cd /d %FISHEYE_HOME% set JAVA_HOME=P:\dev\apps\prg\java-x64\jdk1.8.0_162 fisheyectl.bat run
Terminate the server with Ctrl+C.
Warning
If you now notice that FishEye tries to start spontaneously in its own Windows Command Processor (cmd.exe) when the server is rebooted, you might have some startup setting that triggers the start.bat file. In my case it was the auto-start of the Task Manager during boot up. Just delete the start.bat, stop.bat and run.bat (or rename to start.bat.disable for example) because they redirect to fisheyectl.bat anyway.
Terminate the server with Ctrl+C.
For the ease of future upgrades press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add a New system variable FISHEYE_INST pointing to P:\dev \data\vcs\fisheye. And perform the following steps:
Restart FishEye again with fisheyectl start. Verify that the server is running in your browser at http://local- host:8060/.
Note
In case of “java.lang.OutOfMemoryError: PermGen space” just set FISHEYE_OPTS=- Xmx512m -Xms128m before running fisheyectl start (see also Section 11.3.5, “Windows service”).
When you access FishEye for the first time you will be asked for:
1. To Obtain evaluation license and Enter existing license. 2. Please include Crucible as part of this evaluation. 3. Skip for now Connect to JIRA. 4. Create password and Confirm password.
If you need to reset the administrator password, delete the admin-hash in the config element of your config.xml.
Connect to the FishEye Admin GUI http://localhost:8060//admin/ and configure FishEye. 11.3.3.1. Global Settings
Select Global Settings → Server → Web Server → Edit settings:
• Web context: fisheye. • HTTP Bind Address: 127.0.0.1:8060. • Ajp13 Bind Address: 127.0.0.1:8061. • Site URL: http://freedumbytes.dev.net/fisheye/. • Timezone : Europe/Amsterdam and click Update.
Select Global Settings → Server → Mail Server → Edit config:
• Send mail from: Server Address (below). • From Address: [email protected]. • SMTP Host name: mail.freedumbytes.dev.net and click Save, followed by Send test email.
Stop/start FishEye using fisheyectl. Verify that the server is running in your browser at http://local- host:8060/fisheye. 11.3.4. Apache configuration
Instruct Apache to proxy (using mod_proxy_ajp) all URLs whose path portions begin with /fisheye/ using the following P:\dev\apps\httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# FishEye Include ../apache-conf/httpd-fisheye.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-fisheye.conf:
Alias /fisheye/static "P:\dev\apps\vcs\fecru-4.5.1\content\static"
ProxyPass /fisheye/static/ !
Include ../apache-conf/httpd-freedumbytes-realm.conf
When the Apache HTTP Server is restarted you should be able to browse FishEye at http://freedumbytes.dev.net/ fisheye. 11.3.4.1. Privacy Policy
Next configure if Analytics collects data about the usage of FishEye and sends it to Atlassian (see the privacy policy for details). 11.3.4.2. Security
Next configure at the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ the Security Settings → Authentication:
• For Global Permissions: Global Anonymous Access and Crucible Anonymous Access select Turn Off. • For Built-in: Public Signup select Turn Off. • For User List Visibility and Email Visibility: select Visible to logged in users only. • For Authentication settings click Setup AJP13 authentication: • Cache TTL (positive): 5 mins. • Auto-add: Create a FishEye user on successful login and Apply those settings.
Also configure the Repository Settings → Defaults:
• For Permissions: Anonymous uncheck Can Read, All logged-in users check Can Read and Save changes.
Stop/start FishEye using fisheyectl. When the Apache HTTP Server is restarted you should be able to browse FishEye with your freedumbytes.dev.net realm username and password.
Once the user, who is also the FishEye administrator, has logged in select Security Settings → Administrators and place yourself in the Admin Users. This way you no longer need to enter the Administration Password.
Tip
The HTTP Bind Address 127.0.0.1:8060 could be removed now that the AJP authentication is setup and running. In case of problems with the AJP Bind Address you will be able to reset the HTTP Bind Address by reinserting it in the config.xml as follows:
11.3.5. Windows service
To restart automatically on Microsoft Windows, create a Windows service.
Download the Tanuki Java Service Wrapper archive: wrapper.zip.
Extract this .zip file to P:\dev\apps\vcs\fecru-4.5.1.
90 Version Control System
Note
In case of an upgrade just copy the one extracted in the prior FishEye version:
Optionally comment out the jmx entries and edit the following entries - in the service wrapper config- uration file P:\dev\apps\vcs\fecru-4.5.1\wrapper\conf\wrapper.conf:
wrapper.java.additional.104=-Dfile.encoding=UTF-8 wrapper.java.additional.5=-Dfisheye.inst=%FISHEYE_INST% #wrapper.java.additional.6=-Xrs see also Section 3.1.3.1, “Standard Edition Tools Reference”
# Initial Java Heap Size (in MB) wrapper.java.initmemory=256
# Maximum Java Heap Size (in MB) wrapper.java.maxmemory=1024
…
# Log file to use for wrapper output logging. wrapper.logfile==var/log/wrapper.log%FISHEYE_INST%/var/log/wrapper.log
…
#******************************************************************** # Wrapper Windows Properties #******************************************************************** # Title to use when running as a console wrapper.console.title=Fisheyefreedumbytes.dev.net FishEye
#******************************************************************** # Wrapper Windows NT/2000/XP Service Properties #******************************************************************** # WARNING - Do not modify any of these properties when an application # using this configuration file has been installed as a service. # Please uninstall the service before modifying this section. The # service can then be reinstalled.
# Name of the service wrapper.ntservice.name=Fisheyefisheye wrapper.ping.timeout=300 (see also Error "JVM appears hung" in wrapper.log)
91 Chapter 11.
# Display name of the service wrapper.ntservice.displayname=freedumbytes.dev.net FishEye
# Description of the service wrapper.ntservice.description=FishEye gives Git, Subversion and CVS a Web interface.
Register the FishEye service from folder cd /d %FISHEYE_HOME% in an elevated command prompt: wrap- per\bin\Fisheye-Install-NTService.bat. Start FishEye with net start fisheye.
Note
In case of an upgrade sc delete fisheye first (see also Section 1.1.5, “Task Manager re- placement” in case of “The specified service has been marked for deletion.”).
Important
In case of a new jdk installation use an elevated command prompt just run net stop fisheye and net start fisheye.
Note
Should the following error “Error: no `server' JVM at `P:\dev\apps\prg\java-x64\jre…\bin\serv- er\jvm.dll'” occur check for stray java.exe files (see tip and warning in Section 3.1.2, “JDK instal- lation guide”)
Verify the installation and the JVM memory usage at http://freedumbytes.dev.net/fisheye/admin/sysinfo.do. 11.3.5.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net FishEye and select Properties → Log On to use This ac- count .\Tomcat. After password conformation click Apply and restart the service. 11.3.6. Database migration
For migration from the default embedded FishEye HSQLDB database to an external database create a MySQL user called fishseye-user and database called fisheye with the following commands:
mysql -u root -p mysql> create database fisheye character set utf8 collate utf8_bin; mysql> grant all privileges on fisheye.* -> to 'fisheye-user'@'localhost' identified by 'password'; mysql> flush privileges; mysql> select host,user from mysql.user; mysql> select host,db,user from mysql.db; mysql> quit
Download the Connector/J [version 5.1.45] and add the driver mysql-connector-java-5.1.45-bin.jar from the archive mysql-connector-java-5.1.45.zip to the FishEye classpath placing it into P:\dev\da- ta\vcs\fisheye\lib. For older versions of FishEye remove the one that is already in P:\dev\apps\vcs \fecru-4.5.1\lib\dbdrivers\mysql. Requires a restart of FishEye to locate this new jar.
Connect to the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ and configure MySQL database.
Select Systems Settings → Database → Edit:
• Test Connection of Built-In hsqldb. • Type: MySQL. • Driver Location: User Supplied - FISHEYE_INST\lib. • URL: jdbc:mysql://localhost/fisheye. • User Name: fisheye-user. • Password: password. • Minimum Pool Connections: 5.
92 Version Control System
• Maximum Pool Connections: 20. • Parameters: • useUnicode=true • characterEncoding=UTF8 • connectionCollation=UTF8_bin • useSSL=false • Test Connection and click Save & Migrate. 11.3.7. Repository Settings
Connect to the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ and configure Repository Set- tings.
• Repository type: Git. • Name: dpl-manual. • Description: Production Development Line manual. and click Next. • Repository Location: http://[email protected]/git-repo/dpl-manual.git (see al- so Section 7.1.4.1, “Authentication, authorization and access control”). • Authentication Style: Password for http(s). • Password: password and click Show advanced settings. • Rename Detection: copies and click Next. • Store Diff Info: enabled. • Enable Repository After Adding:disabled, click Test Connection and Add this repository when Connection succeeded.
• Repository type: Subversion. • Name: svn. • Description: Subversion Repository. and click Next. • SVN URL: http://localhost/svn-repo. • Username: fisheye (see also Section 7.1.4.1, “Authentication, authorization and access control”). • Password: password and click Show advanced settings. • Charset: default (UTF-8). • Initial Import: Do not Import. • Use Built-in Symbolic Rules: enabled. • And then Apply the Following Rules: /project/trunk/…, /project/branches/NAME/…, /project/ tags/NAME/… and click Next. • Store Diff Info: enabled. • Enable Repository After Adding:disabled, click Test Connection and Add this repository when Connection succeeded.
To activate a repository select for example Repositories → dpl-manual → Summary and Actions → Enable → Start → Save. 11.3.8. User profile
On the Dashboard page click on the logged-in user at the top right of the menu bar and select Settings to edit for example:
• Display Settings. • Profile & Email.
Important
The repository types CVS and Subversion map commits based on the username. To match Git commits select Administration → Users and Edit for example jjasper to set his Email to the one supplied to git config --global user.email [email protected].
• Author Mapping and Add for CVS the commiter: %USERDOMAIN%\jjasper.
JIRA is a bug tracking, issue tracking, and project management application. 12.1.1. Resources
• JIRA Documentation. • Download JIRA and Plugins Marketplace such as FishEye integration, Jenkins integration and SonarQube in- tegration. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: JIRA - HTTP Binding address, JIRA - MySQL Connector/J mysql-con- nector-java-x.y.z-bin.jar, JIRA - Properties jira.home, JIRA - Apache Configuration Context path / Connector protocol AJP, JIRA - Windows service tomcat8.exe.x64 / tcnative-1.dll.x64 / tomcat8.exe //US//jira and JIRA - FishEye plugin upgrade.
Note
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe P:\dev\apps\is- sue\atlassian-jira-6.4.11-standalone\conf P:\dev\apps\issue\at- lassian-jira-software-7.6.2-standalone\conf to copy the changes in for ex- ample server.xml,
Note
And run p:\dev\apps\editor\winmerge\WinMergeU.exe P:\dev\apps\is- sue\atlassian-jira-6.4.11-standalone\atlassian-jira\WEB-INF \classes\jira-application.properties P:\dev\apps\issue\atlass- ian-jira-software-7.6.2-standalone\atlassian-jira\WEB-INF\classes to copy the changes in for example jira-application.properties.
12.1.2. JIRA installation guide
Download the ZIP Archive atlassian-jira-software-7.6.2.zip [version 7.6.2].
Extract this .zip file to P:\dev\apps\issue.
Important
As of JIRA 4.3 “in-place database upgrades” is officially supported.
12.1.2.1. HTTP Binding
Change the listening ports and and restricted to localhost by editing the configuration file P:\dev \apps\issue\atlassian-jira-software-7.6.2-standalone\conf\server.xml:
…
12.1.2.2. Database settings
Create a MySQL user called jira-user and database called jira with the following commands:
mysql -u root -p mysql> create database jira character set utf8 collate utf8_bin; mysql> grant all privileges on jira.* -> to 'jira-user'@'localhost' identified by 'password'; mysql> flush privileges;
95 Chapter 12.
mysql> select host,user from mysql.user; mysql> select host,db,user from mysql.db; mysql> quit
Download the Connector/J [version 5.1.45] and add the driver mysql-connector-java-5.1.45-bin.jar from the archive mysql-connector-java-5.1.45.zip to the JIRA classpath placing it into P:\dev\apps \issue\atlassian-jira-software-7.6.2-standalone\lib. For older versions of JIRA remove the one that is already there.
Create the JIRA database connection information in P:\dev\data\issue\jira\dbconfig.xml (see also Tun- ing Database Connections):
Edit the data directory by changing the following property in the P:\dev\apps\issue\atlassian-jira-software-7.6.2-standalone\atlassian-jira\WEB-INF \classes\jira-application.properties file:
# The jira.home configuration must be set and specifies the directory # in which JIRA will store its data files. # This must be set to an absolute path. Relative paths are not allowed. # Ensure that only one JIRA instance uses the selected JIRA Home. # jira.home = P:/dev/data/issue/jira
Note
It is possible to overwrite this value with JIRA_HOME (see also Section 12.1.5, “Windows ser- vice”).
Disable Secure Administrator Sessions by creating the following property in the P:\dev\data\issue\ji- ra\jira-config.properties file:
jira.websudo.is.disabled=true
Verify the installation by changing to cd /d P:\dev\apps\issue\atlassian-jira-soft- ware-7.6.2-standalone and running bin\startup.bat.
Verify that the server is running in your browser at http://localhost:8063. The Server ID for this JIRA instance is required to retrieve a new license from my.atlassian.com.
96 Issue Tracker
12.1.3.1. Required settings
You should see the JIRA setup wizard:
1. Application Properties: • Application Title: freedumbytes.dev.net JIRA. • Mode: Private. • Base URL: http://freedumbytes.dev.net/jira and click Next. • Automated Backups: Disable Automated Backups. TODO
2. Specify Your License Key: • Your License Key: … and click Next.
3. Set Up Administrator Account: • Fullname: JIRA Administrator. • Email: [email protected]. • Username: admin. • Password: password. • Confirm Password: password and click Next.
4. Set Up Email Notifications: • Configure Email Notifications: Now. • Name: JIRA SMTP Server. • From address: [email protected]. • Email prefix: [JIRA]. • Server Type: SMTP Host. • Service Provider: Custom. • Host Name: mail.freedumbytes.dev.net. • Protocol: SMTP. • SMTP Port: 25. • Username: leave empty. • Password: leave empty and click Finish. 12.1.3.2. User, groups & roles
User Management menu:
• Select Groups and in Add Group form type: • Name: jira-system-administrators and click Add Group.
For jira-system-administrators select Edit Members: • Under Add members to selected group(s) select user for group by clicking . • Check admin and click Select. • Next click << Join.
• Select Users and click Add User: • Username: jjasper. • Password: password. • Confirm: password. • Full Name: Jene Jasper. • Email: [email protected] and click Create and Edit Groups: • In Available Groups select jira-system-administrators / jira-administrators / jira-developers and click Join >> and << Return to viewing user 'Jene Jasper'.
• Select Global permissions in Add Permission form choose: • Permission: JIRA System Administrators. • Group: jira-system-administrators and click Add.
In JIRA Permissions form: • For JIRA System Administrators select delete the jira-administrators and confirm Delete.
• Select User Default Settings and click Edit default values: • Default outgoing email format: html. • Notify users of their own changes: No and click Update.
And Apply the email format setting for existing users. 12.1.4. Apache configuration
Configure the JIRA application server to accept an ajp connection on web context /jira by edit- ing the configuration file P:\dev\apps\issue\atlassian-jira-software-7.6.2-standalone\conf \server.xml:
97 Chapter 12.
…
…
…
Instruct Apache to proxy all URLs whose path portions begin with /jira/ using the following P:\dev\apps \httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# JIRA Include ../apache-conf/httpd-jira.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-jira.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf #
When JIRA and the Apache HTTP Server are restarted you should be able to browse JIRA at http:// freedumbytes.dev.net/jira.
Tip
If the browser displays a 401 HTTP error then the Apache HTTP Server logged-in user doesn't exist in JIRA or the password doesn't match.
In case of a pop-up with “The Gadget feed URL health check has failed in your system.” or “JIRA is not able to access itself through the Gadget feed URL. This is necessary so that dashboard gadgets can be generated successfully. Please verify the current Base URL and if necessary, review your network configurations to resolve the problem.” on JIRA ADMINISTRATION → System → Troubleshooting and support tools → Instance health resolve this with anonymous access for the JIRA Server IP itself:
98 Issue Tracker
Include ../apache-conf/httpd-freedumbytes-realm.conf + + + + Include ../apache-conf/httpd-server-access.conf + Include ../apache-conf/httpd-freedumbytes-realm.conf + +
12.1.4.1. Privacy Policy
Next configure if Analytics collects data about the usage of JIRA and sends it to Atlassian (see the privacy policy for details).
12.1.5. Windows service
To restart automatically on Microsoft Windows create a service using an elevated command prompt as follows:
cd /d P:\dev\apps\issue\atlassian-jira-software-7.6.2-standalone
set JAVA_HOME=P:\dev\apps\prg\java-x64\jdk1.8.0_162 set CATALINA_HOME=P:\dev\apps\issue\atlassian-jira-software-7.6.2-standalone bin\service.bat install jira
In case of an upgrade sc delete jira first (see also Section 1.1.5, “Task Manager replace- ment” in case of “The specified service has been marked for deletion.”).
Tip
As an alternative to change the additional settings of the JIRA service run: P:\dev\apps \issue\atlassian-jira-software-7.6.2-standalone\bin\tomcat8w.exe // ES//jira such as Java → Java Virtual Machine P:\dev\apps\prg\ja- va-x64\jdk1.8.0_162\jre\bin\server\jvm.dll.
Important
In case of a new jdk installation just run P:\dev\apps\issue\atlassian-jira-soft- ware-7.6.2-standalone\bin\tomcat8.exe //US//jira --Jvm auto (when JAVA_HOME is still pointing to the 64-bit version; see also Section 8.1.3.2, “Tomcat Manager”).
Verify the installation and the JVM memory usage at http://freedumbytes.dev.net/jira/secure/ad- min/ViewSystemInfo.jspa.
99 Chapter 12.
12.1.5.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net JIRA and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service.
12.1.6. Global settings
System menu:
• Select General Configuration and click Edit Settings: • Allow unassigned issues: On. • Accept remote API calls: On (for Eclipse Mylyn plugin support) and click Update.
Note
When using Mylyn and “Submit failed” in Eclipse with following error message “JIRA could not complete this action due to a missing form token.” disable Form Token Check- ing by creating/editing the following property in the P:\dev\data\issue\ji- ra\jira-config.properties file:
Shut down JIRA first before editing P:\dev\data\issue\jira\ji- ra-config.properties.
• Under Advanced select Attachments and click Edit Settings: • Attachment Path: Use Default Directory and click Update.
• Under Issue Features activate Time Tracking.
• Under Issue Features activate Issue Linking and define your own Types:
100 Issue Tracker
Figure 12.1. JIRA Issue Linking Types
• Under User Interface select Look and Feel and click Edit Settings: • Time Format: HH:mm. • Day Format: EEEE HH:mm. • Complete Date/Time Format: yyyy-MM-dd HH:mm. • Day/Month/Year Format: yyyy-MM-dd. • Check Use ISO8601 standard in Date and click Update.
• Under General Configuration click Advanced Settings and select every Value to be edited: • jira.date.picker.java.format: yyyy-MM-dd and click Update. • jira.date.picker.javascript.format: %Y-%m-%d. • jira.date.time.picker.java.format: yyyy-MM-dd HH:mm. • jira.date.time.picker.javascript.format: %Y-%m-%d %H:%M.
• Under User Interface select System Dashboard and click Add Gadget such as FishEye Charts.
Issues menu:
• Under Issue Types select Sub-Tasks and Enable Sub-Tasks. 12.1.7. Project
Projects menu:
• Select Projects and in Add Project form type: • Name: Development Production Line manual. • Key: DPLMNL. • Project Lead: Jene Jasper and click Add.
Click Edit Project to supply a Description: DocBook about setting up a Production Development Line. and click Update.
Under Roles for Default Assignee click to change it to Unassigned and click Update.
Under Components:
101 Chapter 12.
• Name: manual with Description: The Short Story. and click Add.
Under Versions: • Name: 2.0 Description: The complete picture. with expected Release Date 2011-08-31 and click Add. 12.1.8. FishEye integration [ToDo Disabled or OAuth only] 12.1.8.1. Integration users
To auto-add the jira user (see also Section 7.1.4.1, “Authentication, authorization and access control”) in FishEye just login with that user at http://freedumbytes.dev.net/fisheye/ and logout again.
Optionally at the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ select the User Settings → Users and Edit user jira:
At the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ select the Global Settings → Applica- tion Links and Create new link: http://freedumbytes.dev.net/jira. Disable I am an administrator on both instances and click Continue. Ignore Create reciprocal link and click Close. Edit this new Application Link freedumbytes.dev.net JIRA to change the Outgoing Authentication to Basic Access using the fisheye inte- gration user from the prior paragraph and click Close.
Next at the JIRA Admin GUI Add-ons http://freedumbytes.dev.net/jira/plugins/servlet/applinks/listApplication- Links Create new link: http://freedumbytes.dev.net/fisheye. Disable I am an administrator on both instances and click Continue. Ignore Create reciprocal link and click Close. Edit this new Application Link free- dumbytes to change:
• Application Details the Application Name to freedumbytes.dev.net FishEye. • Outgoing Authentication to enable Basic Access using the jira integration user from the prior paragraph and click Close.
Optionally Configure FishEye Server to switch Integration setup to FishEye integration only,
12.1.8.2.1. FishEye entity mappings
Optionally link a FishEye repository to a JIRA project via a project link. For example select Projects → Develop- ment Production Line manual → Settings → Configure Project Links → Add Link → freedumbytes.dev.net FishEye and in the Enter link details form type:
• Type: FishEye Repository. • Key: dpl-manual and click Create.
After this there should be something to see under the FishEye/Crucible entity mappings in the FishEye Con- figuration. 12.1.8.3. FishEye plugin upgrade
At the FishEye Admin GUI http://freedumbytes.dev.net/fisheye/admin/ select the Systems Settings → Manage Add-ons to filter Action required and just Update or Update All.
Next at the JIRA Admin GUI Add-ons http://freedumbytes.dev.net/jira/plugins/servlet/upm#manage filter Man- age add-ons that have Action required and just Update or Update All. 12.1.9. User profile
On the Dashboard page click on the logged-in user at the top right of the menu bar and select Profile to edit for example:
Nexus is a Maven repository manager, created to provide reliable access to artifacts required for development and provisioning. 13.1.1. Resources
• Download Nexus. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: Nexus - HTTP Binding application-host / application-port / nexus-work and Nexus - Windows service wrapper.conf / nexus.bat install.
Note
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe P:\dev\apps\re- po\nexus-2.11.1-01\conf P:\dev\apps\repo\nexus-2.11.4-01\conf to copy the changes in for example nexus.properties,
Note
To reset Nexus Logging Configuration to the latest changes delete them from P:\dev \data\repo\maven\nexus\conf P:\dev\apps\repo\nexus-2.11.4-01\conf to manually copy those changes in for example logback.properties, log- back-nexus.xml and logback-overrides.xml,
Note
And run p:\dev\apps\editor\winmerge\WinMergeU.exe P: \dev\apps\repo\nexus-2.11.1-01\bin\jsw\conf P:\dev\apps\re- po\nexus-2.11.4-01\bin\jsw\conf for wrapper.conf.
• Books such as Repository Management with Nexus. 13.1.2. Nexus installation guide
Download the archive: nexus-2.11.4-01-bundle.zip [version 2.11.4-01].
Extract this .zip file to P:\dev\apps\repo.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add the New system variables NEXUS_HOME and PLEXUS_NEXUS_WORK pointing to P:\dev\apps \repo\nexus-2.11.4-01 and P:\dev\data\repo\maven\nexus. 13.1.2.1. HTTP Binding
Configure the container for Nexus - in the Plexus property file P:\dev\apps\re- po\nexus-2.11.4-01\conf\nexus.properties:
# Sonatype Nexus # ======# This is the most basic configuration of Nexus.
To restart automatically on Microsoft Windows, create a Windows service. Use the Tanuki Java Service Wrapper shipped with Nexus.
Edit the following entries - in the service wrapper configuration file P:\dev\apps\re- po\nexus-2.11.4-01\bin\jsw\conf\wrapper.conf:
# Set the JVM executable # (modify this to absolute path if you need a Java that is not on the OS path) wrapper.java.command=javaP:\dev\apps\prg\java-x64\jdk1.8.0_162\bin\java
# Additional JVM parameters (tune if needed, but match the sequence of numbers!) # wrapper.java.additional.1=-XX:MaxPermSize=192m wrapper.java.additional.21 =-Djava.io.tmpdir=./tmp # wrapper.java.additional.3=-Djava.net.preferIPv4Stack=true wrapper.java.additional.42 =-Dcom.sun.jndi.ldap.connect.pool.protocol="plain ssl" wrapper.java.additional.42 .stripquotes=TRUE #wrapper.java.additional.5=-Xdebug #wrapper.java.additional.6=-Xnoagent #wrapper.java.additional.7=-Djava.compiler=NONE #wrapper.java.additional.8=- Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 #wrapper.java.additional.9=-XX:+HeapDumpOnOutOfMemoryError wrapper.java.additional.3=-Xms128m wrapper.java.additional.4=-Xmx1024m #wrapper.java.additional.5=-Xrs see also Section 3.1.3.1, “Standard Edition Tools Reference”
…
# Size Java memory, in MB (-Xms) # wrapper.java.initmemory=256 # Size Java memory, in MB (-Xmx). # This option only supports a setting up to 4000 (4Gb). # If you need more, comment this option out # and use an explicit wrapper.java.additional option with -Xmx # wrapper.java.maxmemory=768
…
# Set up JSW Console wrapper.console.title=Sonatype Nexusfreedumbytes.dev.net Nexus …
# Set up JSW as NT Service (unused on other OSes) wrapper.ntservice.name=nexus-webappnexus wrapper.ntservice.displayname=freedumbytes.dev.net Nexus wrapper.ntservice.description=Nexus is a Maven repository manager.
Register the Nexus service with the command in an elevated command processor: %NEXUS_HOME%\bin \nexus.bat install. Start Nexus with net start nexus. A nexus.xml file should now be created in the P:\dev\data\repo\maven\nexus\conf folder. Verify that the server is running in your browser at http://lo- calhost:8067/nexus.
Note
In case of an upgrade sc delete nexus first (see also Section 1.1.5, “Task Manager replace- ment” in case of “The specified service has been marked for deletion.”).
Important
In case of a new jdk installation use an elevated command prompt just run net stop nexus and net start nexus.
Verify the installation and the JVM memory usage at http://localhost:8067/nexus/#supporttools.
104 Repository Manager
13.1.3.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Nexus and select Properties → Log On to use This ac- count .\Tomcat. After password conformation click Apply and restart the service. 13.1.4. Apache configuration
To proxy Nexus add another Virtual Host on port 81 using the following P:\dev\apps\httpserver\apache- conf\httpd-vhosts.conf include :
…
…
…
# Nexus Include ../apache-conf/httpd-nexus.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-nexus.conf for anonymous access:
When the Apache HTTP Server is restarted you should be able to browse Nexus at http://freedumbytes.dev.net/ nexus. 13.1.5. Nexus configuration 13.1.5.1. Users
Log In with the admin and default password admin123. Select Security → Users to change the Email addresses of the admin, anonymous and deployment users to for example [email protected].
Right click on the admin and deployment user to Set Password and Log Out.
• User ID: jjasper. • First Name: Jene. • Last Name: Jasper. • Email: [email protected]. • Status: Active. • New Password and Confirm Password. • Role Management: Add and for example check Nexus Administrator Role and click OK followed by Save. 13.1.5.2. Server Administration
Log In with the admin and the new password. Select Administration → Server.
• Base URL: http://freedumbytes.dev.net/nexus and click Save. 13.1.5.3. Capabilities
Optionally select Administration → Capabilities (see also Configuring Repository Health Check and Accessing and Configuring Capabilities).
Configure the Health Check Configuration → Settings:
• Enabled: Checked. • Configure for all proxy repositories: Checked. 13.1.5.4. Repositories
Activate downloading of the remote indexes for the proxy repositories that Nexus ships. Select Views/Reposi- tories → Repositories and repeat the following step for Central, Apache Snapshots and Codehaus Snapshots:
• Left click on the repository and switch to the Configuration tab to set Download Remote Indexes: True and click Save.
Select Administration → Scheduled Tasks → Refresh to check the Status of the Repair Repositories Index task (see also NEXUS-431), that was started automatically. When the task is already finished it is possible to view the information in Views/Repositories → System Feeds → System changes in Nexus.
When the repository indexing is finished it is possible to use Artifact Search and actually find something.
Important
All Codehaus services have been terminated.
13.1.5.4.1. JBoss
Optionally configure the JBoss repository (see also JBBUILD-688) by selecting Views/Repositories → Reposito- ries → Add… → Proxy Repository with:
Select Views/Repositories → Repositories → Refresh → Public Repositories and on the Configuration tab drag JBoss from the Available Repositories somewhere below Central in Ordered Group Repositories and click Save.
Then drag JBoss 3rd party from the Available Repositories just below JBoss in Ordered Group Repositories and click Save. 13.1.5.4.2. Spring
Optionally configure the SpringSource Enterprise Repositories by selecting Views/Repositories → Repositories → Add… → Proxy Repository with:
Select Views/Repositories → Repositories → Refresh → Public Repositories and on the Configuration tab drag SpringSource EBR - SpringSource Bundle Releases from the Available Repositories somewhere below Central in Ordered Group Repositories.
Then drag SpringSource EBR - External Bundle Releases from the Available Repositories just below SpringSource EBR - SpringSource Bundle Releases in Ordered Group Repositories.
Finally drag SpringSource EBR – External Bundle Milestones from the Available Repositories just below SpringSource EBR - External Bundle Releases in Ordered Group Repositories and click Save. 13.1.5.5. Scheduled tasks
When the below mentioned tasks are configured it is possible to start extra unscheduled runs with Administra- tion → Scheduled Tasks, right clicking on the desired task and selecting Run. Follow the task progress in Views/ Repositories → System Feeds → System changes in Nexus → Refresh.
13.1.5.5.1. Indexing
Configure a task to publish hosted releases index by selecting Administration → Scheduled Tasks → Add with:
Configure Maven to use the Nexus Repository with the following mirror setting in the Maven configuration file P:\dev\apps\build\apache-maven-3.5.4\conf\settings.xml to correspond with in the Views/Repos- itories → Repositories available Repository Path of the Public Repositories:
Configure Maven to use the Nexus Public Repositories by default (see also * mirrorOf ) with the follow- ing profile and activateProfile settings in the Maven configuration file P:\dev\apps\build\apache- maven-3.5.4\conf\settings.xml:
development central.dev.net http://freedumbytes.dev.net/nexus/content/groups/public truetruealwayscentral.dev.net http://freedumbytes.dev.net/nexus/content/groups/public truetruealways
development
Important
The repositories defined in the development profile have releases as well as snapshots en- abled because the Views/Repositories → Repositories for the Public Repositories contains the hosted Releases (internal releases), 3rd party and Central as well as the hosted Snapshots (internal snapshots). The updatePolicy is set to always to make sure that from the hosted internal snapshot repository the latest version is used.
For an existing pom.xml run from the command line mvn help:active-profiles to verify these settings. 13.1.6.2. Artifact deployment
Configure Maven projects to use the LAN Nexus Repository for deployments with the following distributionMan- agement repository settings in the Maven project pom.xml to correspond with in the Views/Repositories → Repositories available Repository Path of the hosted Releases and Snapshots repositories:
Configure Maven to use the Nexus Repository deployment user with the following server settings LAN and Open Source in the Maven configuration file P:\dev\apps\build\apache-maven-3.5.4\conf \settings.xml:
It is possible to encrypt the password using Section 4.2.5, “Password Encryption”.
Important
In case of partial deployments, with the maven release plugin for instance, the retries will get HTTP 400 errors. Therefore select Views/Repositories → Repositories → Releases → Con- figuration and change the Deployment Policy to Allow Redeploy.
13.1.6.3. Third-party artifacts
As an alternative to adding a missing third-party artifact from the command line with:
select Views/Repositories → Repositories → 3rd party and on the Artifact Upload tab configure the Group, Artifact and Version (GAV) settings for the third-party artifact.
In a nutshell, Jenkins (formerly known as Hudson) provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity. 14.1.1. Resources
• Download Jenkins and plugins. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: Jenkins - Installation guide Tomcat webapps. • Jenkins Documentation such as Plugins, Pipeline (Best Practices / Examples), Blue Ocean (#blueocean), Secu- rity (#security), Handbook, Blog, Extend and Containers (Tomcat). • Combining Jenkins and Docker together can bring improved speed and consistency to your automation tasks. • Jenkins Features Controlled with System Properties. • Quick overview of Continuous Integration by Martin Fowler. • Jenkins CI: The Origins of Butlers, Build Masters and Bowties. 14.1.2. Jenkins installation guide
Download the latest archive: jenkins.war [version 2.64 or LTS 2.46.3].
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add a New system variable JENKINS_HOME pointing to P:\dev\data\ci\jenkins. You need to reboot the server for the Path setting to take effect.
Deploy the .war in Tomcat by dropping it into the directory P:\dev\apps\webserver\apache- tomcat-8.0.26\webapps.
Note
Stop Tomcat service when Jenkins doesn't restart properly because the P:\dev\apps\web- server\apache-tomcat-8.0.26\webapps\jenkins could not be cleaned completely. Af- ter Tomcat is stopped just delete the folder webapps\jenkins and restart Tomcat service.
14.1.2.1. Unlock Jenkins
Unlock Jenkins at http://localhost:8068/jenkins with the Administrator password from %JENKINS_HOME%\se- crets\initialAdminPassword.
Next install suggested plugins:
• CloudBees Folders allows users to create folders to organize jobs. Users can define custom taxonomies (like by project type, organization type etc). Folders are nestable and you can define views within folders. • OWASP Markup Formatter uses policy definitions to allow limited HTML markup in user-submitted text. • Build-timeout allows you to automatically abort a build if it's taking too long. Once the timeout is reached, Jenkins behaves as if an invisible hand has clicked the abort build button. • Credentials Binding allows credentials to be bound to environment variables for use from miscellaneous build steps. • Timestamper adds timestamps to the Console Output with the option to switch to between System clock time and Elapsed time. • Workspace Cleanup deletes the workspace before the build or when a build is finished and artifacts saved. • Ant adds Apache Ant support to Jenkins. • Gradle makes it possible to invoke a Gradle build script as the main build step. • Pipeline is a suite of plugins that lets you orchestrate automation, simple or complex. • GitHub Organization Folder is Pipeline-as-Code support for a whole GitHub organization. Scans all the branches & repositories in GitHub organization and build them via Jenkins pipelines automatically. • Pipeline Stage View includes an extended visualization of Pipeline build history on the index page of a flow project, under Stage View. • Git allows use of Git as a build SCM, including repository browsers for several providers. A recent Git runtime is required. • Subversion adds the Subversion support (via SVNKit) to Jenkins. • SSH Slaves allows you to manage slaves running on *nix machines over SSH. • Matrix Authorization Strategy offers matrix-based security authorization strategies (global and per-project). • PAM Authentication adds Unix Pluggable Authentication Module (PAM) support to Jenkins.
111 Chapter 14.
• LDAP provides yet another way of authenticating users using LDAP. It can be used with LDAP servers like Active Directory or OpenLDAP among others. • Email Extension allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says. • Mailer allows you to configure email notifications for build results. This is a break-out of the original core based email component.
Or select additional plugins to install:
• Dashboard View provides a portal-like view for the Jenkins instance. • Windows Slaves allows you to connect to Windows machines and start slave agents on them. 14.1.2.2. Access
Create first Admin User: jjasper.
Select Jenkins → Manage Jenkins → Configure Global Security or type config in the search box to set:
• Enable security and configure Access Control: • Security Realm: Delegate to servlet container. • Authorization: Logged-in users can do anything. • Markup Formatter: Safe HTML. • Prevent Cross Site Request Forgery exploits with Crumb Algorithm: Default Crumb Issuer (see also Section 8.1.5.1, “Remote authentication”) and click Save. 14.1.2.3. Maven integration TODO
When using Maven with password encryption it will always look for the file settings-security.xml in the fixed location %USERPROFILE%\.m2. Therefore make sure to create one for the Log On Windows user with a copy of for example C:\Users\jjasper\.m2\settings-security.xml into C:\Users\tomcat\.m2.
Important
This still may not be enough when using for example truezip-maven-plugin which fails with “IOException: Unable to create P:\dev\data\ci\jenkins\jobs\…\${project.artifactId}- ${project.version}.war\extras” trying to copy additional files into a war. To fix this right- click on the directory P:\dev\data\ci\jenkins\jobs and select Properties → Security → Edit… → Users (COMPUTERNAME\Users) to allow Modify and thus Write and click OK.
14.1.3. Apache configuration
Instruct Apache to proxy all URLs whose path portions begin with /jenkins/ using the following P:\dev\apps \httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# Jenkins Include ../apache-conf/httpd-jenkins.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-jenkins.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf
112 Continuous Integration
Note
The entries at are required to suppress the message “It appears that your reverse proxy set up is broken.” on the Manage Jenkins page.
When the Apache HTTP Server is restarted you should be able to browse Jenkins at http://freedumbytes.dev.net/ jenkins.
TODO Unprotected URLs These URLs (and URLs starting with these prefixes plus a /) should require no authen- tication. If possible, configure your container to pass these requests straight to Jenkins without requiring login. assets cli git github-webhook instance-identity jnlpJars subversion whoAmI 14.1.4. Jenkins configuration 14.1.4.1. Plugins
Select Jenkins → Manage Jenkins → Manage Plugins and click Check now to find updates. When done switch to the Available tab to check (one-at-a-time works best) for example the following plugins (the other project information will be handled with Chapter 17, Quality Assurance):
• Blue Ocean provides a new user experience for Jenkins. • JIRA integration (see also Section 14.1.5, “JIRA integration”).
After successful Install of all selected plugins reload Jenkins at http://freedumbytes.dev.net/tomcat/manag- er/html for the latest version of the plugin to show up in the Installed tab.
Note
To manual install the plugin when Jenkins is deployed on Tomcat it is also possible to just copy the .hpi file to P:\dev\data\ci\jenkins\plugins.
14.1.4.2. System TODO obsolete
Select Jenkins → Manage Jenkins → Configure System or type config in the search box to set:
• # of executors: 6 (where a good value to start with would be the number of processors on your system). • Quiet period: 30. • SCM checkout retry count: 3. • Add Ant installation: • Name: ANT_HOME. • Disable Install automatically. • ANT_HOME: P:\dev\apps\build\apache-ant-1.10.4. • Add Maven installation: • Name: M2_HOME. • Disable Install automatically. • MAVEN_HOME: P:\dev\apps\build\apache-maven-3.5.4. • Maven Project Configuration: • Optionally disable Help make Jenkins better by sending anonymous usage statistics and crash reports to the Jenkins project. • Jenkins Location: • Jenkins URL: http://freedumbytes.dev.net/jenkins/. • System Admin e-mail address: [email protected]. • E-mail Notification: • SMTP server: mail.freedumbytes.dev.net. • Default user e-mail suffix: @company.org and click Save. 14.1.4.3. Plugins TODO obsolete
Select Jenkins → Manage Jenkins → Manage Plugins. Switch to the Advanced tab and click Check now to find updates. When done Go back to update center and switch to the Available tab to check (one-at-a-time works best) for example the following plugins (the other project information will be handled with Chapter 17, Quality Assurance):
• Modern Status provides an alternative set of status and action icons to provide a fresh look to Jenkins and to be friendly for all users i.e. iconic not just color indication.
113 Chapter 14.
• Build Trigger Badge displays icon representing the cause of a build directly in the build history. It lets you quickly know which cause triggered a build. • JIRA integration (see also Section 14.1.5, “JIRA integration”).
After successful Install of all selected plugins reload Jenkins at http://freedumbytes.dev.net/tomcat/manag- er/html for the latest version of the plugin to show up in the Installed tab.
Note
To manual install the plugin when Jenkins is deployed on Tomcat it is also possible to just copy the .hpi file to P:\dev\data\ci\jenkins\plugins.
14.1.4.4. Subversion authentication
Setup Subversion authentication in Jenkins at http://freedumbytes.dev.net/jenkins/scm/SubversionSCM/enter- Credential:
When creating a new job for the Subversion scm you would otherwise have been presented with a link in “Maybe you need to enter credential?” to supply them for the specified URL.
Or alternatively use credentials (see also Section 14.1.4.5, “Git authentication”):
14.1.4.5. Git authentication
For Git authentication by Jenkins look at _netrc setup in Section 8.1.3.1.1, “netrc” or select Jenkins → Manage Jenkins → Manage Credentials and Add Credentials → Username with password:
• Scope: Global. • Username: jenkins. • Password: password. • Description: apache proxy user and click Save. 14.1.5. JIRA integration
JIRA Plugin for Jenkins and Jenkins Plugin for JIRA integrate Jenkins and Hudson/Jenkins CI Servers with JIRA to display builds in JIRA. 14.1.5.1. Jenkins Plugin installation guide
Download the archive: jenkins-jira-plugin-1.4.5.hpi [version 1.4.5].
Just drop the downloaded *.hpi or *.jpi file into the P:\dev\data\ci\jenkins\plugins directory and rename to jenkins-jira-plugin.hpi. You will then need to restart Jenkins (many containers let you do this without restarting the container).
114 Continuous Integration
Next at the JIRA Admin GUI Add-ons http://freedumbytes.dev.net/jira/plugins/servlet/upm/marketplace/fea- tured Find new add-ons with Search the Marketplace for jenkins and just Install.
Note
In case of an issue (see also JJI-134) with the latest version download an older *.obr version and select Add-ons → Manage add-ons → Upload add-on to install this prior version.
14.1.5.2. Application links
At the JIRA Admin GUI Add-ons http://freedumbytes.dev.net/jira/plugins/servlet/applinks/listApplicationLinks Create new link: http://freedumbytes.dev.net/jenkins. In pop-up Configure Application URL click Con- tinue to configure:
Edit this new Application Link freedumbytes.dev.net Jenkins to change:
• Outgoing Authentication to enable Basic Access reusing the jira integration user from the section Sec- tion 12.1.8.1, “Integration users” and click Close.
Next at the JIRA Admin GUI Add-ons http://freedumbytes.dev.net/jira/secure/admin/ConfigureJenkinsIntegra- tion!default.jspa Jenkins Configuration select Actions to Auto enable new Jobs and enable the already listed jobs followed by Actions → Refresh Job List to Sync Jobs. 14.1.6. Slave configuration
A butler is a domestic worker in a large household. A valet is a servants who serve as personal attendants to their employer.
Select Jenkins → Manage Jenkins → Manage Nodes to add:
• New Node with: • Node name: jeeves. • As Permanent Agent and click OK.
Configure this new node as follows:
• # of executors: 6 (where a good value to start with would be the number of processors on your system). • Remote root directory: P:\dev\data\ci\jeeves. • Labels: windows. • Usage: Use this node as mush as possible. • Launch method: Launch agent via Java Web Start. • Availability: Keep this slave online as much as possible and click Save.
Select Jenkins → Build Executor Status → jeeves to connect the slave to Jenkins using Launch agent from browser on slave by clicking the orange Launch button. This will download the file slave-agent.jnlp. To install the slave jeeves run this file and supply proxy credentials:
• User name: jjasper. • Password: password. • Labels: windows. • Leave Save this password in your password list unchecked and click OK → Run → File → Install as a service.
Note
Don't forget to create the slave folder first with mkdir P:\dev\data\ci\jeeves.
Edit the generated id and name in P:\dev\data\ci\jeeves\jenkins-slave.xml to insert /tomcat :
jenkinsslave-P__dev_data_ci_jeeves Jenkins Slavefreedumbytes.dev.net Jeeves This service runs an agent for Jenkins automation server. P:\dev\apps\prg\java-x64\jdk1.8.0_162\bin\java.exe
To activate the changes reinstall the service with the following commands in an elevated command processor (see also Section 1.1.5, “Task Manager replacement” in case of “WMI.WmiException: ServiceMarkedForDeletion”):
cd /d P:\dev\data\ci\jeeves jenkins-slave.exe stop jenkins-slave.exe uninstall TODO before changes… del *.log jenkins-slave.exe install jenkins-slave.exe start services.msc
Important
In case of a new jdk installation use an elevated command prompt just run net stop jeeves and net start jeeves.
Note
The jdk upgrade for Jenkins master is handled when the upgrade is done for Tomcat.
Verify the installation and the JVM memory usage at http://freedumbytes.dev.net/jenkins/systemInfo and http:// freedumbytes.dev.net/jenkins/computer/jeeves/systemInfo.
Next fix the following error in P:\dev\data\ci\jeeves\jenkins-slave.err.log:
Failing to obtain http://freedumbytes.dev.net/jenkins/computer/jeeves/slave- agent.jnlp?encrypt=true java.io.IOException: Failed to load …: 401 Unauthorized at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:275) at hudson.remoting.Launcher.run(Launcher.java:219) at hudson.remoting.Launcher.main(Launcher.java:192) Waiting 10 seconds before retry
Which is also displayed in http://freedumbytes.dev.net/jenkins/computer/jeeves/ node page as “Connection was broken” and P:/dev/logs/httpserver/freedumbytes.dev.net-access.log:
Just allow anonymous access for the local area network by editing P:\dev\apps\httpserver\apache-conf \httpd-jenkins.conf:
…
Include ../apache-conf/httpd-freedumbytes-realm.conf
Include ../apache-conf/httpd-lan-access.conf
116 Continuous Integration
Include ../apache-conf/httpd-lan-access.conf
After restarting jeeves fix the next error in P:\dev\data\ci\jeeves\jenkins-slave.err.log:
May 07, 2016 1:25:28 AM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: jeeves May 07, 2016 1:25:28 AM hudson.remoting.jnlp.Main$CuiListener INFO: Jenkins agent is running in headless mode. May 07, 2016 1:25:28 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://freedumbytes.dev.net/jenkins/] May 07, 2016 1:25:28 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: http://freedumbytes.dev.net/jenkins/tcpSlaveAgentListener/ is invalid: 401 Unauthorized java.lang.Exception: …/tcpSlaveAgentListener/ is invalid: 401 Unauthorized at hudson.remoting.Engine.run(Engine.java:215)
See also P:/dev/logs/httpserver/freedumbytes.dev.net-access.log:
Again allow anonymous access for the local area network by editing P:\dev\apps\httpserver\apache-conf \httpd-jenkins.conf:
…
Include ../apache-conf/httpd-lan-access.conf
Include ../apache-conf/httpd-lan-access.conf TODO disable to download jnlp
Now after restarting jeeves check P:\dev\data\ci\jeeves\jenkins-slave.err.log:
May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: jeeves May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener INFO: Jenkins agent is running in headless mode. May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://freedumbytes.dev.net/jenkins/] May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to freedumbytes.dev.net:49403 May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect May 07, 2016 1:38:02 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected
See also P:/dev/logs/httpserver/freedumbytes.dev.net-access.log:
• master → Configure with: • # of executors: 2. • Labels: master. • Usage: Only build jobs with label restrictions matching this node and click Save.
Historically, Jenkins master and slaves behaved as if they altogether form a single distributed process. This means a slave can ask a master to do just about anything within the confinement of the operating system, such as access- ing files on the master or trigger other jobs on Jenkins. Should you use Slave To Master Access Control Jenkins → Manage Jenkins → Configure Global Security and check Enable Slave → Master Access Control. 14.1.6.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net Jeeves and select Properties → Log On to use This ac- count .\Tomcat. After password conformation click Apply and restart the service.
118 Chapter 15. Chapter 15. Integrated Development Environment 15.1. Eclipse
Eclipse is an open source community whose projects are focused on building an open development platform comprised of extensible frameworks, tools and runtimes for building, deploying and managing software across the lifecycle. 15.1.1. Resources
• Download or Upgrading Eclipse . • Upgrade checklist: • Related products: Plugins installation & configuration guide. • References: -. • Integration configuration changes: Plugins Eclipse - installation guide vm / javaagent lombok / eclipse.exe -initialize / osgi.instance.area.default, Eclipse - Configuration and Eclipse - encoding. • Eclipse Tutorials, Articles and plugins: EGit, JGit, Mylyn, Subclipse, M2Eclipse. • Git version control with Eclipse (EGit). • Developing with Eclipse & Maven. • Unicode/UTF-8 in your Eclipse Java projects. ResourceBundle Editor plugin is for editing Java resource bundles. It lets you manage all localized .properties files in one screen. Some of its features include sorted keys, warning icons on missing keys/values, conversion to/from Unicode, hierarchical view of keys, and more. UTF-8 encoding and Spring message sources. 15.1.2. Eclipse installation guide
Download the archive: eclipse-jee-mars-2-win32-x86_64.zip [version 4.5.2].
Extract this .zip file to P:\dev\apps\ide.
Now rename P:\dev\apps\ide\eclipse to P:\dev\apps\ide\eclipse-4.5.2.
Configure the vm settings (and windows processes) - in P:\dev\apps\ide \eclipse-4.5.2\eclipse.ini:
For ease of use create a shortcut on the Quick Launch Toolbar by dragging eclipse.exe from P:\dev\apps \ide\eclipse-4.5.2 onto it. Click this shortcut to start Eclipse with this now default Workspace at C:\work- space-4.5.2.
15.1.3. Eclipse upgrading
Figure 15.1. Eclipse Updates Available
Tip
In case of a newer version of the Eclipse IDE itself create copies of the prior one to for example P:\dev\apps\ide\eclipse-4.5.2 and also its workspace to C:\workspace-4.5.2. Edit any shortcut on the Quick Launch Toolbar according. Click this shortcut to start Eclipse and switch to the new default Workspace at C:\workspace-4.5.2.
Figure 15.2. Eclipse prior version
Click on the Updates Available popup or:
• Select Help → Check for Updates:
120 Integrated Development Environment
Figure 15.3. Eclipse Available Updates
• Check the ones that must be installed and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Note
A second restart of Eclipse is necessary to show to latest splash screen:
Figure 15.4. Eclipse latest version
Note
To revert an installation select Help → About Eclipse → Installation Details → Installation History and move from Current Installation down and click Revert.
121 Chapter 15.
Figure 15.5. Eclipse Installation History
15.1.4. Eclipse configuration
Set the following Code Style preferences at Window → Preferences → Java → Code Style:
• Select Clean Up and click Edit for Active profile: Eclipse [built-in] to roll your own Profile name freedumbytes.dev.net. Export those Clean Up settings as eclipse.codestyle.cleanup.xml. • Select Formatter and click Edit for Active profile: Eclipse [built-in] to roll your own Profile name freedumbytes.dev.net. Export the Formatter settings as eclipse.codestyle.formatter.xml.
122 Integrated Development Environment
Tip
To turn the formatter off and on use @formatter:off and @formatter:on.
• Select Organize Imports and click New for the company package net.dev.freedumbytes and nl.demon.shadowland. Export the Organize Import settings as eclipse.importorder. And change/leave both the number of imports needed for * to 99.
How to add static imports for code completion:
1. First verify at Window → Preferences → Java → Editor → Content Assist if the defaults are still checked for Add import instead of qualified name and Use static imports (only 1.5 or higher).
2. Second add the desired list of New Type… to Window → Preferences → Java → Editor → Content Assist → Favorites: • org.assertj.core.api.Assertions • org.hamcrest.MatcherAssert • org.hamcrest.Matchers • org.junit.Assert • org.mockito.Matchers • org.mockito.Mockito
By the way the shortcuts for formatting source and organizing imports are Ctrl+Shift+F and Ctrl+Shift+O.
123 Chapter 15.
Tip
To activate formatting and import organizing when the file is saved select Window → Prefer- ences → Java → Editor → Save Actions and check Perform the selected actions on save for Format source code → Format edited lines and Organize imports.
Important
Unless you are going to use Lombok then select Format source code → Format all lines (and just use aforementioned @formatter) to avoid “ArrayIndexOutOfBoundsException: -1” during CleanUpPostSaveListener on Ctrl+S.
Caution
Do not run the Additional actions because it can be expensive and slow down the workbench and you might want to preview those changes before accepting them (see also Keep your code clean with Eclipse).
To use the same settings for the other editors select Window → Preferences and type filter text “indent”:
• Editor for CSS, HTML and XML: • Line width 256. • Indent using spaces. • Indentation size 2.
To disable auto folding at Window → Preferences → Java → Editor → Folding and uncheck all elements under Initially fold these elements.
Change code completion in Window → Preferences → Java → Editor → Content Assist to Completion over- writes.
Optionally select Window → Preferences and type filter text “typing” and uncheck Tab key adjusts the inden- tation of the current line for Editor for Java and JavaScript. Don't forget to Apply the changes.
Create a shortcut to remove trailing whitespace (useful for non-java files) at Window → Preferences → General → Keys with filter text trailing and select Binding and just click Ctrl+Alt+Backspace. Shouldn't Java files automatically remove trailing whitespace it is also possible to activate this at Window → Preferences → Java → Editor → Save Actions → Additional actions → Configure → Code Organizing → Remove trailing whitespace → All lines, but also see the caution mentioned above.
124 Integrated Development Environment
Figure 15.7. Eclipse Remove Trailing Whitespace
15.1.4.1. Perspectives
Set the following Perspective preferences:
• Left click on the icon Open Perspective to add the Java shortcut and the Team Synchronizing short- cut. • Right click on the icon Java to no longer Show Text. • Optionally right click on the icon Java EE to Close it.
Use Ctrl+F8 to cycle through the perspectives.
15.1.4.2. Views
Open the view Package Explorer with Alt+Shift+Q, P and left click on the icon local menu to make the following changes:
• Package Presentation → Flat. • Filters… and check Libraries from external. • Top Level Elements → Working Sets.
Tip
The shortcut for the local menu is Ctrl+F10 when the view is active. This is also the shortcut for the ruler menu in editors.
Cycle through the views to Outline using Ctrl+F7. This view can be kept closed (unless you are going to use Lombok) because Ctrl+O shows the outline of the current source when needed. And Ctrl+F3 shows the outline at the current cursor position.
125 Chapter 15.
Tip
Instead of clicking through the view Package Explorer just open the resources with Ctrl+Shift+R or the java types with Ctrl+Shift+T. Now in the editor click Alt+Shift+W and se- lect Package Explorer to locate it in that view, should you be interested in the files at the same location.
The view Console can be opened with Alt+Shift+Q, C. 15.1.4.3. Editors
Useful general editor shortcuts:
• Use F12 to jump back to the last active editor from any view and Ctrl+F6 to cycle through the editors. • With Alt+Left and Alt+Right it is possible to move through the file history. • Goto line with Ctrl+L. • Search in files with Incremental Find Ctrl+J and typing the search text. Find next occurrence with Ctrl+K and the prior one with Ctrl+Shift+K. • Search in workspace with Ctrl+H.
Tip
Because the Java shortcut for finding references is easier to use (see also Java editor short- cuts [126] below) Customize… the Search popup by unchecking Java Search, JavaScript Search, Plug-in Search and Remote Search (that option will still be available through the Search menu).
When working in Ctrl+M maximized mode and a search result is selected this resource is also opened in max- imized mode. Simply switch back to the search view with Ctrl+F7. Use Delete to remove search results from the tree. Navigate to the prior search result with Ctrl+, and the next search result with Ctrl+..
Tip
To exclude unused working sets or projects in the workspace from search results and other views just Close Project or the complete working set in the view Package Explorer.
Note
When encountering the following error “An internal error occurred during: Items filtering. Class file name must end with .class” during search, close Eclipse: • Delete /.metadata/.plugins/org.eclipse.jdt.core/*.index. • Delete /.metadata/.plugins/org.eclipse.jdt.core/ savedIndexNames.txt.
Useful Java editor shortcuts:
• Goto declaration with F3. • Find references in workspace with Ctrl+Shift+G. • Find declarations in workspace with Ctrl+G. • Popup type hierarchy with Ctrl+T. • Change case to lower Ctrl+Shift+Y or upper Ctrl+Shift+X.
Spell checking is supported by Eclipse and when you see a squiggly mark place the cursor on the word and press Ctrl+1 to view the correction proposals.
Should a plugin Update Site only list an earlier version try the above described Eclipse upgrad- ing option. This resulted in the latest version for BIRT (see also Section 15.1.5.8, “Memory An- alyzer”).
It is also possible to search online at http://marketplace.eclipse.org/ and drag to install:
Figure 15.8. Eclipse Marketplace Drag to Install
15.1.5.1. Lombok
Project Lombok (features) to spice up your java by Reducing Boilerplate Code.
Download the archive: lombok.jar [version 1.16.21].
Copy this .jar file to P:\dev\apps\ide. Edit in P:\dev\apps\ide\eclipse-4.5.2\eclipse.ini as follows:
Verify installation with Help → About Eclipse which should list Lombok now also.
Tip
When using Lombok the getters and setters are no longer available in the code. To be able to find references in the workspace (Ctrl+Shift+G) you are going to need the Outline view. To save space just drag it next to the Task List tab.
15.1.5.2. EGit
EGit [version 4.3.0] is an Eclipse Team provider for the Git version control system. Git is a distributed SCM, which means every developer has a full copy of all history of every revision of the code, making queries against the history very fast and versatile.
JGit [version 4.3.0] is the Java implementation of Git. It is a library, that also can be used in your own applications. It also provides some sort of CLI operations. EGit on the other side is the Eclipse team provider plugin for Git, which uses JGit as Git implementation. Simplified you could say EGit is the UI part, and JGit the background part. JGit doesn't depend on EGit, but EGit does depend on JGit.
Install the plugin EGit in Eclipse:
127 Chapter 15.
• Select Help → Install New Software…. • Add… the Location http://download.eclipse.org/egit/updates/ with Name EGit P2 Reposi- tory and click Ok. • Select to Work with: EGit Update Site - http://download.eclipse.org/egit/updates/. • Select the following components: • Eclipse Git Team Provider [4.3.0] • Eclipse Git Team Provider - Task focused interface [4.3.0] • Command Line Interface for Java implementation of Git [4.3.0] • Java implementation of Git [4.3.0] • Java implementation of Git - optional Http support using Apache httpclient [4.3.0] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Important
Should the Commit Message in the Git Staging view not match with the curent Mylyn Task just copy its details:
Figure 15.9. Eclipse Mylyn Task List view
after that paste the correct details:
Figure 15.10. Eclipse EGit Staging view
128 Integrated Development Environment
15.1.5.2.1. EGit configuration
Set the following preferences at Window → Preferences:
• Select Team → Git and as Default Repository folder enter C:\dev\projects and click OK.
15.1.5.3. Subclipse
Subclipse [version 1.10.13] is an Eclipse Team Provider plug-in providing support for Subversion within the Eclipse IDE.
Install the plugin Subclipse in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://subclipse.tigris.org/update_1.10.x/ with Name Subclipse 1.10.x Update Site and click Ok. • Select to Work with: Subclipse 1.10.x Update Site - http://subclipse.tigris.org/update_1.10.x/. • Select the following components: • CollabNet Merge Client [4.1.0] • Subclipse (Required) [1.10.13] • Subclipse Integration for Mylyn 3.x (Optional) [3.0.0] • Subversion Client Adapter (Required) [1.10.3] • Subversion JavaHL Native Library Adapter [1.8.15] • Subversion Revision Graph [1.1.1] • SVNKit Library [1.8.12] • JNA Library [4.1.0] • SVNKit Client Adapter (Not required) [1.8.9] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Note
Optionally select Window → Preferences → Team → SVN → Usage Reporting to uncheck Al- low the Subclipse team to receive anonymous usage statistics for this Eclipse installation.
15.1.5.3.1. Subclipse configuration
Set the following preferences at Window → Preferences:
• Select Team → SVN and as SVN Interface Client choose SVNKit (Pure Java) and click OK.
15.1.5.4. Mylyn
Mylyn [version 3.19.0] is a task-focused interface for Eclipse that reduces information overload and makes multi-tasking easy. It does this by making tasks a first class part of Eclipse, and integrating rich and offline editing for repositories such as Bugzilla, Trac, and JIRA. Once your tasks are integrated, Mylyn monitors your work activity to identify relevant information, and uses this task context to focus the user interface on the task-at-hand. This puts the information you need at your fingertips and improves productivity by reducing searching, scrolling, and navigation. By making task context explicit Mylyn also facilitates multitasking, planning, reusing past efforts, and sharing expertise.
Update the plugin Mylyn in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://download.eclipse.org/mylyn/releases/latest with Name Mylyn for Eclipse and click Ok. • Select to Work with: Mylyn for Eclipse - http://download.eclipse.org/mylyn/releases/latest. • Select the following components (uncheck Hide items that are already installed): • Mylyn Task List [3.19.0] • Mylyn Task-Focused Interface [3.19.0] • Mylyn WikiText [2.8.0] • Mylyn Builds Connector: Hudson/Jenkins [1.11.0] • Mylyn Context Connector: Eclipse IDE [3.19.0] • Mylyn Context Connector: Java Development [3.19.0] • Mylyn Context Connector: Plug-in Development [3.19.0] • Mylyn Context Connector: Team Support [3.19.0] • Mylyn Versions Connector: Subclipse [1.11.0] • Mylyn Versions Connector: Git [1.11.0] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse. 129 Chapter 15.
For JIRA support:
• Select Help → Install New Software…. • Add… the Location http://update.atlassian.com/atlassian-eclipse-plugin/rest/e3.7/ with Name Atlassian Connector for Eclipse and click Ok. • Select to Work with: Atlassian Connector for Eclipse - http://update.atlassian.com/atlassian-eclipse-plu- gin/rest/e3.7/. • Select the following component: • Atlassian Connector for Eclipse [3.2.5] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Note
Optionally select Window → Preferences → Atlassian Connector → Usage Data to uncheck Enable monitoring and automatic submission.
Warning
We are discontinuing the support for Atlassian IDE Connectors (source and fork).
15.1.5.4.1. Mylyn configuration
Set the following Window → Preferences → Mylyn:
• In Tasks define: • Change Week Start to Monday. • Enable Track time spent within Eclipse when a task is active. • Optionally set Stop time accumulation after 15 minutes of inactivity. • In Team define Commit Comment Template:
In Java perspective activate the Mylyn views with Window → Show View → Other… or shortcut Alt+Shift+Q, Q and from Mylyn select: Task List, Task Repositories and Builds.
Open the Task List view with Alt+Shift+Q, K (see also the User Guide for shortcuts and a UI Legend, which is also available in the local menu → Show UI Legend).
15.1.5.4.2. JIRA integration
In the Task Repositories view open the local menu with Ctrl+F10 to Add Task Repository…. Select JIRA and click Next:
• Server: http://freedumbytes.dev.net/jira. • Label: freedumbytes.dev.net JIRA. • User ID: jjasper. • Password: password. • Open Additional Settings: • Subtasks: Show linked tasks. • Advanced Configuration: • Date Picker Format: yyyy-MM-dd. • Date Time Picker Format: yyyy-MM-dd HH:mm. • Open Task Editor Settings: Confluence (default). • Validate Settings and click Finish and do not create a query yet.
Add queries to the freedumbytes.dev.net JIRA task repository by right clicking on it and selecting New Query…. Select Create query using form and click Next:
• Query Title: freedumbytes.dev.net JIRA issues. 130 Integrated Development Environment
• Assigned To: Current User and click Finish.
Note
The other issues, that are viewed in Eclipse, will show up in the category Uncategorized.
15.1.5.4.3. Jenkins integration
In the Builds view select Add Build Server…. Select Hudson (supports Jenkins) and click Next:
• Server: http://freedumbytes.dev.net/jenkins. • Label: freedumbytes.dev.net Jenkins builds. • Keep: Anonymous checked. • Open Http Authentication and check Enable Http Authentication: • User ID: jjasper. • Password: password. • Validate and click Refresh for Build Plans. • Click Select All in Build Plans and click Finish.
Set the following Window → Preferences:
• In Mylyn → Builds check Automatically refresh builds and click Apply.
15.1.5.5. Maven
Maven integration [version 1.6.2] for Eclipse provides tight integration for Maven into the IDE.
Install the plugin M2Eclipse in Eclipse:
• Select Help → Install New Software…. • Add… the L ocation http://download.eclipse.org/technology/m2e/releases/ with Name m2e Update Site and click Ok. • Select to Work with: m2e Update Site - http://download.eclipse.org/technology/m2e/releases/. • Type filter text: m2e. • Select the following components: • m2e - Maven Integration for Eclipse (includes Incubating components) [1.7.0] • m2e - slf4j over logback logging (Optional) [1.7.0] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Important
If you get the following JDK warning: “The Maven Integration requires that Eclipse be running in a JDK, because a number of Maven core plugins are using jars from the JDK. Please make sure the -vm option in eclipse.ini [119] is pointing to a JDK and verify that Installed JREs are also using JDK installs.”
Select Window → Preferences → Java → Installed JREs → Add… a Standard VM and Next set the JRE home with Directory… to P:\dev\apps\prg\java-x64\jdk1.8.0_162 and click Finish. Now check this jdk1.8.0_162 and Remove the old jre1.x.x_xx and apply with OK (see also Section 15.1.2, “Eclipse installation guide” [119]).
Caution
After upgrading maven-jar-plugin to 3.x Eclipse reports the following error “org.apache.maven.archiver.MavenArchiver.getManifest(org.apache.maven.project.MavenProject, org.apache.maven.archiver.MavenArchiveConfiguration)”
• Select Help → Install New Software…. • Add… the L ocation https://otto.takari.io/content/sites/m2e.extras/ m2eclipse-mavenarchiver/0.17.2/N/LATEST/ and click Ok. • Select the following components: • m2e connector for mavenarchiver pom properties [0.17.2] and click Next twice.
131 Chapter 15.
15.1.5.5.1. Maven configuration
Set the following preferences Window → Preferences:
• In Team → Ignored Resources → Add Pattern… for target, .settings, .classpath and .project and click Apply.
Note
As an alternative see also Section 11.1.2.5, “Ignores template” for Git and Section 11.2.3, “Global ignores” for Subversion.
• In Maven → Installations: • Add… the latest Maven installation Directory…: P:\dev\apps\build\apache-maven-3.5.4 and click OK. Check this new Maven installation and click Apply. • Next in Maven → User Settings point to the corresponding Maven settings: • Global Settings: P:\dev\apps\build\apache-maven-3.5.4\conf\settings.xml. • In Maven: • Check Hide folders of physically nested modules (experimental) when using a git project (see also the note below).
Important
Using the option Hide folders of physically nested modules (experimental) can lead to merge conflicts messages in Team Synchronizing.
Note
As an alternative right-click on the submodule and select Properties or Alt+Enter to set the attribute Derived. Alas the Git Plugin will then mark the submodule as ignored in .gitig- nore. By the way the M2E plugin should set the derived flag on modules folders as well as on output folders, such as target/classes, on project import. But there seems to be a bug because the flag ain't set for the module folders.
Tip
When running mvn clean the derived flag is reset on the target folder. Instead of setting it for every single target folder, right-click on the Working Set, select Maven → Update Project… and click OK or use Alt+F5.
• In Maven → User Interface: • Optionally check Open XML page in the POM editor by default and click OK.
For configuration of the lifecycle mappings see Section 16.1.13, “M2Eclipse lifecycle configuration”.
Occasionally Eclipse build fails with the error “/target/m2e-wtp/web-resources/META-INF/MANIFEST.MF (No such file or directory)”. Change the preferences at Window → Preferences → Maven → Java EE Integration → WAR Project preferences and uncheck Maven Archiver generates files under the build directory (see also M2E-WTP FAQ).
15.1.5.6. e(fx)clipse
e(fx)clipse [version 2.3.0] provides JavaFX tooling for the Eclipse IDE. Read more about the latest develop- ments on BestSolution's specific blog.
Install the plugin e(fx)clipse in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://download.eclipse.org/efxclipse/updates-released/2.3.0/site/ with Name e(fx)clipse Update Site and click Ok. • Select to Work with: e(fx)clipse Update Site - http://download.eclipse.org/efxclipse/updates-re- leased/2.3.0/site/. • Select the following components: 132 Integrated Development Environment
• e(fx)clipse - IDE [2.3.0] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Note
This also should take care of “Access restriction: … is not API (restriction on required library '… \jre\lib\ext\jfxrt.jar')” messages.
15.1.5.7. JD-Eclipse
JD-Eclipse [version 1.0.0-RC2]allows you to display all the Java sources during your debugging process, even if you do not have them all.
Install the plugin JD-Eclipse in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://jd.benow.ca/jd-eclipse/update/ with Name JD-Eclipse Update Site and click Ok.
Note
For installation of a more recent version see also GitHub.
• Select to Work with: JD-Eclipse Update Site - http://jd.benow.ca/jd-eclipse/update/. • Select the following components: • JD-Eclipse Plug-in [0.1.5] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
15.1.5.8. Memory Analyzer
BIRT [version 4.5.0] is an open source technology platform used to create data visualizations and reports that can be embedded into rich client and web applications.
Install the plugin BIRT in Eclipse to be able to use the optional Memory Analyzer Charts:
• Select Help → Install New Software…. • Add… the Location http://download.eclipse.org/birt/update-site/4.4/ with Name BIRT Up- date Site and click Ok. • Select to Work with: BIRT Update Site - http://download.eclipse.org/birt/update-site/4.4/. • Select the following components: • BIRT Framework [4.4.2] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Memory Analyzer (MAT) [version 1.5.0] is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.
Install the plugin Memory Analyzer in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://download.eclipse.org/mat/1.5/update-site/ with Name MAT Update Site and click Ok. • Select to Work with: MAT Update Site - http://download.eclipse.org/mat/1.4/update-site/. • Select the following components: • Memory Analyzer [1.5.0] • Memory Analyzer (Charts) [optional] [1.5.0] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Select Window → Perspective → Open perspective → Other… → Memory Analysis to analyze .hprof files.
VisualVM is a visual tool integrating several commandline JDK tools and lightweight profiling capabilities. De- signed for both production and development time use, it further enhances the capability of monitoring and per-
133 Chapter 15.
formance analysis for the Java SE platform. It is supplied with the JDK and can be run from the commandline with jvisualvm and used to create for example Thread and Heap dumps.
Or run an Java applications with the following options:
• -XX:+HeapDumpOnOutOfMemoryError to enable the JVM to dump the Java heap in HPROF binary format (.hprof file) when an OutOfMemoryError error occurs. • -XX:HeapDumpPath=C:\dev\projects to specify the path and file name of the HPROF dump file.
15.1.5.9. Code Coverage
EclEmma [version 2.3.3] is a free Java code coverage tool for Eclipse.
Install the plugin EclEmma in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://update.eclemma.org/ with Name EclEmma and click Ok. • Select to Work with: EclEmma - http://update.eclemma.org/. • Select the following components: • EclEmma Java Code Coverage [2.3.3] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
Tip
Instead of using the shortcut Alt +Shift+X, T to Run As → 1 JUnit Test use the shortcut Alt+Shift+E, T to Coverage As → 1 JUnit Test. After its first usage a Coverage view appears in the tabs at the bottom of the screen. Optionally just drag it next to the Task List and Outline tabs.
SonarLint [version 2.0.2] is an Eclipse plugin that provides on-the-fly feedback to developers on new bugs and quality issues injected into Java, JavaScript and PHP code.
Install the plugin SonarLint in Eclipse:
• Select Help → Install New Software…. • Add… the Location http://eclipse.sonarlint.org/ with Name SonarLint for Eclipse Update Site and click Ok. • Select to Work with: SonarLint for Eclipse Update Site - http://eclipse.sonarlint.org/. • Select the following components: • SonarLint for Eclipse [2.0.2]. • SonarLint for Eclipse Java Configuration Helper [2.0.2] and click Next twice. • Choose I accept the terms of the license agreements after reading them and Finish. • To complete the installation restart Eclipse.
15.1.5.10.1. SonarLint configuration
In Java perspective configure the connection to SonarQube with Window → Show View → Other… or shortcut Alt+Shift+Q, Q and from SonarLint select: SonarQube Servers.
In the SonarQube Servers view create the New → Server connection with:
• Server URL: http://freedumbytes.dev.net/sonarqube. • Server Label: freedumbytes.dev.net SonarQube. • Username / Token: sonarlint (see also Section 17.2.4.3, “Users”). • Password: password and after Test connection click Finish.
Note
See also Section 17.2.5, “Apache configuration” about allowing anonymous access to Sonar- Qube for SonarLint because it uses the /api to get the information:
Now bind one or more Eclipse project(s) in a Working Set to their remote SonarQube pair. Right click on project(s) and select SonarLint → Bind to a SonarQube project… → Auto bind selected projects and click Finish.
It is possible to filter completed tasks with Hide Completed Tasks.
Important
When you didn't save passwords in the Repository Settings you will see the warning icon syn- chronization failed in front of the related queries. In that case just double click on the repos- itory name in the Task Repositories view, reenter the required passwords, Validate Set- tings and click Finish. After that it should be possible to Synchronize with the repository in the Task List view.
New tasks/issues can now be created in Eclipse by clicking New Task, New Subtask or Alt+Shift+N, T (or creating one directly from events in the Error Log view with Report as Bug):
• Selecting the appropriate repository and click Next. • Select the product after Update Projects from Repository when it isn't already there and click Finish. • After entering the issue click Submit.
Tip
If you get the following error: “Unable to submit at this time. Check connectivity and retry.” when the connection is valid, try to reproduce the error using the Open with Web Browser option. For example when running Bugzilla without an SMTP server configured this causes the above mentioned error. In the Web Browser you will see the following error: “There was an error sending mail from '[email protected]' to '[email protected]':Couldn't connect to localhost”. In this case make sure that all users have Bugmail Disabled.
136 Integrated Development Environment
If you notice that the changes did make it in the issue tracker open the Task again. Click Synchronize Incoming Changes to fetch them and then delete the now obsolete local changes with Clear outgoing.
The following shortcuts are available for tasks:
• Ctrl+F12: Open task dialog. • Ctrl+Shift+F12: Open repository task dialog to find tasks not available on the Task List view. Then drag and drop this Issue or Bug onto the Task List if necessary. • Ctrl+F9: Activate task dialog. • Ctrl+Shift+F9: Deactivate current task.
Important
It is possible to hyperlink other task in comments but for these hyperlinks to be useful also for the issue trackers themselves use only the following formats:
• Bugzilla: bug 1 or bug #2. • JIRA: the project key followed by issue number, for example: JMW-1.
These same entries can be used in comments in java and resource files. To activate hyperlinking in those files associate the appropriate Task Repository by right clicking on the project and open Properties → Task Repository and selecting the appropriate one.
15.1.6.2. Mylyn task focus
When you first activate a task you will notice that the Package Explorer view is empty in Focus on Active Task mode. It is possible to drill down from the Working Set or any other node to the interesting bits using Alt+click or the plus icon displayed behind the node to Show Filtered Children. Use Ctrl+click to select mul- tiple interesting nodes. When finished marking interesting nodes click (without Ctrl) on whitespace or an already marked node to filter the uninteresting nodes again. Type Ctrl+Alt+Shift+Up to Mark as Landmark an ele- ment in the resource tree. Type Ctrl+Alt+Shift+Down to Remove from Context the selected node.
Note
Closing a file will make its node uninteresting immediately (see also Window → Preferences → Mylyn → Context → Remove file from context when editor is closed).
Tip
Once the task is finished attach the current context to it. This way it is possible to restore the last context should the task (re)enter someone's Task List. Just right click on the task and select Context → Retrieve… to do so.
15.1.6.3. Bugfixing
Should the code require bugfixing and a resource needs to be changed in the bugfix as well as a new feature being worked on one might consider to create a patch for the new feature and revert the current changes.
Open the Team Synchronizing perspective and in the Synchronize view right click on the Change Set for the conflicting new feature. Select Create Patch… to open the Changes pane in the Show Change Sets mode. Select the Save to clipboard setting and click Finish.
Right click on the Change Set for the conflicting new feature again and select Open Corresponding Task. Open the Attachments, click Attach…, with Clipboard setting and click Next. Check Patch and enter a Description of backup patch and click Finish (or Next to verify what you are going to save first).
Finally right click on the Change Set for the conflicting new feature to Override and Update all the changed resources.
Once the bugfix is commited reactive the new feature with Ctrl+F9 in the activate task dialog and go to the task itself with Ctrl+F12. Open the Attachments, select the attachment with your Description of backup patch
137 Chapter 15.
by double clicking on it. The patch will be opened in the Eclipse Web Browser. First type Shift+F10 and Select All then type Shift+F10 again and select Copy. Now in the Package Explorer view right click on a project node and select Team → Apply Patch… from the Clipboard with Next → Finish. In case of conflicts you might need to play with or Guess the Maximum fuzz factor and Generate a .rej file for unmerged hunks next to the problem resource to manually apply some of the changes.
Important
Make sure that formatting changes aren't a big part of the patch (see also Tip on Eclipse Save Actions).
138 Chapter 16. Chapter 16. Project
Lets setup a base structure, for Maven projects to come, with the following JIRA project:
• Maven Setup [MVNSTP]: Maven support components (see also repository https://gitlab.com/freedum- bytes/setup and site https://freedumbytes.gitlab.io/setup).
With the following components: 1. setup: Setup the site, license and plugin configuration. 2. java: Java Project Object Model: profile, build and reporting settings. 16.1. Setup base components
Start a new bare Git repository maven-setup by calling /p/dev/apps/windows/batch/ git-cre- ate-repo.sh /p/dev/data/repo/git maven-setup.git "Maven support components" -jenkin- sHook in Git Bash.
Clone this remote repository with the following command git-clone jjasper maven-setup.
Start by adding the configuration of the Ignores template .gitignore and the Attributes template .gitat- tributes.
Create the minimal POM in C:\dev\projects\maven-setup\pom.xml:
For the release procedure the version should end with -SNAPSHOT, but it is left out in this document.
In Eclipse create a Working Set Maven Setup and import this Maven Setup project into it with File → Import… → Maven → Existing Maven Projects from Root directory C:\dev\projects\maven-setup and click Finish.
Note
To exclude the target directory from search actions and the like, right click on it and select Properties → Derived.
Tip
When running mvn clean the derived flag is reset on the target folder. Instead of setting it for every single target folder, right-click on the Working Set, select Maven → Update Project… and click OK or use Alt+F5.
16.1.1. Team Synchronizing Git
Right-click on the Working Set Maven Setup and select Team → Share Project… → Git → Next → Use or create Repository in parent folder of project and click Finish.
Open the Team Synchronizing perspective and select Synchronize… → Git and click Next to synchronize with the Destination refs/heads/master for the maven-setup [master] and Include local uncommited changes in comparison and click Finish.
139 Chapter 16.
Other available options are Pull and Push.
16.1.2. Project information
Supply the project information in C:\dev\projects\maven-setup\pom.xml:
setup4.3pom + + ${organizationName} Maven Setup + Setup the site, license and plugin configuration. + ${projectRoot} + 2010
All patch snippets are split into logical blocks and non essential lines are removed for better page breaks in the generated pdf and the html print out.
16.1.3. Environment
Define the Continuous Integration build system, the Issue Tracker system and the Version Control System man- agement in C:\dev\projects\maven-setup\pom.xml:
Above settings use local tools for source, ci, issue and artifacts manager by default. To overwrite those settings with open source tools GitLab (CI) and Sonatype OSS Repository Hosting activate the following profile openSource.
Should any of the hosts change and a project could not upgrade to the latest Maven Setup parent it now is possible to overwrite it in the Maven configuration file P:\dev\apps\build \apache-maven-3.5.4\conf\settings.xml:
It is even possible to overwrite that settings from the command line with the option - DgitRepo=http://freedumbytes.dev.net/scm/git.
16.1.4. Site
The Site Plugin is used to generate a site for the project. The generated site also includes the project's reports that are configured in the section of the POM.
16.1.4.1. Configuration
Setup the initial site descriptor in C:\dev\projects\maven-setup\src\site\site.xml:
Because the LAN only tooling links will work for the locally hosted Maven sites only, optionally hide them with some Javascript C:\dev\projects\maven-setup\src\site\resources\js\hide-lan-links.js: TODO
+(function($) { + $(window).on('load', function () { + deactivateLANOnlyLinks(); + }); +}(jQuery)); + +function deactivateLANOnlyLinks() { + var hostname = window.location.hostname; + + if (hostname=="freedumbytes.dev.net") { TODO != + $('.nav>li>a[href*="freedumbytes.dev.net"]').css({"display": "none"}); + } +}
Prepare Javascript activation in site footer C:\dev\projects\maven-setup\src\site\site.xml (see also Section 16.1.6.4, “Reuse site descriptor”): TODO
+ +
Important
Now the LAN only tooling links will be removed from the Open Source site at a later stage when jQuery is also available. TODO
Setup the Project Info Reports in C:\dev\projects\maven-setup\pom.xml:
When using Nexus disable the generation of the repository locations in the maven-project-in- fo-reports-plugin of the section with set to false (see also issue MPIR-137). This is also a performance boost.
Verify those settings with mvn site in your browser at file:///C:/dev/projects/maven-setup/target/site/ index.html.
16.1.4.2. Apache Maven Fluido Skin
The Apache Maven Fluido Skin is an Apache Maven site skin built on top of Twitter's Bootstrap.
To switch from the Maven Default Skin to the Apache Maven Fluido Skin edit C:\dev\projects\maven-set- up\src\site\site.xml:
When the property is redefined in the child project (for example projectRoot in say ver- sions-rules TODO link) then the prefix this. isn't necessary. The difference in the resulting footer is as follows:
159 Chapter 16.
Because the JavaScript file hide-lan-links.js isn't redistributed the later is in this case still required.
Verify those settings with mvn deploy site-deploy in your browser at http://freedumbytes.dev.net/nexus/ index.html#nexus-search;gav~nl.demon.shadowland.freedumbytes.maven~setup~~~site.
16.1.6.5. PGP Signatures
One of the requirements for publishing the artifacts to the Central Repository, is that they have been signed with PGP.
GnuPG is a complete and free implementation of the OpenPGP standard as defined by RFC4880 (also known as PGP). GnuPG allows to encrypt and sign your data and communication, features a versatile key management system as well as access modules for all kinds of public key directories.
Download the binary “Simple installer for the current GnuPG”: gnupg-w32-2.2.0_20170828.exe [version 2.2.0].
Run this .exe file to install GnuPG in C:\Program Files (x86)\GnuPG.
Verify the installation with gpg --version.
Note
The agent is automatically started on demand by gpg, gpgsm, gpgconf , or gpg-con- nect-agent. Thus there is no reason to start it manually. In case you want to use the included Secure Shell Agent you may start the agent using gpg-connect-agent /bye or gpg-con- nect-agent reloadagent /bye.
To stop the gpg-agent you can use gpgconf --kill gpg-agent or gpg-connect-agent killagent /bye.
Tip
To use PGP to exchange secure emails you have to bring together three programs: GnuPG, Mozilla Thunderbird and Enigmail (see also How to: Use PGP for Windows).
Important
For some older version of GnuPG the gpg-agent locks the directory of the initial signature causing the following error in the consecutive Jenkins jobs: “remote file operation failed: P:\dev\data\ci\jeeves\workspace\… at hudson.remoting.Channel@…:JNLP4-connect connec- tion from …: java.io.IOException: Unable to delete 'P:\dev\data\ci\jeeves\workspace\…'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.”. Use the Resource Monitor for example to kill the gpg-agent daemon that locks the folder when the options mentioned in the above note don't work.
Generate a key with:
gpg --gen-key
160 Project
Real name: The Who Email address: guess.who@company.org Passphrase: guess
gpg --list-keys gpg --list-secret-keys
Sign file with gpg -ab sample.java resulting in a separate file sample.java.asc.
Verify signature with gpg --verify sample.java.asc sample.java or gpg --verify sample.java.asc.
Distribute public key with gpg --keyserver hkp://pool.sks-keyservers.net --send-keys 18A6705DE836A357D96D6D98DD605CC8A9582C0D.
Import public key with gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys D8692123C4065DEA5E0F3AB5249B39D24F25E3B6.
Configure the GPG credentials using a gnupg profile in the Maven configuration file P:\dev\apps\build \apache-maven-3.5.4\conf\settings.xml:
When using git-bash an older gpg [1.4.20] is included causing the following error: “gpg: skipped "DD605CC8A9582C0D": secret key not available. gpg: signing failed: secret key not available.” because this one can't locate the key that was generated above. Therefore specify the .
Update the Open Source profile to also create PGP Signatures in the C:\dev\projects\maven-set- up\pom.xml:
To prevent the pinentry popup use the loopback pinentry option. This fakes a pinentry by using inquiries back to the caller to ask for a passphrase.
16.1.7. Java setup
Configure the default Java projects settings. 16.1.7.1. Compiler
The Compiler Plugin is used to compile the sources of your project. The default compiler is javac and is used to compile Java sources. To compile the sources using the latest available JDK assume it is located at JAVA_HOME in C:\dev\projects\maven-setup\pom.xml:
The fork option needs to be set to true for the ${base…} settings to take effect. TODO remnant
Warning
Compilation with lower jdk -source and -target options could casue “java.lang.NoClassDefFoundError: java/lang/reflect/Executable” and “java.lang.ClassNotFoundException: java.lang.reflect.Executable” errors when Java 1.8-only classes sneak into the bytecode (see also JDK 1.8/1.7 Compatibility Gotcha).
Verify the compiler settings with mvn help:effective-pom. 16.1.7.2. Surefire
Split unit and integration tests by delegating integration tests to the Maven Failsafe plugin (see also The correct way to use integration tests in your build process).
The Surefire Plugin is used during the test phase of the build lifecycle to execute the unit tests of an application. Configure default settings in C:\dev\projects\maven-setup\pom.xml:
**/*Tests.java is added for JUnit 5 tests (see also SUREFIRE-1260).
The Failsafe Plugin is designed to run integration tests while the Surefire Plugin is designed to run unit tests. The name (failsafe) was chosen both because it is a synonym of surefire and because it implies that when it fails, it does so in a safe way. Configure default settings in C:\dev\projects\maven-setup\pom.xml:
Also added two extra wildcard patterns **/*UnitTest.java and **/ *IntegrationTest.java to the defaults. Also added like JUnit 5 support mentioned above **/*IntegrationTests.java .
16.1.7.2.1. Java profile
Option 1 to add both plugins to the build process with a java profile in C:\dev\projects\maven-set- up\pom.xml:
The Surefire Report Plugin parses the generated TEST-*.xml files and renders them to Doxia which creates the web interface version of the test results. Configure default settings in C:\dev\projects\maven-set- up\pom.xml:
Single module projects should use false, otherwise the surefire report will show number of tests is 0 and the failsafe report will be missing altogether.
Option 1 the java profile in C:\dev\projects\maven-setup\pom.xml:
Since Jenkins also shows the Latest Test Result info, this report isn't really necessary. TODO SonarQube
16.1.7.2.4. Unit vs Integration tests
Now we reach the most important point regarding unit tests. In a big enterprise application integration and unit tests must be handled differently. Most importantly your build server (e.g. Jenkins) should contain a mixture of jobs that deal with both kinds of tests (see also The correct way to use integration tests in your build process).
Create separate options to disable unit and integration tests during builds in C:\dev\projects\maven-set- up\pom.xml:
${skipTests}${skipTests} …
Also supply a convenience profile to activate the integration tests only with mvn verify -PintegrationTest- sOnly:
+
167 Chapter 16.
+ integrationTestsOnly + + + true + false + +
This leads to the following list of command line options:
@rem defaults to unit and integration tests mvn clean verify
@rem skip all tests mvn clean verify -DskipTests mvn clean verify -Dmaven.test.skip=true (see also issue MCOMPILER-284)
@rem unittests only mvn clean test mvn clean verify -DskipIntegrationTests mvn clean verify -DskipIntegrationTests=true mvn surefire:test [-Dtest=…Test [-DfailIfNoTests=false]]
Both commands mvn surefire:test and mvn failsafe:integration-test will result in “No tests to run” after running mvn clean.
16.1.7.3. JaCoCo
The JaCoCo Plugin provides the JaCoCo (Java Code Coverage Library) runtime agent to your tests and allows basic report creation. Configure it in C:\dev\projects\maven-setup\pom.xml:
To skip code coverage supply the following command line option: -DskipCodeCoverage or - Djacoco.skip=true because of the newly defined property skipCodeCoverage.
16.1.7.4. Source
The Source Plugin creates a jar archive of the source files of the current project:
It is possible to force some profiles to get enabled during release (for example to enable time-consuming tasks that are only relevant at release-time) by setting a comma separated list of profiles names in the releasePro- files parameter in C:\dev\projects\maven-setup\java\pom.xml:
128m512m + + true + true + false
false${jacoco.skip}false … truetruetrue -
170 Project
+ documentsfalse
true
16.1.7.5. Javadoc
The Javadoc Plugin uses the Javadoc tool to package the generated javadocs into a jar file for distribution during for example the Jenkins build:
Set to false because those links will be created as external links during site generation anyway and (see also issue MJAVADOC-275) to prevent build errors when using the javadoc goal before all project artifacts are installed in the verify phase).
Note
Anyone running javadoc is advised to be aware of the problems that may arise when generat- ing non-conformant output or output containing executable content, such as JavaScript. The
171 Chapter 16.
standard doclet does provide the doclint feature to help developers detect common problems in documentation comments; but, it is also recommended to check the generated output with any appropriate conformance and other checking tools. Possible settings -Xdoclint:(all| none|[-]group) where group is comma separated set of accessibility, html, missing, reference and syntax (see also Turning off doclint in JDK 8 Javadoc).
Also enable conditional activation of the javadoc archive creation through the profile named documents:
It is possible to aggregate-jar Javadoc into the parent project of a multi-module project, but because inherited should be false it can't be preconfigured in an ancestor project. There- fore add the following to the parent project of a multi-module project:
Javadoc Reports aggregate should have inherited false, but in that case an ancestor profile isn't applied anyway.
16.1.7.6. Animal Sniffer
The Animal Sniffer Plugin is used to build signatures of APIs and to check your classes against previously generated signatures. This plugin is called animal sniffer because the principal signatures that are used are those of the Java Runtime, and since Sun traditionally names the different versions of its Java Runtimes after different animals, the plugin that detects what Java Runtime your code requires was called Animal Sniffer:
When baseJdkVersion is earlier than baseJavacExecutable an apiCheck might be in or- der (see also How Spring achieves compatibility with Java 6, 7 and 8). TODO mvn …
16.1.7.7. Enforcer
The Enforcer Plugin provides goals to control certain environmental constraints such as Maven version, JDK ver- sion and OS family along with many more built-in rules and user created rules.
Extra Rules are available for use with Apache's Maven Enforcer Plugin such as enforceBytecodeVersion in addition to above mentioned Animal Sniffer:
OWASP Dependency Check is a utility that identifies project dependencies and checks if there are any known, publicly disclosed, vulnerabilities. Just add the Maven Plugin configuration:
… 3.0.0 + 3.2.1
… truetrue + + ${project.build.directory}/owasp-reports + all + 24
Should the default location of the NVD Data Feeds be moved again to a version subfolder only, just override the following properties (see also release notes 3.1.2):
16.1.7.8.1. Suppressing False Positives and Resolving False Negatives
Because of the way dependency-check works both false positives and false negatives may exist. Please read How to read the report to get a better understanding of sorting through the false positives and false negatives. See the Suppressing False Positives page for more information on how to generate and use the suppression file.
With the reusable custom suppression and hints files from Custom Dependency Check change plugin setup to:
… all24 +
Note
In case of pre-release testing use for example the latest deployed SNAPSHOT:
The custom files were created because dependency mysql-connector-java caused all MySQL database vul- nerabilities to be included in the list. More information can be found at:
• Issue 952 False positive: MySQL JDBC driver reported as cpe:/a:mysql:mysql. • Issue 434 MySQL Connector/J Misidentified as MySQL Server. • Google Groups Forum Topic: mysql-connector-java shows wrong vulnerabilities? • xmlns, xmlns:xsi, xsi:schemaLocation, and targetNamespace?
Note
Since OWASP Dependency Check Plugin release 3.1.0 (see also pull request 1028 False Posi- tive Reduction) these custom suppression and hints files are obsolete unless the Vulnerabilities should show up as Suppressed.
Based on that information the following suppression and hints xmls where created:
We're not interested in vulnerabilities in MySQL server, only the Connector/J.^mysql:mysql-connector-java:.*$cpe:/a:mysql:mysql:.*
We're not interested in vulnerabilities in MySQL server, only the Connector/J.^mysql:mysql-connector-java:.*$cpe:/a:oracle:mysql:.*
We're not interested in vulnerabilities in MySQL Connector/C.^mysql:mysql-connector-java:.*$CVE-2017-3635
When tested with the following dependency:
mysqlmysql-connector-java5.1.34
The number of Vulnerabilities Found drop from 449 to 3, leaving only:
I don't think CVE-2015-2575 , which was mentioned in the above listed Issue 434, is a Connector/J issue, but you can check it yourself under the collapsed Suppressed Vulnerabilities at the end of the report.
16.1.7.8.2. NVD CVE/CPE data
It is important to understand that the first time this task is executed it may take minutes as it downloads and processes the data from the National Vulnerability Database (NVD) hosted by NIST. After the first batch download, as long as the plugin is executed at least once every seven days the update will only take a few seconds.
178 Project
Known issues:
• Scan crashes with error -14 reading a corrupt MANIFEST.MF (see also issue 1024). • As stated in the configuration the default data directory to hold SQL CVEs contents is the Maven localRepository\org\owasp\dependency-check-data with a subfolder per version. Although this should generally not be changed it could be useful when the CI tools do not reuse the local Maven reposito- ry. Because the time for a complete database creation is rather long and thus a central location could save minutes.
Note
When the database is accessed, there will be a dc.lock.db file visible. If updates are in- stalled, then an additional dc.update.lock file is visible. Thus concurrent dependen- cy-check-maven runs might cause delays during separate CI jobs. Those extra delays are still a lot smaller than downloading the whole database every run.
A sample initial run:
$ mvn org.owasp:dependency-check-maven:aggregate -DdataDirectory=/c/dev/data/ owasp-dependency-check-data Tue 2017-12-26 00:06:13.501 CET [INFO] Scanning for projects... Tue 2017-12-26 00:06:15.688 CET [INFO] --- dependency-check- maven:3.0.2:aggregate Tue 2017-12-26 00:06:16.643 CET [INFO] Checking for updates Tue 2017-12-26 00:06:16.646 CET [INFO] starting getUpdatesNeeded() ... Tue 2017-12-26 00:06:17.758 CET [INFO] NVD CVE requires several updates; this could take a couple of minutes. Tue 2017-12-26 00:06:17.765 CET [INFO] Download Started for NVD CVE - 2004 … Tue 2017-12-26 00:06:17.778 CET [INFO] Download Started for NVD CVE - 2017 Tue 2017-12-26 00:06:17.778 CET [INFO] Download Started for NVD CVE - Modified … Tue 2017-12-26 00:06:25.778 CET [INFO] Download Complete for NVD CVE - 2017 (8000 ms) … Tue 2017-12-26 00:06:51.500 CET [INFO] Processing Started for NVD CVE - 2017 … Tue 2017-12-26 00:08:03.185 CET [INFO] Processing Complete for NVD CVE - 2017 (71685 ms) Tue 2017-12-26 00:08:03.186 CET [INFO] Begin database maintenance. Tue 2017-12-26 00:08:03.985 CET [INFO] End database maintenance. Tue 2017-12-26 00:08:04.376 CET [INFO] Check for updates complete (107733 ms) Tue 2017-12-26 00:08:04.724 CET [INFO] Analysis Started … Tue 2017-12-26 00:08:06.734 CET [INFO] Analysis Complete (2 seconds) Tue 2017-12-26 00:08:07.000 CET [INFO] ------Tue 2017-12-26 00:08:07.000 CET [INFO] Total time: 01:53 min Tue 2017-12-26 00:08:07.110 CET [INFO] ------
A sample run with above configured 24 hours to wait before checking for new updates from the NVD:
$ mvn org.owasp:dependency-check-maven:aggregate -DdataDirectory=/c/dev/data/ owasp-dependency-check-data Tue 2017-12-26 00:09:13.898 CET [INFO] Scanning for projects... Tue 2017-12-26 00:09:15.144 CET [INFO] --- dependency-check- maven:3.0.2:aggregate Tue 2017-12-26 00:09:16.085 CET [INFO] Checking for updates Tue 2017-12-26 00:09:16.088 CET [INFO] Skipping NVD check since last check was within 24 hours. Tue 2017-12-26 00:09:16.122 CET [INFO] Check for updates complete (37 ms) Tue 2017-12-26 00:09:16.502 CET [INFO] Analysis Started … Tue 2017-12-26 00:09:18.578 CET [INFO] Analysis Complete (2 seconds) Tue 2017-12-26 00:09:18.861 CET [INFO] ------Tue 2017-12-26 00:09:18.861 CET [INFO] Total time: 5.216 s
179 Chapter 16.
Tue 2017-12-26 00:09:19.023 CET [INFO] ------
A sample run with forced update, for the original Maven repo location, by supplying a smaller number for cveValidForHours:
$ mvn org.owasp:dependency-check-maven:aggregate -DcveValidForHours=0 Tue 2017-12-26 00:10:17.884 CET [INFO] Scanning for projects... Tue 2017-12-26 00:10:19.164 CET [INFO] --- dependency-check- maven:3.0.2:aggregate Tue 2017-12-26 00:10:20.127 CET [INFO] Checking for updates Tue 2017-12-26 00:10:20.130 CET [INFO] starting getUpdatesNeeded() ... Tue 2017-12-26 00:10:20.813 CET [INFO] Download Started for NVD CVE - Modified Tue 2017-12-26 00:10:23.452 CET [INFO] Download Complete for NVD CVE - Modified (2639 ms) Tue 2017-12-26 00:10:23.454 CET [INFO] Processing Started for NVD CVE - Modified Tue 2017-12-26 00:10:25.817 CET [INFO] Processing Complete for NVD CVE - Modified (2363 ms) Tue 2017-12-26 00:10:25.818 CET [INFO] Begin database maintenance. Tue 2017-12-26 00:10:26.752 CET [INFO] End database maintenance. Tue 2017-12-26 00:10:26.877 CET [INFO] Check for updates complete (6750 ms) Tue 2017-12-26 00:10:27.326 CET [INFO] Analysis Started … Tue 2017-12-26 00:10:29.809 CET [INFO] Analysis Complete (2 seconds) Tue 2017-12-26 00:10:30.170 CET [INFO] ------Tue 2017-12-26 00:10:30.171 CET [INFO] Total time: 12.548 s Tue 2017-12-26 00:10:30.357 CET [INFO] ------
Note
Alternatively force update only with mvn org.owasp:dependency-check- maven:update-only -DcveValidForHours=0 or use mvn org.owasp:dependency- check-maven:purge to delete the local copy of the NVD.
16.1.7.9. JDeps
The JDeps Plugin uses the jdeps tool to analyze classes for internal API calls. For more information about the standard jdeps tool, please refer to Java Dependency Analysis Tool.
In JDK 8, a new command-line tool, jdeps, is added that developers can use to understand the static dependencies of their applications and libraries. It also provides an -jdkinternals option to find dependencies to any JDK internal APIs that are unsupported and private to JDK implementation (see Why Developers Should Not Write Programs That Call sun Packages).
Tip
SonarQube has a lock-in, pitfall rule: “Classes from sun packages should not be used”.
16.1.7.10. Shade
The Shade Plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade, i.e. rename, the packages of some of the dependencies.
16.1.8. Continuous integration Jenkins vs GitLaB CI (see gitlab-ci.png) TODO
Browse to http://freedumbytes.dev.net/jenkins/newView to create the following List Views: Application , Component, Demo, EOL, Maven and Sandbox.
Finally select the Component view http://freedumbytes.dev.net/jenkins/view/Maven/ and create a New Job in Jenkins with Job name maven-setup to Build a maven2/3 project with the following settings:
180 Project
Tip
To keep the folder name short use the shorter git project name as Job name maven-setup and define under Advanced Project Options the longer Display Name.
• Maven project job name / Discard Old Builds / Log Rotation:
• Click Advanced… to edit Advanced Project Options to Block builds / Project display name:
• Source Code Management for Git with Credentials: • Configure Git proxy authentication for the Repository URL (see also Section 14.1.4.5, “Git authentication” to prevent error “stderr: fatal: Authentication failed for 'http://freedumbytes.dev.net/git-repo/maven- setup.git/'”) with add Credentials, when the _netrc solution isn't used.
• Branches to build such as master:
Tip
If not all future branches should trigger build just specify the required ones, for example for qa-sandbox project:
181 Chapter 16.
Note
In both cases a manual Build Now will only trigger a build of the branch that was build during the prior build #.
Test those different settings for example by making changes to two branches and pushing those changes all at once with git push --all origin.
Warning
Ran into a “Detached HEAD” situation with the following combination of settings: Branches to build ** and Additional Behaviours Shallow clone (see below).
• Repository browser such as FishEye:
• Additional Behaviours for clone:
Or retrieve some more:
• Build Triggers such as hooks and poll:
182 Project
Note
Make sure the poll interval is larger than the quiet period [113] otherwise the build will never start. When using the hooks/post-receive of the git repo instead, make sure the Poll SCM option is ticked.
• Build Goals and Options:
• -B or --batch-mode: Batch mode is essential if you need to run Maven in a non-interactive, continuous integration environment. When running in non-interactive mode, Maven will never stop to accept input from the user. Instead, it will use sensible default values when it requires input. • -e, --errors: Will come in handy if you are a Maven developer, or if you need to diagnose an error in a Maven plugin. • -P, --activate-profiles: Comma-delimited list of profiles to activate. Currently available profiles (TO- DO update list): • integrationTestsOnly: activate the integration tests only; thus skipping unit tests (see also Sec- tion 16.1.7.2.4, “Unit vs Integration tests”). • documents: activation of the source and javadoc archive creation (see also Section 16.1.7.4, “Source” and Section 16.1.7.5, “Javadoc”). • enableUpdatesReports: activation of the reports of those project dependencies, plugins and prop- erties, which have newer versions available (see also Section 16.1.5, “Plugin Versions”). • Disable automatic artifact archiving. • Disable automatic site documentation artifact archiving. • Disable automatic fingerprinting of consumed and produced artifacts.
Note
See also Section 13.1.6.2, “Artifact deployment” and Sec- tion 4.2.5, “Password Encryption” to make sure the “Upload- ing: http://freedumbytes.dev.net/nexus/content/repositories/snapshots/net/dev/free- dumbytes/maven/setup/…-SNAPSHOT/setup-x.y-20131013.201411-1.pom” succeeds.
Tip
When the project invokes plugins with pre-site configuration use phase site-deploy instead of goals site:site site:deploy (see also Section 17.3.7, “PressGang Example”).
• Build Settings for notifications:
183 Chapter 16.
Note
In case of the following error “Cause: C:\Windows\System32\config\systempro- file\.m2\settings-security.xml (The system cannot find the file specified)” during a Jenkins build make sure the Tomcat service is not running under the Local system account.
16.1.9. Release
Let Jenkins take care of the snapshot deployments, but handle the release action from the command line with the Release Plugin.
The Nexus Repository is already configured as the default for deployments.
Make sure all local changes have been committed with commands:
git status git tag -l git log --tags --simplify-by-decoration [--pretty="format:%ai %s"] git log --oneline --decorate
Also check if all remote changes are “up to date” with command:
git remote show origin
To change the 3.0-SNAPSHOT version to a release 3.0 version, which also will be tagged in the SCM as maven- setup-3.0, and to create the next 3.1-SNAPSHOT version use the command:
(or try a Dry Run with the option -DdryRun=true followed by mvn release:clean or mvn release:rollback to remove all of the created files).
When the POM is configured to not pushChanges the options username and password are superfluous.
Important
In that case we need to git push --follow-tags origin to push the prepared commits for the release, inclusive its tag, and the one for the next devel- opment iteration.
With git config --global push.followTags true the option follow-tags is no longer necessary.
TODO 2 builds GitLab and no tag build on merge vs 1 Jenkins and opensource helper script.
TODO now SNAPSHOTS in ossrh (see link).
TODO when -DpreparationGoals="clean install".
Note
When using release plugin version 2.5.1 or earlier with Maven 3.3.3 and later the following error occurs “Failed to invoke Maven build. Error configuring command-line. Reason: Maven
184 Project
executable not found at: …\apache-maven-3.3.3\bin\mvn.bat” because the mvn.bat is re- named to mvn.cmd. When it isn't possible to upgrade the parent base-pom overwrite ver- sion 2.5.1 from the command line with mvn -Dusername=jjasper -Dpassword=password org.apache.maven.plugins:maven-release-plugin:2.5.2:prepare.
Then deploy the release 3.0 version to Nexus using the command:
This will result in a checkout of tag setup-3.0 into C:\dev\projects\maven-setup\target\checkout from which the artifacts and sites will be deployed. Afterwards delete this directory otherwise it might show up in the Eclipse Team Synchronizing perspective.
Tip
Because Jenkins will overwrite the site with the next snapshot anyway, save time by skipping site-deploy with:
Instead of using release:perform use Jenkins CI to release the tag or use script git-mvn- deploy.sh.
Again verify if all remote changes are “up to date” with command:
git status git push git remote show origin git log -1
Note
Now might be a good time to create a Custom skin.
16.1.10. Versions
The Release Plugin is handy for updating SNAPSHOT versions of the modules in a project. But when used on SNAPSHOT dependencies the next project SNAPSHOT version will be pointing to the next SNAPSHOT of the de- pendencies also. Which is usually not what is required. In comes the Versions Plugin option mvn versions:use- releases (for usage see also Section 17.3.8, “Publication”).
Look for updates of dependencies with mvn versions:display-dependency-updates and plugins with mvn versions:display-plugin-updates and the:
• Advance the dependency version to the next higher with mvn versions:use-next-releases with or the latest with mvn versions:use-latest-releases. • Switch from a snapshot dependency to its released version with mvn versions:use-releases. • Upgrade the parent reference with mvn versions:update-parent. • The Versions Plugin has the following reporting goals [153]: • versions:dependency-updates-report produces a report of those project dependencies which have newer versions available. • versions:plugin-updates-report produces a report of those plugins which have newer versions available. • versions:property-updates-report produces a report of those properties which are used to control artifact versions and which properies have newer versions available.
To lock the version of a SNAPSHOT for say a release candidate build use mvn versions:lock-snapshots and revert with mvn versions:unlock-snapshots.
185 Chapter 16.
To set the patch version for a branch version of for example 1.0 to 1.0.1-SNAPSHOT use mvn versions:set - DnewVersion=1.0.1-SNAPSHOT.
Remove the backup files with mvn versions:commit or revert changes with mvn versions:revert.
16.1.11. Branch
To create a branch on for example the qa-sandbox project to use Clover instead of JaCoCo code coverage, the Release Plugin should be able to handle the branch action from the command line with:
Alas mvn release:branch fails on Windows to commit changed POM in branch (see also MRELEASE-881). Thus next as a workaround use the Versions Plugin as follows.
Switch to the new branch:
git branch git checkout code-coverage-clover git status
Fix project version:
mvn versions:set -DnewVersion=1.0.1-SNAPSHOT git status
Remove pom.xml.versionsBackup files:
mvn versions:commit git status
Important
In case of “org.codehaus.mojo:versions-maven-plugin:2.2:set failed. Null- PointerException” switch back to version 2.1 (and join set and commit goals): mvn org.codehaus.mojo:versions-maven- plugin:2.1:set org.codehaus.mojo:versions-maven-plugin:2.1:commit - DnewVersion=1.0.1-SNAPSHOT.
Commit changed pom.xml files:
git diff git commit -am "branch code-coverage-clover" git status
Push the local branch to the remote repository (publish) and set up the tracking information:
git push -u origin code-coverage-clover git status
Note
In case of the error “src refspec master matches more than one. failed to push some refs to” there might be a tag with the same name.
Working on this branch in Eclipse:
186 Project
Figure 16.4. Eclipse Switch To Branch
Don't forget to switch the synchronization destination also:
Figure 16.5. Eclipse Synchronize Branch
16.1.12. Grouping Dependencies
If you have a set of dependencies which are logically grouped together. You can create a project with pom pack- aging that groups dependencies together. To solve version mismatch issues, you can use the concept of a Bill of Materials (BOM) dependency. A BOM dependency keep track of version numbers and ensure that all dependen- cies (both direct and transitive) are at the same version.
187 Chapter 16.
16.1.12.1. Bill of Materials
Lets setup a base structure, for Bill of Materials code to come, with the following JIRA project:
• Bill of Materials [BOMGRP]: of the most popular Maven dependencies (see also repository branches/bill-of- materials).
With the following components: 1. bom: Setup the required versions of the most popular Maven dependencies. 2. grouping: Set of dependencies which are logically grouped together.
Start a new bare Git repository bill-of-materials by calling /p/dev/apps/windows/batch/git-cre- ate-repo.sh /p/dev/data/repo/git bill-of-materials.git "Maven Bill of Materials" - jenkinsHook in Git Bash.
Clone this remote repository with the following command git-clone jjasper bill-of-materials.
Create the minimal POM in C:\dev\projects\bill-of-materials\pom.xml:
+ Free Dumb Bytes Maven Bill of Materials + + Setup the required versions of the most popular Maven dependencies. + + ${mavenHost}/bill-of-materials + 2015
Remove the now obsolete version settings in C:\dev\projects\skins\maven-reflow-skin\pom.xml:
log4jlog4j - 1.2.17provided
194 Project
commons-loggingcommons-logging - 1.2provided
junitjunit - 4.12test
org.assertjassertj-core - 3.2.0test
org.mockitomockito-all - 1.10.19test
16.1.12.2. Groups
Add the grouping parent module in C:\dev\projects\bill-of-materials\pom.xml:
+ + + grouping +
Create the minimal POM in C:\dev\projects\bill-of-materials\grouping\pom.xml:
+ + + + 4.0.0 + + + nl.demon.shadowland.freedumbytes.maven.dependencies + bom + 3.1 + + + grouping + pom + + Free Dumb Bytes Maven Dependencies Grouping + + If you have a set of dependencies which are logically grouped together. + You can create a project with pom packaging + that groups dependencies together. + + 2015 +
16.1.12.2.1. Common group
Add the Common module in C:\dev\projects\bill-of-materials\grouping\pom.xml:
195 Chapter 16.
2015 + + + common +
Group the Common dependencies in C:\dev\projects\bill-of-materials\grouping\com- mon\pom.xml:
The term Mock Objects has become a popular one to describe special case objects that mimic real objects for testing.
Szczepan Faber and his own Monkey Island.
“Expect-run-verify is the most common pattern in mock frameworks. What's wrong with it? Can I test what I want, please? I'm a mockist but existing mock frameworks just don't appeal to me. They spoil my TDD experience. They harm code readability. I needed something better. That's why I came up with Mockito.”
“I'm about to show that it is very useful to distinguish two basic kinds of interactions: asking an object of data and telling an object to do something. Fair enough, it may sound obvious but most mocking frameworks treat all interactions equally and don't take advantage of the distinction.”
• Mockito - The New Mock Framework on the Block • Mockito - one of the better mocks • Mocks and Stubs aren't Spies • Mocks Aren't Stubs • On PowerMock abuse.
16.1.12.2.5. Log group
Add the Log module in C:\dev\projects\bill-of-materials\grouping\pom.xml:
201 Chapter 16.
springhibernatetest + log
Group the Log dependencies in C:\dev\projects\bill-of-materials\grouping\log\pom.xml:
This artifact has default packaging jar instead of type pom because a default log4j.xml is sup- plied for test usage (see also type in the figure below).
202 Project
Figure 16.10. Free Dumb Bytes Log Group
The above mentioned default implementation of C:\dev\projects\bill-of-materials\grouping\log \src\main\resources\log4j.xml:
For example the Custom Reflow Skin module uses the unittest BootStrapResponsiveStyleTest to log 1920 and bigger responsive css settings and is already using a Bill of Materials (see also Section 16.1.12.1.7, “Bill of Materials usage”).
Change the project to use dependencies grouping and to allow to separately upgrade the Bill of Materials intro- duce a groupVersion property in C:\dev\projects\skins\pom.xml:
3.1 + ${bomVersion}
1.2 …
Replace the specific dependencies with a dependencies grouping in C:\dev\projects\skins\maven-re- flow-skin\pom.xml:
Start a new bare Git repository lifecycle-mapping by calling /p/dev/apps/windows/batch/git-cre- ate-repo.sh /p/dev/data/repo/git lifecycle-mapping.git "Eclipse workspace project con- figuration and behaviour during Eclipse workspace build." -jenkinsHook in Git Bash (see also repository branches/lifecycle-mapping).
To avoid “Plugin execution not covered by lifecycle configuration” error messages and POMs cluttered with lifecycle-mapping entries create a metadata file C:\dev\projects\lifecy- cle-mapping\src\config\lifecycle-mapping-metadata.xml (see also GitHub eclipse/m2e-core):
To use this metadata.xml, first deploy it to Nexus with Maven Release and then download the latestLife- cycelMappingVersion to current Maven config folder with mvn process-resources -PdownloadLife- cycleMapping. Configure Eclipse Window → Preferences → Maven → Lifecycle Mappings and point file lo- cation to P:\dev\apps\build\apache-maven-3.5.4\conf\lifecycle-mapping-metadata.xml:
Figure 16.12. Eclipse Maven Lifecycle Mappings
Don't forget to Reload the configuration every time this file is edited or the latest release is downloaded. And select Maven → Update Project Configuration… Alt+F5 for projects that were having errors with any newly added pluginExecutionFilters.
16.1.14. Application Integration overview
The Development Production Line in screenshots:
211 Chapter 16.
Figure 16.13. Maven Setup About
Figure 16.14. Nexus Maven Setup Release 2.0
212 Project
Figure 16.15. Git Maven Setup Tags
Figure 16.16. Eclipse Task MVNSTP-20
213 Chapter 16.
Figure 16.17. Jenkins View Component
Figure 16.18. Jenkins Maven Setup Build
Figure 16.19. Jenkins Maven Setup Build Changes
214 Project
Figure 16.20. JIRA Issue MVNSTP-20
Figure 16.21. FishEye Revisions MVNSTP-20
215 Chapter 16.
Figure 16.22. FishEye Changeset MVNSTP-20
Figure 16.23. FishEye Maven Setup Commit Graph
216 Chapter 17. Chapter 17. Quality Assurance
Quality Assurance, or QA for short, refers to planned and systematic production processes that provide confi- dence in a product's suitability for its intended purpose. 17.1. Maven reports
Instead of configuring the Maven Quality Assurance plugins, of which the output requires a lot of click around between those reports (see also screenshots) and the cross-reference, combine that information using SonarQube for better evaluation. 17.2. SonarQube
SonarQube is an open platform to manage code quality. As such, it covers the 7 axes of code quality. 17.2.1. Resources
• Download SonarQube. • Setup, Documentation, User Guide and Administration Guide. • SonarLint. • Upgrade checklist: • Related products: -. • References: -. • Integration configuration changes: SonarQube - installation guide SONARQUBE_HOME / sonar.properties, SonarQube - Temp folder issue temp-wrapper, SonarQube - Rolling policy of log files, SonarQube - Database settings sonar.properties, SonarQube - Windows Service wrapper.conf / Log On and SonarQube - upgrading.
Note
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe p:\dev\apps\qa \sonarqube-6.7.1\conf p:\dev\apps\qa\sonarqube-6.7\conf to copy the changes in for example sonar.properties and wrapper.conf.
Note
And with p:\dev\apps\editor\winmerge\WinMergeU.exe p:\dev \apps\qa\sonarqube-6.7.1\extensions\plugins p:\dev\apps\qa \sonarqube-6.7\extensions\plugins copy installed plugins such as for example sonar-java-plugin-x.y.jar, sonar-scm-git-plugin-x.y.jar , sonar-scm- svn-plugin-x.y.jar, sonar-widget-lab-plugin-x.y.jar and sonar-depen- dency-check-plugin-x.y.z.jar. Check the Compatibility Matrix for plugins com- patibility and manually upgrade the plugins if necessary.
• SonarQube Products News, Roadmap, Quality Profiles, Blog, Plugin Library and Frequently Asked Questions. • SonarQube Java Analyzer : The Only Rule Engine You Need. • SonarCloud: News. • Water Leak Changes the Game for Technical Debt Management. • SonarQube JavaScript plugin: why compete with JSLint and JSHint? • Housekeeping: When you run a new analysis of your project, some data that was previously available is cleaned out of the database. For example the source code of the previous analysis, measures at directory and file levels, and so on are automatically removed at the end of a new analysis. Additionally, some old analysis snapshots are also removed.
17.2.1.1. Security resources
The SonarJava plugin enables analysis of Java projects. It offers more than 300 rules provided by SonarSource, which span coding conventions, bug detection, and security problems, including coverage of some CERT and CWE rules, including several from the SANS TOP 25 Most Dangerous Software Errors, as well as some OWASP Top Ten vulnerabilities. This plugin is CWE Compatible, which means that you can search for rules in the plugin that are related to CWE items.
The Open Web Application Security Project (OWASP) is a 501(c)(3) worldwide not-for-profit charitable organi- zation focused on improving the security of software. Our mission is to make software security visible, so that individuals and organizations worldwide can make informed decisions about true software security risks.
217 Chapter 17.
The first goal of the OWASP SonarQube Project is to a create a referential of check specifications targeting OWASP vulnerabilities and that can be detected by SAST tools (Static Application Security Testing). From there, the second goal is to provide a reference implementations of most of those checks in the Open Source SonarQube language analyzers (Java, JavaScript, PHP and C#). The News is updated as soon as a SonarQube analyzers con- taining some stuff relating to this OWASP SonarQube project is released.
The Projects pages are constantly being updated. Some pages may contain outdated information. You can help OWASP to keep these pages current by visiting FixME:
• OWASP Zed Attack Proxy (ZAP) is one of the world's most popular free security tools. It can help you automat- ically find security vulnerabilities in your web applications while you are developing and testing your applica- tions. Its also a great tool for experienced pentesters to use for manual security testing. • OWASP Web Testing Environment (WTE), is a collection of application security tools and documentation avail- able in multiple formats such as VMs, Linux distribution packages, Cloud-based installations and ISO images. • OWASP Offensive Web Testing Framework (OWTF) is a project focused on penetration testing efficiency and alignment of security tests to security standards like: The OWASP Testing Guide (v3 and v4), the OWASP Top 10, PTES and NIST. • Dependency-Check is a utility that identifies project dependencies and checks if there are any known, pub- licly disclosed, vulnerabilities. Currently Java, .NET, Ruby, Node.js, and Python projects are supported; addi- tionally, limited support for C/C++ projects (see also Section 16.1.7.8, “OWASP Dependency Check” and Sec- tion 17.2.4.1, “Marketplace”). • OWASP Testing Project and Testing Guide v4. • OWASP Cheat Sheet Series was created to provide a concise collection of high value information on specific web application security topics. These cheat sheets were created by various application security professionals who have expertise in specific topics. 17.2.2. SonarQube installation guide
Download the archive: sonarqube-6.7.1.zip [version 6.7.1].
Important
Better skip version 5.4 because of reverse proxy issue with SONAR-7122 Drop the ability to cus- tomize the web app context and wait for 5.5 (SONAR-7494 Reintroduce the web app context) and hope they don't do it again in 6.x.
Alternatively create a subdomain sonarqube.freedumbytes.dev.net and redirect freedumbytes.dev.net/sonarqube there (see also Section 7.1.9.1.1.1, “Subdomain sample”). But this subdomain will required extra login.
Extract this .zip file to P:\dev\apps\qa.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and add a New system variable SONARQUBE_HOME pointing to P:\dev\apps\qa\sonarqube-6.7.1.
Change the context , the listening port and restricted to localhost by editing the configuration file P: \dev\apps\qa\sonarqube-6.7.1\conf\sonar.properties:
# WEB SERVER #sonar.web.javaOpts=-Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError sonar.web.javaOpts=-Xmx512m -XX:+HeapDumpOnOutOfMemoryError -server
# Binding IP address. #sonar.web.host=0.0.0.0 sonar.web.host=127.0.0.1
# Web context. #sonar.web.context= sonar.web.context=/sonarqube
# TCP port for incoming HTTP connections. #sonar.web.port=9000 sonar.web.port=8072
# Elasticsearch host. # Default is loopback address. #sonar.search.host=
17.2.2.1. Temp folder issue
When running the Java Service Wrapper as a Local System account the following error occurs “WrapperSim- pleApp: Encountered an error running main: java.lang.IllegalStateException: Temp directory is not writable: C: \Windows\system32\config\systemprofile\AppData\Local\Temp\”. To prevent this use This account to Log On (see also Section 17.2.6, “Windows service”) and edit the following entry in the service wrapper configuration file P:\dev\apps\qa\sonarqube-6.7.1\conf\wrapper.conf:
Don't forget to create it: mkdir %SONARQUBE_HOME%\temp-wrapper.
17.2.2.2. Rolling policy of log files
Enable the rolling policy and max number of files by editing the configuration file P:\dev\apps\qa \sonarqube-6.7.1\conf\sonar.properties:
# LOGGING
# Rolling policy of log files #sonar.log.rollingPolicy=time:yyyy-MM-dd sonar.log.rollingPolicy=time:yyyy-MM-dd
# Maximum number of files to keep if a rolling policy is enabled. #sonar.log.maxFiles=7 sonar.log.maxFiles=7
17.2.2.3. Database settings
Create a MySQL user called sonarqube-user and database called sonarqube with the following commands:
mysql -u root -p mysql> create database sonarqube character set utf8mb4 collate utf8mb4_unicode_ci; mysql> grant select, insert, update, delete, create, drop, alter, index on sonarqube.* -> to 'sonarqube-user'@'localhost' identified by 'password'; mysql> flush privileges; mysql> select host,user from mysql.user; mysql> select host,db,user from mysql.db; mysql> select * from information_schema.schemata where schema_name = 'sonarqube'; mysql> quit
Configure the SonarQube application server to connect to MySQL (instead of us- ing the H2 embedded database) - with url jdbc:mysql://localhost:3306/sonar- qube?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false by editing the configuration file P:\dev\apps\qa\sonarqube-6.7.1\conf\sonar.properties:
# DATABASE
# User credentials. sonar.jdbc.username=sonarqube-user sonar.jdbc.password=password
219 Chapter 17.
#----- Embedded Database (default) # H2 embedded database server listening port, defaults to 9092 #sonar.embeddedDatabase.port=9092
#----- MySQL 5.6 or greater # Only InnoDB storage engine is supported (not myISAM). # Only the bundled driver is supported. It can not be changed. #sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?… sonar.jdbc.url=jdbc:mysql://localhost:3306/sonarqube?… (see jdbc url above)
#----- Connection pool settings sonar.jdbc.maxActive=60
Don't forget to comment the lines and to deactivate the default embedded database settings. Drivers for the supported databases (except Oracle) are already provided (see also P:\dev\apps\qa\sonarqube-6.7.1\lib\jdbc\mysql). Do not replace the provided dri- vers; because they are the only ones supported.
17.2.3. Upgrading
After starting SonarQube with %SONARQUBE_HOME%\bin\windows-x86-64\StartSonar.bat or as Windows service browse to http://localhost:8072/sonarqube/setup or http://freedumbytes.dev.net/sonarqube/setup to upgrade the database (see also Upgrade guide).
Also browse to the Marketplace at http://freedumbytes.dev.net/sonarqube/admin/marketplace because of the following errors that might occur after upgrading:
1. “[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.0:sonar …: No quality profiles have been found, you probably don't have any language plugin installed.” when the Java Plugin is not installed. 2. “[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.0:sonar …: Un- able to register extension org.sonar.plugins.java.JavaSquidSensor: Lorg/sonar/api/checks/NoSonarFilter;: org.sonar.api.checks.NoSonarFilter” when the installed Java Plugin is too old.
17.2.4. SonarQube configuration
Verify the installation by running %SONARQUBE_HOME%\bin\windows-x86-64\StartSonar.bat.
Verify that the server is running in your browser at http://localhost:8072/sonarqube or in case of subdomain usage http://localhost:8072. Click Log in as admin with password admin. Click Administrator → My profile to Change password.
17.2.4.1. Marketplace
After logging in as admin select Administration → Marketplace and for example select the following plugins from the SonarQube Plugin Library:
• By default the SonarJava plugin is provided with SonarQube. You can upgrade it from the Marketplace or download the latest release from this page. • The same holds true for SVN Plugin and Git Plugin.
Or drop other plugins (that are not available in the Marketplace) or an external plugin (from for ex- ample excentia ) just in P:\dev\apps\qa\sonarqube-6.7.1\extensions\plugins or P:\dev\apps\qa \sonarqube-6.7.1\extensions\downloads:
• OWASP Dependency Check Plugin (download 1.1.0) integrates OWASP Dependency Check reports into Sonar- Qube. • 3D Code Metrics is a SonarQube extension which allows the visual representation of a software project in 3D in order to detect anomalies.
220 Quality Assurance
Restart SonarQube.
17.2.4.2. Settings
Select Administration → Configuration → General Settings and configure:
• General: • Server base URL: http://freedumbytes.dev.net/sonarqube and Save General Settings. • Email: • SMTP host: mail.freedumbytes.dev.net. • SMTP port: 25. • Use secure connection: No. • From address: sonarqube.noreply@company.org. • Email prefix: [SONARQUBE] and Save Email Settings. • Look & Feel: • Enable support of gravatars: False. • Logo URL: http://freedumbytes.dev.net/mvn-sites/maven-setup/images/lo- gos/shadowrun.png. • Width of image in pixels: 110 and Save Look & Feel Settings.
17.2.4.3. Users
To enable connecting from Eclipse with SonarLint select Administration → Security → Users and Create User:
The Quality Gates page is the place to define your requirements on projects. Select Quality Gates → SonarQube way and configure:
• When upgrading there might be no default set. In that case click Set as Default when for example SonarQube way is selected. • Delete condition New Blocker issues. • And replace it with Add condition: Blocker issues with Value that is greater than Errror Threshold 0.
17.2.4.5. Module Key
In case of the following error “org.sonar.api.utils.MessageException: Validation of project failed: o Module "net.dev.freedumbytes.manual.maven:java-se-3-base-pom" is already part of project "net.dev.freedumbytes.manual.maven:setup"” when it is for example moved to another project:
Figure 17.1. SonarQube background tasks
causing the background task to fail:
Figure 17.2. SonarQube background task log
To fix this issue just rename the old module key:
221 Chapter 17.
Figure 17.3. SonarQube module update key
17.2.5. Apache configuration
Instruct Apache to proxy all URLs whose path portions begin with /sonarqube/ using the following P:\dev \apps\httpserver\apache-conf\httpd-vhosts.conf include :
…
…
…
# SonarQube Include ../apache-conf/httpd-sonarqube.conf
with the following content in P:\dev\apps\httpserver\apache-conf\httpd-sonarqube.conf:
Include ../apache-conf/httpd-freedumbytes-realm.conf
Because since version 5.2 the server is in charge to process reports submitted by batch analysis, allow anony- mous access from the local area network to the following URIs in P:\dev\apps\httpserver\apache- conf\httpd-sonarqube.conf:
…
222 Quality Assurance
Include ../apache-conf/httpd-freedumbytes-realm.conf
Include ../apache-conf/httpd-lan-access.conf
Include ../apache-conf/httpd-lan-access.conf
Include ../apache-conf/httpd-lan-access.conf
Include ../apache-conf/httpd-lan-access.conf
To monitor the queue of pending reports to process and the history of past analysis select Administration → Projects → Background Tasks.
When the Apache HTTP Server is restarted you should be able to browse SonarQube at http:// freedumbytes.dev.net/sonarqube.
To test the anonymous access browse to:
• http://freedumbytes.dev.net/sonarqube/api/plugins/installed. • http://freedumbytes.dev.net/sonarqube/deploy/plugins/index.txt. • http://freedumbytes.dev.net/sonarqube/batch_bootstrap/index which should redirect to http:// freedumbytes.dev.net/sonarqube/batch/index. • http://freedumbytes.dev.net/sonarqube/batch/project.protobuf?key=groupId:artifactId.
Note
The last URI in case of https://sonarcloud.io/batch/project.protobuf?key=org.codehaus.sonar- plugins:maven-report will still return the following message “You're only authorized to exe- cute a local (preview) SonarQube analysis without pushing the results to the SonarQube server. Please contact your SonarQube administrator.” because a login token is required.
17.2.5.1. Login Token
Configure Maven with the sonarcloud.io token profile in the Maven configuration file P:\dev\apps\build \apache-maven-3.5.4\conf\settings.xml:
sonarCloudReplace with the value of your own tokenReplace with the value of your own organization key
17.2.6. Windows service
To restart automatically on Microsoft Windows, create a Windows service. Use the Tanuki Java Service Wrapper shipped with SonarQube.
223 Chapter 17.
Optionally edit the following entries - in the service wrapper configuration file P:\dev\apps\qa \sonarqube-6.7.1\conf\wrapper.conf:
# Path to JVM executable. By default it must be available in PATH. # Can be an absolute path, for example: #wrapper.java.command=/path/to/my/jdk/bin/java wrapper.java.command=P:\dev\apps\prg\java-x64\jdk1.8.0_162\bin\java
#******************************************************************** # Wrapper Windows Properties #******************************************************************** # Title to use when running as a console wrapper.console.title=SonarQubefreedumbytes.dev.net SonarQube
…
#******************************************************************** # Wrapper Windows NT/2000/XP Service Properties #******************************************************************** # WARNING - Do not modify any of these properties when an application # using this configuration file has been installed as a service. # Please uninstall the service before modifying this section. The # service can then be reinstalled.
# Name of the service wrapper.ntservice.name=SonarQubesonarqube
# Display name of the service wrapper.ntservice.displayname=freedumbytes.dev.net SonarQube
# Description of the service wrapper.ntservice.description=SonarQube is an open source quality management platform.
Register the SonarQube service with the command in an elevated command processor: %SONARQUBE_HOME%\bin \windows-x86-64\InstallNTService.bat. Start SonarQube with net start sonarqube.
Note
In case of an upgrade sc delete sonarqube first (see also Section 1.1.5, “Task Manager replacement” in case of “The specified service has been marked for deletion.”).
Important
In case of a new jdk installation use an elevated command prompt just run net stop sonar- qube and net start sonarqube.
Verify the installation and the JVM memory usage at http://freedumbytes.dev.net/sonarqube/system. 17.2.6.1. Log On Windows user
For the same reason as Section 8.1.3.1, “Log On Windows user” launch the Service Management Console with services.msc. Right click on freedumbytes.dev.net SonarQube and select Properties → Log On to use This account .\Tomcat. After password conformation click Apply and restart the service.
224 Quality Assurance
17.2.7. Maven integration
Configure Maven Setup project in C:\dev\projects\maven-setup\pom.xml to enable command line action mvn sonar:sonar (see also Analyzing with SonarQube Scanner for Maven). The SonarQube Maven Plugin trig- gers the code analyzers using Analysis Parameters and the SonarQube/SonarCloud Maven Report Plugin adds a re- port link to the Maven site, that redirects to the project dashboard in SonarQube (see also Section 17.2.5, “Apache configuration” about allowing anonymous access to SonarQube for the batch analysis to submit the process re- ports):
• [WARNING] Missing blame information for the following files:pom.xml. This may lead to missing/broken fea- tures in SonarQube.
Even when the SCM provider for a project is: git and thus all information should be available this message is sometimes listed. And apparently it is used because it is displayed in the program listing of for example SonarReportMojo.
Tip
For single user projects just disable this information clutter with sonar.scm.disabled.
• [ERROR] Reports path not found or is not a directory: target\surefire-reports.
[WARNING] Reports path contains no files matching TEST-.*.xml : target\failsafe-reports.
Due to the fact that Surefire doesn't create the surefire-reports folder when there aren't any unit tests. When there aren't any integration tests Failsafe does create the failsafe-reports folder, but only with the failsafe-summary.xml in it.
• [ERROR] Reports path not found or is not a directory: target\surefire-reports.
225 Chapter 17.
[ERROR] Reports path not found or is not a directory: target\failsafe-reports.
Similar to the error mentioned just above, but due to for example mvn clean or a separate GitLab/Jenkins CI build job for mvn sonar:sonar for which test result dependencies weren't restored/unstashed.
• [WARNING] Property 'sonar.jacoco.reportPath' is deprecated. Please use 'sonar.jacoco.reportPaths' in- stead.
[WARNING] Property 'sonar.jacoco.itReportPath' is deprecated. Please use 'sonar.jacoco.reportPaths' instead.
Since version 6.2 there is no longer a differentiation between unit tests, integration tests, et cetera.
• [WARNING] Property 'sonar.jacoco.reportMissing.force.zero' is deprecated and its value will be ignored.
• [WARNING] Dependency-Check report does not exist. SKIPPING. Please check property sonar.dependencyCheck.reportPath: target/owasp-reports/dependency-check-report.xml.
[WARNING] Dependency-Check report does not exist. SKIPPING. Please check property sonar.dependencyCheck.reportPath: target/owasp-reports/dependency-check- report.html.
Due to org.owasp:dependency-check-maven:aggregate usage these warnings are displayed for mod- ules.
• [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.4.0.905:sonar (de- fault-cli) on project : Please provide compiled classes of your project with sonar.java.binaries proper- ty.
Due to the fact that the target folder doesn't contain a classes folder after for example mvn clean or a separate GitLab/Jenkins CI build job for mvn sonar:sonar for which compiled class dependencies weren't restored/unstashed. See also properties mentioned below, which should not be configured for Maven.
• [WARNING] Classes not found during the analysis : [org.apache.commons.logging.Log].
Due to using target/**/*.jar with Sonar Maven Plugin. See also properties mentioned below, which should not be configured for Maven.
• [ERROR] Invalid value for sonar.java.libraries.
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.4.0.905:sonar (de- fault-cli) on project : No files nor directories matching 'target/**/*.jar'.
Due to using target/**/*.jar with Sonar Maven Plugin and the fact that the target folder doesn't contain even the module jar after for example mvn clean or a separate GitLab/Jenkins CI build job for mvn sonar:sonar. See also properties mentioned below, which should not be configured for Maven.
• [WARNING] Bytecode of dependencies was not provided for analysis of source files, you might end up with less precise results. Bytecode can be provided using sonar.java.libraries property.
Due to using with Sonar Maven Plugin. See also properties mentioned below, which should not be configured for Maven.
• [WARNING] Property 'sonar.php.file.suffixes' is not declared as multi-values/property set but was read using 'getStringArray' method. The SonarQube plugin declaring this property should be updated.
Some code change needed for one of the Code Analyzers: SonarPHP.
Important
The following Sonar Java Plugin properties should not be configured for Maven usage:
SonarQube screenshots with the information of the OWASP Dependency Check incorporated for project Cus- tom Dependency Check with suppressionFile and hintsFile containing the rules that control how to suppress false positives and resolve false negatives, which happens to have a sample module and test vulnerabilities included:
The Dependency Check measures are there, but they are not fully integrated.
229 Chapter 17.
Figure 17.8. SonarQube Project More - Custom Dependency Check
Important
Only the project parent has a submenu More with the OWASP Dependency-Check Re- port. Thus for a complete list of all vulnerabilities in a project run the Maven Plugin org.owasp:dependency-check-maven with the goal aggregate instead of check. Alas, this way the project module will not show its Dependency Check vulnerabilities. But otherwise the ones in the project parent would be counted double (or more depending on the number of modules) in the SonarQube Project Overview anyway.
Figure 17.16. Maven Quality Assurance Sandbox SonarQube Code Violations Drilldown (all violations at one glance)
17.3. jDocBook
The purpose of the jDocBook Plugin is to allow the DocBook transformations to occur as a natural part of the users Maven build. It is used to create this documentation from the master.xml (for more resources see also Section 5.1, “DocBook”). 17.3.1. Docbook configuration
Lets setup a base configuration, for Maven Docbooks to come, with the following JIRA project:
• Maven Docbook [MVNDBK]: Maven DocBook configuration (see also repository branches/maven-docbook).
With the following components: 1. docbook: Common setup for all DocBook documentation. 2. example: Pressgang Tools example.
and versions: • 3.0-alpha - HTML and PDF generation with Pressgang Tools example.
Start a new bare Git repository maven-docbook by calling /p/dev/apps/windows/batch/git-cre- ate-repo.sh /p/dev/data/repo/git maven-docbook.git "Maven docbook configuration" - jenkinsHook in Git Bash.
Clone this remote repository with the following command git-clone jjasper maven-docbook.
Start out with the Look and Feel of the PressGang, which is the centralized hub JBoss projects can use to get assistance with documentation. If you have any questions about writing professional open source documentation using DocBook XML, you've come to the right place to have them answered.
Create the maven-docbook initial settings POM in C:\dev\projects\maven-docbook\pom.xml:
In Eclipse create a Working Set Maven Docbook and import this Maven Docbook project into it with File → Import… → Maven → Existing Maven Projects from Root directory C:\dev\projects\maven-docbook and click Finish.
With this setup the Development Production Line manual would look something like this:
236 Quality Assurance
Figure 17.17. Development Production Line - PressGang Look and Feel
Important
When building a jDocBook using maven in combination with java.home pointing to a jdk1.7 the following error occured: “Failure reading http://docbook.sourceforge.net/release/xsl/1.76.1/ xhtml/chunk.xsl: Connection timed out: connect”. Just update MAVEN_OPTS as mentioned in the Maven installation guide.
17.3.2. Documentation setup
Lets setup a base structure, for Maven Documentation Customization, with the following JIRA project:
• Maven Docs [MVNDCS]: Maven documentation setup (see also repository branches/maven-docs).
With the following components: 1. docs: Documentation setup. 2. style: Custom HTML and PDF styling. 3. xslt: Custom XHTML and PDF XSLTs. 4. jdocbook: Maven jDocBook plugin adds support for DocBook handling to Maven. 5. fop: The Apache FOP (Formatting Objects Processor) is a print formatter driven by XSL formatting objects (XSL-FO) and an output independent formatter.
and versions: • 1.0-alpha - Custom Look and Feel. • 2.0-alpha - Fluido Look and Feel.
Note
This second project is necessary because the custom style and xslt, that will be defined in maven-docs and used in maven-docbook, cause a chicken and egg problem because those artifacts are referenced as dependencies for the jdocbook plugin during the build.
Start a new bare Git repository maven-docs by calling /p/dev/apps/windows/batch/git-create-repo.sh /p/dev/data/repo/git maven-docs.git "Maven documentation setup" -jenkinsHook in Git Bash.
Clone this remote repository with the following command git-clone jjasper maven-docs.
237 Chapter 17.
Supply some information in C:\dev\projects\maven-docs\src\site\apt\index.apt.vm:
+Free Dumb Bytes documentation setup + + {{{http://www.docbook.org/} DocBook}} is, in part, . + Another aspect of DocBook is the rendering of that content into various formats using DocBook-supplied (or custom) XSLT stylesheets. + Basically, DocBook allows you to write and maintain a single source for documentation, and to then render that single source into multiple formats such as PDF or HTML. + + {{{http://docbook.sourceforge.net/} The DocBook Project} supports the open-source development of a variety of DocBook resources; in particular, {{{http://wiki.docbook.org/DocBookXslStylesheets} the DocBook XSL stylesheets}}. + + {{{http://pressgang.jboss.org/} PressGang}} is {{{https://github.com/pressgang} the home of PressGang}} CCMS and the JBoss Base Styles. We are here to help you write awesome user documentation that meets the needs of your users. + + The purpose of {{{http://maven-jdocbook-plugin.jboss.org/} the jDocBook Plugin}} is to allow these DocBook transformations to occur as a natural part of the users Maven build. + The main difficulty with this has always been the fact that DocBook transformations are usually very closely tied to the user's local environment. + The design goal with writing this plugin was to utilize Maven's dependency mechanism to bring all the pieces together on demand. Those pieces are: + + * {{{https://github.com/pressgang/maven-jdocbook-plugin} Maven jDocBook plugin}} adds support for DocBook handling to Maven. + + * {{{https://github.com/pressgang/maven-jdocbook-style-plugin} Maven Docbook Style Plugin}} defines the custom packaging. + + * {{{https://github.com/pressgang/pressgang-tools} Pressgang Tools}} for all artifacts which support JBoss Community documentation. + + * {{{https://github.com/…/…-tools/tree/master/pressgang-jdocbook-style} JBoss Community Documentation Style}} for Blue and Grey CSS and Images to use as a base for your project XHTML JBoss Community Documentation. + + * {{{https://github.com/…/…-tools/tree/master/pressgang-xslt-ns} JBoss Community DocBook XSLT}} is the main project to transform DocBook XML into XHTML and other formats for JBoss Community Documentation. Use this with XSD based docbook XML files. + + * {{{https://github.com/…/…-tools/tree/master/pressgang-highlight} JBoss Community DocBook Code Highlighter}} provides code highlighting for documentation examples. Currently uses the jHighlight library. + + * {{{https://github.com/pressgang/pressgang-tools/tree/master/pressgang-fonts} JBoss.org PDF Fonts}} contains both the fonts themselves and the xslt templates necessary to get them included into the generated PDFs (simply import/include this xslt into your project's pdf xslt). + + [] + + <>: To create the intermediate FO itself execute + <<>> (see also GenerateXslFoMojo.java). + + {{{https://xmlgraphics.apache.org/commons/} Apache XML Graphics Commons}} is a library that consists of several reusable components used by Apache Batik and Apache FOP. Many of these components can easily be used separately outside the domains of SVG and XSL-FO. + You will find components such as a PDF library, an RTF library, Graphics2D implementations that let you generate PDF & PostScript files, and much more. + + {{{http://xmlgraphics.apache.org/batik/} Batik}} is a Java-based toolkit for applications or applets that want to use images in the Scalable Vector Graphics (SVG) format for various purposes, such as display, generation or
238 Quality Assurance
manipulation. + + The {{{https://xmlgraphics.apache.org/fop/} Apache FOP}} (Formatting Objects Processor) is a print formatter driven by XSL formatting objects (XSL-FO) and an output independent formatter. + It is a Java application that reads a formatting object (FO) tree and renders the resulting pages to a specified output. + Output formats currently supported include PDF, PS, PCL, AFP, XML (area tree representation), Print, AWT and PNG, and to a lesser extent, RTF and TXT. + The primary output target is PDF. + + {{{https://xmlgraphics.apache.org/fop/trunk/hyphenation.html} Apache FOP Hyphenation}} uses Liang's hyphenation algorithm, well known from {{{https://tug.org/} TeX}}. + It needs language specific pattern and other data for operation. + + Because of licensing issues (and for convenience), all hyphenation patterns for FOP are made available through the OFFO ({{{http://offo.sourceforge.net/hyphenation/index.html} Objects For Formatting Objects}}) project.
Figure 17.18. Free Dumb Bytes Documentation Setup
Create the maven-docs minimal project base POM in C:\dev\projects\maven-docs\pom.xml:
In Eclipse create a Working Set Maven Docs and import this Maven Docs project into it with File → Import… → Maven → Existing Maven Projects from Root directory C:\dev\projects\maven-docs and click Finish.
Make an anynomous clone from the pressgang-tools with:
• MVNDBK-1: Create the minimal POM with PressGang Look and Feel (see also commits for PressGang Tools). • MVNDCS-1: Create the minimal POMs for custom Look and Feel modules. • MVNDCS-2: Initialize custom style based on PressGang style. • MVNDCS-3: Initialize custom xslt based on PressGang xslt. • MVNDBK-2: Switch to custom Look and Feel artifacts. • MVNDCS-4: Refactor initial PressGang xslt. • MVNDCS-5: Refactor initial PressGang style. • MVNDBK-3: Switch to refactored XSLs. • MVNDCS-7: Reenable status draft. • MVNDCS-8: Customize admonitions. • MVNDCS-9: Customize list style. • MVNDCS-10: Customize code style. • MVNDCS-11: Customize link style. • MVNDCS-12: Support role strikethrough. • MVNDCS-13: Support element accel. • MVNDCS-14: Wrap long lists of menu choices. • MVNDCS-15: Screen and Printer output improvements. • MVNDCS-16: Customize community imagery. • MVNDCS-17: Highlight chapters and appendices in the table of contents. • MVNDCS-18: Reenable unused callout XSLs for xHTML. • MVNDCS-19: Fix unwanted line breaks before anchors. • MVNDCS-20: Customize navigation. • MVNDCS-21: Highlight gui, application and command elements. • MVNDCS-22: Customize title page. • MVNDCS-23: Customize pagesetup. • MVNDCS-24: Uniform menuchoice separator. • MVNDCS-25: Create security confidential mode. • MVNDCS-26: Highlight xref links. • MVNDBK-4: Activate hyphenation. • MVNDCS-28: Stop HTML generation from outputing a 'title' attribute. • MVNDCS-29: Switch to Fluido Skin style. • MVNDCS-30 Upgrade DocBook XSL Version 1.78.1. • MVNDCS-31 Upgrade FOP Version 1.0. • MVNDCS-32 Fix programlisting.xsl highlighting with hightlight.js (xHTML only). • MVNDCS-34 Fix programlisting code wrapping (which causes it to look ugly) by simply not auto-wrapping but adding a horizontal scrollbar. • MVNDCS-35 Large images should auto scale to fit witdh and auto scale height of inline images for PDF output. • MVNDCS-36 Fix console-en-US-pdf.log output errors and warnings. • MVNDCS-37 Title must be bigger then subtitle. • MVNDCS-38 Add cover support to title page. • MVNDBK-8: Copy and extend PressGang example. • MVNDCS-39: Upgrade FOP version 1.1 using a patch. • MVNDCS-40: Fonts information and usage.
243 Chapter 17.
• MVNDCS-44: Overrule table pgwide=1 required for PDF but unwanted for HTML with width attribute. • MVNDCS-43: Suppress the overkill of FOP 'Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".' and similar DEBUG messages by switching back to FOP LoggingEventListener an EventListener implementation that redirects events to Commons Logging. • MVNDCS-45: [PATCH] FontInfo.findAdjustWeight cannot find Bold weight fonts. • MVNDCS-42: Header anchor on hover based on github-markdown.css of github-markdown-css.
Important
Processing DocBook V5.0 with XSL Stylesheets or XSL-NS Stylesheets (namespace-aware ver- sions). The DocBook XSL-NS stylesheets are released side-by-side with the DocBook XSL stylesheets, as a separate docbook-xsl-ns package. They are the recommended XSLT 1.0 stylesheets to use for transforming namespaced (DocBook V5.0) documents. But when parsing xhtml-chunk.xsl still using the old import http://docbook.sourceforge.net/re- lease/xsl/1.78.1/xhtml5/chunk.xsl instead of recommended http://docbook.sourceforge.net/ release/xsl-ns/1.78.1/xhtml5/chunk.xsl the documents are rendered about 2-3 times faster.
With this custom setup
Figure 17.19. Development Production Line - Custom Look and Feel
Important
When building a jDocBook using maven in combination with java.home pointing to a jdk1.7 the following error occured: “Failure reading http://docbook.sourceforge.net/release/xsl/1.76.1/ xhtml/chunk.xsl: Connection timed out: connect”. Just update MAVEN_OPTS as mentioned in the Maven installation guide.
17.3.3. Hyphenation
To activate hyphenation for the jDocBook Plugin define the version in C:\dev\projects\maven-docs \pom.xml:
2.0.0
244 Quality Assurance
1.76.13.1.3 + 2.0
And add a dependency for it in C:\dev\projects\maven-docbook\pom.xml:
The new version of docbook-xsl is already included with this version property docbookXs- lVersion in the above mentioned custom-docbook-xslt artifact, that also was already added as a custom dependency to maven-jdocbook-plugin. Because of this old entry the directo- ry C:\dev\data\repo\maven\local\net\sf\docbook\docbook-xsl\1.76.1 will still
245 Chapter 17.
contain a pom, but C:\dev\data\repo\maven\local\net\sf\docbook\docbook-xsl \1.78.1 will contain both pom and zip.
17.3.5. Upgrade FOP
To upgrade the fop dependency (and a little later on the avalon-framework transitive dependencies) define the new version in C:\dev\projects\maven-docs\pom.xml:
1.78.13.1.32.0 + 1.1 + 4.3.1
And add a dependency for it in C:\dev\projects\maven-docbook\pom.xml:
Because for the artifacts of the avalon-framework the groupId changed from avalon-frame- work to org.apache.avalon.framework as of version 4.3.1 we end up with the follow- ing error “[ERROR] Unresolveable build extension: Plugin org.jboss.maven.plugins:maven-jdocbook-plu- gin:2.3.9 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.avalon.framework:avalon-framework-api:jar:4.2.0, org.apache.avalon.framework:avalon- framework-impl:jar:4.2.0: Failure to find org.apache.avalon.framework:avalon-framework-api:jar:4.2.0 in http:// freedumbytes.dev.net/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of nexus-central has elapsed or updates are forced” when trying to mvn install the maven-docbook project.
To fix this issue just add the avalon-framwork artifacts as dependencies for the jdocbook-core plugin at the same level as the fop artifact in C:\dev\projects\maven-docbook\pom.xml:
To suppress the remaining warnings “[WARNING] The POM for org.apache.avalon.framework:avalon- framework-api:jar:4.2.0 is missing, no dependency information available” and “[WARNING] The POM for org.apache.avalon.framework:avalon-framework-impl:jar:4.2.0 is missing, no dependency information available” exclude them in C:\dev\projects\maven-docbook\pom.xml:
With this setup the next error is “[ERROR] Failed to execute goal org.jboss.maven.plugins:maven-jdoc- book-plugin:2.3.9:resources (default-resources) on project development-production-line: Unexpected problem: Unable to locate artifact [org.apache.avalon.framework:avalon-framework-api:jar:4.2.0]: Could not find arti- fact rg.apache.avalon.framework:avalon-framework-api:jar:4.2.0 in nexus-central (http://freedumbytes.dev.net/ nexus/content/groups/public)” when trying to generate this manual with mvn package. To fix this change the dependencyManagement in C:\dev\projects\maven-docs\pom.xml:
Alas the next error “[ERROR] Failed to execute goal org.jboss.maven.plugins:maven-jdoc- book-plugin:2.3.9:generate (default-generate) on project development-production-line: Rendering problem: er- ror building transformation result [C:\dev\projects\dpl-manual\target\docbook\publish\en-US\pdf\Production evelopment Line (en-US).pdf]: org.apache.fop.fonts.FontSetup.createMinimalFontResolver()Lorg/apache/fop/ fonts/FontResolver;” is caused by a change of the method public static FontResolver createMinimal- FontResolver() into public static FontResolver createMinimalFontResolver(boolean useC- omplexScriptFeatures) in the class FontSetup since fop version 1.1.
Thus upgrade the fop dependency only to 1.0 in C:\dev\projects\maven-docs\pom.xml for now:
1.78.13.1.32.0 + 1.0 + 4.3.1
17.3.5.1. Patch FOP version
Alternatively start a new bare Git repository custom-jdocbook-core by calling /p/dev/apps/win- dows/batch/git-create-repo.sh /p/dev/data/repo/git custom-jdocbook-core.git "Patch the core jDocBook functionality to support fop 1.1." -jenkinsHook in Git Bash (see also repository branches/custom-jdocbook-core).
Therefore upgrade the fop to 1.1 in C:\dev\projects\maven-docs\pom.xml first:
1.78.13.1.32.0 - 1.0 + 1.14.3.1
Create the custom-jdocbook-core project POM in C:\dev\projects\custom-jdocbook-core\pom.xml:
This way we can always revert to the original version of this jar in the Nexus JBoss repository by deleting the patched jar from the local and Nexus releases repositories.
The actual patch to fix “[ERROR] … org.apache.fop.fonts.FontSetup.createMinimalFontResolver()Lorg/apache/ fop/fonts/FontResolver;” when running jDocbook core with fop 1.1 is made to FopConfigHelper:
FontCache fontCache = componentRegistry.getConfiguration().isUseFopFontCacheEnabled() ? FontCache.loadFrom( new File( getFopWorkDirectory(), FONT_CACHE_FILE ) ) : new FontCache(); - FontResolver fontResolver = FontSetup.createMinimalFontResolver(); + FontResolver fontResolver = FontSetup.createMinimalFontResolver(true); FontInfoFinder fontInfoFinder = new FontInfoFinder(); fontInfoFinder.setEventListener( new FontEventListener() { … + + @Override + public void fontDirectoryNotFound(Object source, String dir) { + log.trace("Font directory not found dir=[" + dir + + "]; source=" + source); + } + + @Override + public void svgTextStrokedAsShapes(Object source, String fontFamily) { + log.trace("SVG text stroked as shapes fontFamily=[" + fontFamily + + "]; source=" + source); + } } );
251 Chapter 17.
for ( File fontDirectory : componentRegistry.getEnvironment().getFontDirectories() ) {
17.3.5.2. Patch FOP logging
The actual patch to suppress the overkill of FOP “Font "Symbol,normal,700" not found. Substituting with "Symbol,normal,400".” DEBUG messages when running jDocbook core with fop 1.1 is made to ResultImpl:
With the upcoming pressgang-tools-example this fix reduced the number of logged messages in C:\dev\projects\maven-docbook\pressgang-tools-example\target\docbook\work\log\con- sole-en-US-pdf.log from 4590 to 220. 17.3.5.3. Patch FOP font weight
Start a new bare Git repository custom-fop by calling /p/dev/apps/windows/batch/git-create-repo.sh /p/dev/data/repo/git custom-fop.git "Patch the core FOP functionality." -jenkinsHook in Git Bash (see also repository branches/custom-fop).
The actual patch to fix bold weight font with fop 1.1 is made to FontInfo (see also bug 49301):
In metal typesetting, a font is a particular size, weight and style of a typeface. Each font was a matched set of type, one piece (called a sort) for each glyph, and a typeface consisting of a range of fonts that shared an overall design.
In modern usage, with the advent of digital typography, font is frequently synonymous with typeface, although the two terms do not necessarily mean the same thing. In particular, the use of vector or outline fonts means that different sizes of a typeface can be dynamically generated from one design. Each style may still be in a separate font file—for instance, the typeface Bulmer may include the fonts Bulmer roman, Bulmer italic, Bulmer bold and Bulmer extended—but the term font might be applied either to one of these alone or to the whole typeface.
In typography, a typeface (also known as font family) is a set of one or more fonts each composed of glyphs that share common design features. Each font of a typeface has a specific weight, style, condensation, width, slant, italicization, ornamentation, and designer or foundry (and formerly size, in metal fonts).
Complex text layout (abbreviated CTL) or complex text rendering refers to the typesetting of writing systems in which the shape or positioning of a grapheme depends on its relation to other graphemes. The term is used
252 Quality Assurance
in the field of software internationalization, where each grapheme is a character. Scripts which require CTL for proper display may be known as complex scripts. Examples include the Arabic alphabet and scripts of the Brahmic family, such as Devanagari or the Thai alphabet. Many scripts do not require CTL. For instance, the Latin alphabet or Chinese characters can be typeset by simply displaying each character one after another in straight rows or columns. However, even these scripts have alternate forms or optional features (such as cursive writing) which require CTL to produce on computers.
ISO 639 is a standardized nomenclature used to classify all known languages. Each language is assigned a two-let- ter (639-1) and three-letter (639-2 and 639-3), lowercase abbreviation, amended in later versions of the nomen- clature. The system is highly useful for linguists and ethnographers to categorize the languages spoken on a re- gional basis, and to compute analysis in the field of lexicostatistics. 17.3.6.1. Resources
• Google Noto Fonts (Updates). • Asciidoc Book Editor based on JavaFX 8 with How To install Microsoft Core Fonts on Linux OSes. • How to Install, Remove and Hide Fonts in Windows 7 & Windows 8.1. • Although some people will never use anything other than the default fonts, Microsoft provides a wide range of fonts with Windows: • Arial Unicode MS the extended version of Monotype's Arial contains glyphs for all code points within The Unicode Standard, Version 2.0. • The Verdana typeface family consists of four TrueType fonts created specifically to address the challenges of on-screen display. • Although inspired by the need for - and providing - clarity at low resolutions on the screen, Georgia is a typeface resonant with typographic personality. • Meiryo is a very versatile modern sans serif type designed to give an exceptionally clean appearance on screen, as well as in print. It is optimized for on-screen reading. • Ubuntu font family is a unique, custom designed font that has a very distinctive look and feel. • Papyrus is one of those fonts that are far too popular for their own good. • Lucida is a font family that includes serif, sans-serif and handwritten variants. • Guide to the LaTeX markup language. • Custom Font List for Unicode 8.0. • GitHub Octicons. • Specials is the name of a short Unicode block allocated at the very end of the Basic Multilingual Plane, at U +FFF0–FFFF. • Brahmi is the modern name given to one of the oldest writing systems used in the Indian subcontinent and in Central Asia during the final centuries BCE and the early centuries CE. Like its contemporary, Kharosthi, which was used in what is now Afghanistan and Pakistan, it is an abugida. • Burmese script is used in Myanmar for the Burmese language, and for the country's minority languages. It is also used for the liturgical languages of Pali and Sanskrit. • Javanese (Aksara Jawa) is one of the Austronesian languages, but it is not particularly close to other languages and is difficult to classify. Its closest relatives are the neighbouring languages such as Sundanese, Madurese and Balinese. • Tagalog grammar is the body of rules that describe the structure of expressions in the Tagalog language, the language of the Tagalog region of the Philippines. • Sundanese script (Aksara Sunda) is a writing system which is used by the Sundanese people. It is built based on Old Sundanese script (Aksara Sunda Kuno) which was used by the ancient Sundanese between the 14th and 18th centuries. 17.3.6.2. Additional fonts usage
When text is rendered by a computer, sometimes there will be characters in the text that can not be displayed, because no font that supports them is available to the computer. When this occurs, small boxes are shown to rep- resent the characters. We call those small boxes tofu [for example: ] and we want to remove tofu from the Web. This is how the Noto font families got their name. Noto is a font family designed to cover all the scripts encoded in the Unicode standard. It is designed with the goal of achieving visual harmony (e.g., compatible heights and stroke thicknesses) across multiple languages/scripts. Like Open Sans, it is derived from Droid fonts. Commissioned by Google, the font is licensed under the SIL Open Font License.
Microsoft and Noto Fonts counterparts:
• Meiryo or Noto Sans. • Arial or Noto Sans. • Euphemia or Noto Sans Canadian Aboriginal. • Arial or Noto Kufi Arabic. • Arial or Noto Sans Hebrew. • Batang / Gulim or Noto Sans CJK KR. • MV Boli or Noto Sans Thaana. • DokChampa or Noto Sans Lao. • Microsoft Himalaya / Dzongkha or Noto Sans Tibetan. • Gautami / Vani or Noto Sans Telugu. • Microsoft Himalaya or Noto Sans Tibetan. • Iskoola Pota or Noto Sans Sinhala. • Kalinga or Noto Sans Oriya. • Kartika or Noto Sans Malayalam.
253 Chapter 17.
• Latha / Vijaya or Noto Sans Tamil. • Leelawadee / DokChampa or Noto Sans Thai. • Mangal or Noto Sans Devanagari. • Meiryo or Noto Sans CJK JP. • MoolBoran or Noto Sans Khmer. • Nyala or Noto Sans Ethiopic. • Raavi or Noto Sans Gurmukhi. • SimSun or Noto Sans CJK SC / TC. • Shruti or Noto Sans Gujarati. • Sylfaen or Noto Sans Armenian. • Sylfaen or Noto Sans Georgian. • Tunga or Noto Sans Kannada. • Vrinda or Noto Sans Bengali. • Microsoft Yi Baiti or Noto Sans Yi. • Noto Sans Tai Viet.
tofu or Noto Fonts only:
• Noto Sans Javanese. • Noto Sans Myanmar. • Noto Sans Sundanese. • Noto Sans Tagalog.
After installing the necessary fonts, start using them in the DocBook generated HTML with the follow- ing change in C:\dev\projects\maven-docs\custom-docbook-style\src\docbook\custom-css\css \freedumbytes.css:
body { background-image: url(../images/community/docbook/paper.jpg); - font-family: 'Lucida Grande', Geneva, Verdana, Arial, sans-serif; + font-family: + Ubuntu, + Noto Sans, + Noto Sans Canadian Aboriginal, + Noto Kufi Arabic, + Noto Sans Hebrew, + Noto Sans CJK KR, + Noto Sans CJK JP, + Noto Sans CJK TC, + Noto Sans CJK SC, + Noto Sans Lao, + Noto Sans Armenian, + Noto Sans Bengali, + Noto Sans Devanagari, + Noto Sans Ethiopic, + Noto Sans Georgian, + Noto Sans Gujarati, + Noto Sans Gurmukhi, + Noto Sans Kannada, + Noto Sans Khmer, + Noto Sans Malayalam, + Noto Sans Oriya, + Noto Sans Sinhala, + Noto Sans Tai Viet, + Noto Sans Tamil, + Noto Sans Telugu, + Noto Sans Thaana, + Noto Sans Thai, + Noto Sans Tibetan, + Noto Sans Yi, + Noto Sans Javanese, + Noto Sans Myanmar, + Noto Sans Sundanese, + Noto Sans Tagalog, + Euphemia, + SimSun, + DokChampa, + Batang, + Gulim, + Microsoft Himalaya, + Dzongkha, + MV Boli, + Gautami, + Vani, + Iskoola Pota,
Start using them in the DocBook generated PDF with the following change in C:\dev\projects\maven-docs \custom-docbook-xslt\src\main\resources\xslt\fo\font.xsl:
+ Mangal, + MoolBoran, + Nyala, + Raavi, + Shruti, + Sylfaen, + Tunga, + Vijaya, + Vrinda, + Microsoft Yi Baiti, + Meiryo, + Lucida Sans Unicode, + Georgia, + Verdana, + Arial Unicode MS, + Arial, + Geneva, + Helvetica, + Noto Sans, + Noto Sans Canadian Aboriginal, + Noto Kufi Arabic, + Noto Sans Hebrew, + Noto Sans CJK KR, + Noto Sans CJK JP, + Noto Sans CJK TC, + Noto Sans CJK SC, + Noto Sans Lao, + Noto Sans Armenian, + Noto Sans Bengali, + Noto Sans Devanagari, + Noto Sans Ethiopic, + Noto Sans Georgian, + Noto Sans Gujarati, + Noto Sans Gurmukhi, + Noto Sans Kannada, + Noto Sans Khmer, + Noto Sans Malayalam, + Noto Sans Oriya, + Noto Sans Sinhala, + Noto Sans Tai Viet, + Noto Sans Tamil, + Noto Sans Telugu, + Noto Sans Thaana, + Noto Sans Thai, + Noto Sans Tibetan, + Noto Sans Yi, + Noto Sans Javanese, + Noto Sans Myanmar, + Noto Sans Sundanese, + Noto Sans Tagalog, + sans-serif +
+ + + Ubuntu, + Meiryo, + Lucida Sans Unicode, + Georgia, + Arial Unicode MS, + Arial, + Noto Serif, + Noto Naskh Arabic, + Noto Serif Lao, + Noto Serif Armenian, + Noto Serif Georgian, + Noto Serif Khmer, + Noto Serif Thai, + serif + +
256 Quality Assurance
+ + +
+ + + Symbol, + ZapfDingbats +
For the fonts to be auto detected by FOP and thus displayed in the DocBook generated PDF (see also Figure 17.20, “PDF without auto detect fonts”), edit the maven-docbook project as follows in C:\dev\projects\maven- docbook\pom.xml:
For now PDF selects the Microsoft fonts prior to those of Noto because of a baseline shift (see also Figure 17.21, “PDF with Sans Noto fonts” and Figure 17.22, “PDF with Microsoft fonts”).
Figure 17.20. PDF without auto detect fonts
Figure 17.21. PDF with Sans Noto fonts
257 Chapter 17.
Figure 17.22. PDF with Microsoft fonts
17.3.7. PressGang Example
Create the pressgang-tools-example module in C:\dev\projects\maven-docbook\pressgang-tools-ex- ample\pom.xml:
+ + + + 4.0.0 + + + net.dev.freedumbytes.documentation + docbook + 3.0-alpha + + + pressgang-tools-example + jdocbook + + Pressgang Tools Example (${translation}) + + Shows how to use the pressgang styles in a maven project. + Also shows how each docbook element looks like. + + 2015
And add for example the resulting single html to the Maven site:
Wagon Plugin with site-deploy requires usage of phase site-deploy instead of goals site:site site:deploy during the Jenkins build (see also Build / Goals and Options)
17.3.8. Publication
Make sure all local changes have been committed with commands for both maven-docs and maven-docbook:
git status git log --oneline --decorate
Also check if all remote changes are “up to date” with command:
259 260 Appendix A. Appendix A. Tips & Tricks A.1. Archives A.1.1. How to extract content from MSI files
While there are plenty of utilities available to extract the contents of an MSI file, you can do it straight from the command line: msiexec /a file.msi /qb TARGETDIR=folder. A.1.2. How to determine absolute path of a loaded Class
Of course, this is not always possible (if you think of dynamically created classes), but if the loaded Class is inside a jar how to get the absolute path for this jar?
public class ClassloaderTest { @Test() public void testClassloaderResourceRequiresRelativePath() { assertThat(getClass().getClassLoader().getResource("/net/dev/freedumbytes/ maven/qa/sandbox/classloader/ClassloaderTest.class")).isNull();
A.2. Version Control System A.2.1. GitLab Repository Mirroring
Repository mirroring is a way to mirror repositories from external sources. In the GitLab project just select Set- tings → Repository → Pull from a remote repository → Expand and configure for example:
In case the original repository isn't update regularly uncheck Mirror repository after it is mir- rored. This prevents unnecessary scheduled update jobs. Keep Trigger pipelines for mirror updates checked otherwise the build badge is unknown.
262 Tips & Tricks
Figure A.1. Sonar Maven Report mirror
When Mirror repository is still checked, it is also possible to manually force an update with Repository → Com- mits → Update Now:
263 Appendix A.
Figure A.2. Sonar Maven Report mirror Update Now
A.2.2. SVN to Git Migration
Migrating from Subversion to Git with svn2git, a Ruby tool for importing existing svn projects into git.
svn2git is a tiny utility for migrating projects from Subversion to Git while keeping the trunk, branches and tags where they should be. It uses git-svn to clone an svn repository and does some clean-up to make sure branches and tags are imported in a meaningful way, and that the code checked into master ends up being what's currently in your svn trunk rather than whichever svn branch your last commit was in.
A.2.2.1. Resources
• Migrating from Subversion to Git with svn2git on Windows (the tricky bits explained). • Issue #121 "--exclude" option does not work on Windows. • Issue #20 --exclude works great, but is there a way to not include the empty commits? • Dealing with line endings. • Customizing Git - Git Attributes from the Pro Git book. • git-config - Get and set repository or global options. • Mind the End of Your Line. • Trying to fix line-endings with git filter-branch, but having no luck. • Git lineendings and better diffs • A Collection of Useful .gitattributes Templates. • A Collection of .gitignore templates.
A.2.2.2. Ruby installation guide
Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write.
Download the Ruby Installer rubyinstaller-2.3.3-x64.exe for Windows [version 2.3.3].
Run this .exe file to install Ruby in P:\dev\apps\prg\ruby-2.3.3.
Press +Break keys to open the Windows System Properties. Select Advanced system settings → Environment Variables and Edit the Path to append %RUBY_HOME%\bin;. Also add a New system variable RUBY_HOME point- ing to P:\dev\apps\prg\ruby-2.3.3.
Verify the installation with ruby --version.
A.2.2.3. RubyGems upgrade guide
RubyGems is a package management framework for Ruby.
Download the archive rubygems-2.6.12.zip for Windows [version 2.6.12].
Extract this .zip file to C:\tmp. In C:\tmp\rubygems-2.6.12 run ruby setup.rb.
Verify the installation with gem --version.
264 Tips & Tricks
Important
This upgrade fixes the following issue when installing svn2git: “ERROR: Could not find a valid gem 'svn2git' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate”.
A.2.2.4. svn2git installation guide
Install the svn2git gem with gem install svn2git --version 2.3.2 [version 2.4.0].
Warning
Usage of version 2.4.0 results in an empty Git repository and the error message: “command failed: git checkout -f master”.
Verify the installation with gem list or svn2git --help. A.2.2.5. Migration example
Because the Java.net and Kenai.com forges will be going dark on April 28, 2017 lets migrate the jmodalwindow's sources that I was hosting there.
Note
To convert all the svn authors to git format, create the file C:\dev\projects\de- mo\users.txt with for example:
But this way the whole history of the website that is maintained in the www folder. Thus lets try the option --ex- clude www. Alas this results in the following error “'tags[' is not recognized as an internal or external command,” when Running command: git svn fetch '--ignore-paths=^(?:trunk[/]|tags[/][^/]+[/]) (?:www)' (see also the above mentioned resource about issue #121).
To avoid this error edit P:\dev\apps\prg\ruby-2.3.3\lib\ruby\gems\2.3.0\gems \svn2git-2.3.2\lib\svn2git\migration.rb:
Or just run p:\dev\apps\editor\winmerge\WinMergeU.exe P:\dev\apps\prg \ruby-2.3.3\lib\ruby\gems\2.3.0\gems\svn2git-2.3.2\ P:\dev\apps\prg \ruby-2.3.1\lib\ruby\gems\2.3.0\gems\svn2git-2.3.2\ to copy the changes in lib\svn2git\migration.rb from a prior installation of ruby.
Git's primary solution is to specify that LF is the best way to store line endings for text files in a Git repository's object database (see also Mind the End of Your Line). When writing to the object database or writing out into the working directory Git has a chance to run filters and do things like converting line endings. Originally the conversion was handled with core.autocrlf. Run git config --list --show-origin or git config --show-origin core.autocrlf to see what this value is set to. On Windows, you simply pass true to the configuration with git config --global core.autocrlf true.
Alternatively define all of this in the .gitattributes file that you keep with your repository. This means that line endings can be encapsulated entirely within a repository and don't depend on everyone having the proper global settings. For example C:\dev\projects\jmodalwindow\.gitattributes based on Java.gitattributes of .gitattributes Templates:
# Handle line endings automatically for files detected as text # and leave all files detected as binary untouched. * text=auto
# # The above will handle all files NOT found below # # These files are text and should be normalized (Convert crlf => lf) .gitattributes text .gitignore text .gitkeep text .gitconfig text
*.bat text *.cmd text *.sh text eol=lf Makefile text makefile text Dockerfile text
*.java text *.jsp text *.jspf text
266 Tips & Tricks
*.jspx text *.tag text *.tagx text *.tld text *.ldif text *.ctl text *.sql text *.feature text *.plantuml text *.graphml text Jenkinsfile text *.yml text *.yaml text *.raml text *.diff text diff -merge *.patch text diff -merge *.properties text *.MF text *.mapping text *.ini text *.cnf text *.conf text *.config text *.pageflow text *.launch text
*.txt text *.apt text *.vm text *.log text *.importorder text *.xsl text *.xslt text *.xml text *.dtd text *.xsd text *.wsdl text *.wsdd text *.xjb text *.php text *.md text *.adoc text *.csv text *.ftl text *.jrxml text *.dwt text *.pu text *.archimate text *.ucls text *.uxf text .editorconfig text *.dia binary *.doc binary *.docm binary *.docx binary *.dot binary *.dotx binary *.msg binary *.ppt binary *.pptm binary *.pptx binary *.pdf binary *.odt binary *.ai binary *.rtf binary *.vsd binary *.xls binary *.xlsm binary *.xlsx binary
*.asp text *.h text
267 Appendix A.
*.css text *.less text *.sass text *.scss text *.htm text *.html text *.xhtml text *.faces text *.mht text *.ts text *.js text *.jsx text *.json text *.coffee text .htaccess text
# These files are binary and should be left untouched # (binary is a macro for -text -diff) # #image *.bmp binary *.eps binary *.gif binary *.ico binary *.jng binary *.jpg binary *.jpeg binary *.jpx binary *.jp2 binary *.jxr binary *.mng binary *.png binary *.psb binary *.psd binary *.svg -text diff -merge *.svgz binary *.tif binary *.tiff binary *.wbmp binary *.webp binary Thumbs.db binary
Seeing the .class binary mapping reminds me to add an C:\dev\projects\jmodalwin- dow\.gitignore based on the ignores template.
Tip
Use git ls-files --eol | grep auto to identify files that aren't covered by .gitat- tributes and thus are marked for automatic end-of-line conversion.
A.2.2.6.1. Pattern matches and overrides
.gitattributes defining attributes per path. When more than one pattern matches the path, a later line overrides an earlier line. The rules how the pattern matches paths are the same as in .gitignore files. Unlike .gitignore, negative patterns are forbidden.
For example define project specific override settings for .xml files at the end of the .gitattributes file:
# Project Specific ### **/tst/windows/*.xml text eol=crlf **/tst/linux/*.xml text eol=lf
A.2.2.6.2. Refreshing the repository after changing line endings
Save your current files in Git, so that none of your work is lost (see also Dealing with line endings):
Rewrite the Git index to pick up all the new line endings:
git reset
Add all your changed files back, and prepare them for a commit. This is your chance to inspect which files, if any, were unchanged. Also check that the only changes you are committing are the line endings:
git status git diff --ignore-space-at-eol git add . -u
Commit the changes to your repository:
git commit -m "Normalize all the line endings"
Note
git reset after removing .git/index is enough to mark files with the wrong line endings in the repo and commit those.
Important
When changing file types from text to binary on a Windows machine and the workspace wasn't a result from a svn2git migration but a git clone then you might end up changing the original LF with CRLF because of the initial conversion on checkout.
For example add the following entry *.svg -text diff -merge to .gitattributes (see also SVG debate) and the above described procedure might display the following results:
git status | grep badge-rounded-template.svg modified: src/main/resources/com/qualinsight/plugins/sonarqube/ badges/ws/badge-rounded-template.svg
By the way binary is short for -diff -merge -text.
A.2.2.6.3. Refreshing the workspace after changing line endings
But what if the files in the workspace have the wrong line endings. You will see messages like “warning: LF will be replaced by CRLF in …” and “warning: CRLF will be replaced by LF in …” .
After again removing Git's index:
del .git\index or rm .git/index
270 Tips & Tricks
Rewrite the Git index to pick up all the new line endings by reconverting from HEAD:
git reset --hard
Warning
git reset --hard will revert all uncommitted/unadded modified files also and thus will be gone (see also Undo-ing ‘git reset hard’ about git fsck [--lost-found]).
Might as well use script git-create-repo.sh to add support for creating repositories to GitWeb by patching P:\dev\apps\vcs\git\share\gitweb\gitweb.cgi:
@@ -760,6 +760,8 @@ our @cgi_param_mapping = ( extra_options => "opt", search_use_regexp => "sr", ctag => "by_tag", + new_project_name => "pn", + new_project_description => "pd", # this must be last entry (for manipulation from JavaScript) javascript => "js" );
Make sure git-create-repo.sh is available on the BATCH_HOME [6] path.
Now it is possible to create a remote repository with GitWeb. Just enter a project name (without .git suffix) and description by calling http://freedumbytes.dev.net/git-repo?pn=gitweb&pd=GitWeb+CGI +enhancement&a=create direct or through the user interface enhancement:
273 Appendix A.
Figure A.3. GitWeb enhancement
274 Appendix B. Appendix B. Hosting a group Website in the Cloud
Where and How To host this manual in the cloud at https://freedumbytes.github.io/, https:// freedumbytes.gitlab.io/ and https://freedumbytes.bitbucket.io/, B.1. Resources
• Publishing a Website on Bitbucket Cloud. • GitHub Pages Basics: User, Organization, and Project Pages. • Hosting on GitLab.com with GitLab Pages. • Pull/Push from multiple remote locations. B.2. Bitbucket
You can use Bitbucket to host a static website. A static website contains coded HTML pages with fixed content. Websites hosted in this way have the bitbucket.io domain in their URL.
1. Create an account. 2. After signing in, create a team at https://bitbucket.org/account/create-team/ with: • Team name: Free Dumb Bytes. • Team ID: freedumbytes. 3. Next create the private or public project to contain the website for this team at https://bitbucket.org/ac- count/projects/create with: • Owner: Free Dumb Bytes. • Name: freedumbytes.bitbucket.org (format in case of a site .bitbucket.org). • Key: FDB. 4. Finally create and push a welcome page index.xhtml. B.3. GitHub
There are two basic types of GitHub Pages: User/Organization Pages and Project Pages.
1. Join GitHub. 2. After signing in, create an organization at https://github.com/account/organizations/new with: • Organization name: freedumbytes. 3. Next create the private or public project to contain the website for this organization at https://github.com/ new with: • Owner: freedumbytes. • Repository name: freedumbytes.github.io (format in case of a site .github.io). 4. Finally create and push a welcome page index.xhtml. B.4. GitLab
With GitLab Pages you can host for free your static websites on GitLab. Combined with the power of GitLab CI and the help of GitLab Runner you can deploy static pages for your individual projects, your user or your group.
1. Register. 2. After signing in, create a group at https://gitlab.com/groups/new with: • Group path: freedumbytes. 3. Next create the private, internal or public project to contain the website for this group at https://gitlab.com/ projects/new with: • Project path: freedumbytes. • Project name: freedumbytes.gitlab.io (format in case of a site .gitlab.io). 4. Finally create the CI script to actual deploy the pages to https://freedumbytes.gitlab.io with .gitlab-ci.yml:
276 Appendix C. Appendix C. Continuous Integration Pipeline
Example setups for Jenkins and GitLab CI pipelines originally based on Maven.gitlab-ci.yml. C.1. Resources
• Getting started with GitLab CI and Examples. Continuous Delivery of a Spring Boot application with GitLab CI and Kubernetes. • What is Jenkins Pipeline. C.2. GitLab CI Pipeline
Just add a YAML file gitlab-ci.yml to a project to tell the GitLab runner what to do. The latest version can be found at https://gitlab.com/freedumbytes/setup/blob/master/.gitlab-ci.yml.
Important
Some pipeline settings depend on setting from the Maven POM base/parent setup.
C.2.1. Build Stage (test-)compile
The first stage is created to compile the main en test sources and the resulting target and optional generated content is collected as an artifact for the next stage:
Alas the artifacts cannot be collected recursively. Thus initial setup is for a maven project with 2 levels deep modules hierarchy.
277 Appendix C.
Important
The original template example used --fail-at-end instead of --fail-fast, but this makes the build run longer in case of failure and causing additional logging.
C.2.2. Test Stage deploy
The second stage is created to test and deploy into Nexus the compiled/generated class dependencies from the build stage and saving the test results for the next stage:
Used verify instead of deploy since there isn't a secret variable for ossrh configured (yet). Also disabled openSource profile since there isn't a secret variable for gpg.keyname config- ured (yet).
Important
Instead of only saving surefire/failsafe TEST-*.xml and *.txt files, these paths will al- so fetch a copy of any [date]-jvmRun[N].dump, date]-jvmRun[N].dumpstream and [date].dumpstream files for artifacts download in case of an error.
C.2.3. Test Stage no recompile - Changes detected
Prevent unnecessary recompiling by maven-compiler-plugin when “Changes detected - recompiling the module!” due to the absolute .java file path in target/maven-status/maven-compiler-plugin/com- pile/default-compile/inputFiles.lst:
This happens for example when Jenkins CI runs parallel jobs in different workspace locations (see also MCOMPILER-209).
C.2.4. Test Stage no recompile - Compiling xx source files
Prevent unnecessary recompiling by maven-compiler-plugin when “Compiling xx source files” instead of “Nothing to compile - all classes are up to date” due to git clone action in the test job causing the .java file being more recent than the corresponding .class file:
One could debate that unit and integration tests shouldn't run in the same stage. But the in- tended integration tests are those between classes which do not required a deployed system. How to run system integration tests will be added at a later date.
C.2.7. GitLab CI Cache
Cache downloaded dependencies and plugins between jobs:
The dots between for example date and time are necessary because spaces would cause the following error “Error: Could not find or load main class …” (see also MNG-4559).
281 Appendix C.
C.2.10. Suppress download messages
Suppress download messages for dependencies and plugins:
Alas due to the nature of the pages job it is not possible to deploy the site on failure also (for now just download the generated site artifacts, which will expire in 1 day, to browse the test reports).
284 Continuous Integration Pipeline
C.2.16. Effective Pom
Convenience job to dump the effective-pom in case of failure:
Currently disabled since there isn't a secret variable for sonar.login configured (yet).
C.2.20. SonarQube Dashboard issue workaround
Maven report link to SonarQube is no longer working (see also issue 6 and Section H.3.4, “Open Source Patches”):
stage: docs script: - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports' + - 'find . -type f -name "sonar.html" | xargs --no-run-if-empty sed -i "s/\/ project\/index\//\/dashboard\/index\//g"' - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports' - 'rm -r target/site' - 'mv `find ./target -type f -name "team.html" | sed -r "s|/[^/]+$||" | sort -u | head -1` target/pages'
C.2.21. Optional OWASP Docs Stage
When running the Dependency Check Aggregate on a completely new version of a multi-module project, it fails (TODO reword) when a referenced submodule isn't available in the local or remote repository already with the following error “[ERROR] Could not find artifact …”. So move it from test to docs stage while possibly reducing any build time bonus:
On branches run verify only instead of deploy because it is otherwise no longer clear which SNAPSHOT artifacts will be downloaded from Nexus (the latest master or branch build):
After this change the Animal Sniffer Check fails for multi-module projects, when a referenced submod- ule isn't available in the local or remote repository already with the following error “[ERROR] Failed to execute goal on project sample: Could not resolve dependencies for project …” which is “Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact …” or “[ERROR] … Undefined ref- erence: …”. So switch from verify to install:
For the same reason as branches in the prior paragraph run verify only for master and tags on forks by adding the group and the project of for example https://gitlab.com/freedumbytes/setup to all only and except options:
When only building for a certain jdk version anyway, just create a docker image environment variable and thus requiring only one change to upgrade the script:
Should the project be configured to create jdk7 code for example with default POM properties overrides for compiler, animal-sniffer, enforcer and javadoc plugins:
This way we don't run the risk of the animal-sniffer and enforcer missing the issue of TODO. C.3. Jenkins CI Pipeline
Just add a file Jenkinsfile to a project to tell the Jenkins pipeline what to do. The latest version can be found at https://gitlab.com/freedumbytes/setup/blob/master/Jenkinsfile.
Important
Some pipeline settings depend on setting from the Maven POM base/parent setup.
C.3.1. Build Stage (test-)compile
The first stage is created to compile the main en test sources and the resulting target and optional generated content is collected as an artifact for the next stage:
Alas the artifacts cannot be collected recursively. Thus initial setup is for a maven project with 2 levels deep modules hierarchy.
Important
The original template example used --fail-at-end instead of --fail-fast, but this makes the build run longer in case of failure and causing additional logging.
C.3.2. Test Stage deploy
The second stage is created to test and deploy into Nexus the compiled/generated class dependencies from the build stage and saving the test results for the next stage:
Used verify instead of deploy since there isn't a secret variable for ossrh configured (yet). Also disabled openSource profile since there isn't a secret variable for gpg.keyname config- ured (yet).
One could debate that unit and integration tests shouldn't run in the same stage. But the in- tended integration tests are those between classes which do not required a deployed system. How to run system integration tests will be added at a later date.
C.3.5. Test Stage no recompile - Changes detected
Prevent unnecessary recompiling by maven-compiler-plugin when “Changes detected - recompiling the module!” due to the absolute .java file path in target/maven-status/maven-compiler-plugin/com- pile/default-compile/inputFiles.lst:
The dots between for example date and time are necessary because spaces would cause the following error “Error: Could not find or load main class …” (see also MNG-4559).
C.3.8. Suppress download messages
Suppress download messages for dependencies and plugins:
When running the Dependency Check Aggregate on a completely new version of a multi-module project, it fails when a referenced submodule isn't available in the local or remote repository already with the following error “[ERROR] Could not find artifact …”. So move it from test to docs stage while possibly reducing any build time bonus:
echo "Effective POM..." sh "mvn $MAVEN_CLI_OPTS help:effective-pom ${args.profile}"
C.3.17. Test Stage no recompile - Compiling xx source files
Prevent unnecessary recompiling by maven-compiler-plugin when “Compiling xx source files” instead of “Nothing to compile - all classes are up to date” due to git clone action in the test job causing the .java file being more recent than the corresponding .class file:
This happens after dropping unstash "sourcecode" in favor of checkout scm.
C.3.18. Extra Open Source Stage
Because GitLab CI isn't configured yet to deploy the artifacts to Sonatype Open Source Software Repository Host- ing and to SonarSource SonarCloud Continuous Code Quality let the local hosted Jenkins CI do this also with an extra stage:
The sonarCloud profile is configured in settings.xml with the sonar.login token for https://sonarcloud.io/.
C.3.19. Artifacts verify vs deploy
On branches run verify only instead of deploy because it is otherwise no longer clear which SNAPSHOT artifacts will be downloaded from Nexus (the latest master or branch build):
try
301 Appendix C.
{ echo "Integration Test..." - sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}" + + if (args.branchNames.contains(env.BRANCH_NAME)) + { + sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}" + } + else + { + sh "mvn $MAVEN_CLI_OPTS verify ${args.profile}" + } + sh "mvn $MAVEN_CLI_OPTS animal-sniffer:check ${args.profile}" sh "mvn $MAVEN_CLI_OPTS enforcer:enforce ${args.profile}" }
After this change the Animal Sniffer Check fails for multi-module projects, when a referenced submod- ule isn't available in the local or remote repository already with the following error “[ERROR] Failed to execute goal on project sample: Could not resolve dependencies for project …” which is “Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact …”. So switch from verify to in- stall:
else { - sh "mvn $MAVEN_CLI_OPTS verify ${args.profile}" + sh "mvn $MAVEN_CLI_OPTS install ${args.profile}" }
sh "mvn $MAVEN_CLI_OPTS animal-sniffer:check ${args.profile}"
C.3.20. Javadoc Reports
Enable the Javadoc Reports [153] for the generated project site:
- if (args.branchNames.contains(env.BRANCH_NAME)) + if (args.branchNames.contains(env.BRANCH_NAME) || hasReleaseTag()) { sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}" }
def mvnSonarQubeNode(Map args) { node("${args.nodeName}") { stage("sonarqube") { - if (args.branchNames.contains(env.BRANCH_NAME)) + if (args.branchNames.contains(env.BRANCH_NAME) || hasReleaseTag()) { …
echo "Quality Assurance..." sh "mvn $MAVEN_CLI_OPTS sonar:sonar ${args.profile}"
A tag is marked as release tag when the commit message also contains “[maven-release-plugin] prepare release”, which is placed by the Maven Release Plugin.
Important
When pushChanges is turned off for the Maven SCM Plugin make sure to not push the HEAD which contains “[maven-release-plugin] prepare for next development iteration” but the com- mit before it with for example:
git push origin HEAD~:
303 Appendix C.
In case Jenkins CI failed or missed building the tag just use the alternative release sh script mentioned below.
Since GitLab CI doesn't use SNAPSHOT artifacts from Sonatype Open Source Software Repository Hosting (OSSRH), because that repository isn't defined in the used Maven Docker images, we might as well don't upload them anymore:
The rules.xml was almost removed with version 3.0.0 of versions-maven-plugin (see also issue 157).
To allow free access to this file just upload it as an artifact in the Releases repository of Nexus:
308 Maven Versions Rules
Figure D.2. Maven Versions Rules Nexus Upload
Or just start a new bare Git repository maven-versions-rules by calling /p/dev/apps/win- dows/batch/git-create-repo.sh /p/dev/data/repo/git maven-versions-rules.git "A ruleSet file containing the rules that control how to compare version numbers." - jenkinsHook in Git Bash (see also repository https://gitlab.com/freedumbytes/versions-rules and site https:// freedumbytes.gitlab.io/versions-rules/plugin-updates-report.html).
Finally configure the rule set usage in C:\dev\projects\maven-setup\pom.xml:
Lets setup a base structure, for Maven skins to come, with the following JIRA project:
• Maven Skins [MVNSKN]: Maven custom skins (see also repository branches/skins).
With the following components: 1. paper: Paperlike skin. 2. fluido: Apache Maven Fluido skin enhancement. 3. reflow: Responsive Apache Maven Reflow skin enhancement. 4. sample: Samples of Maven skins: Default, Classic, Application, Stylus and Fluido.
Start a new bare Git repository skins by calling /p/dev/apps/windows/batch/git-create-repo.sh /p/ dev/data/repo/git skins.git "Maven custom skins" -jenkinsHook in Git Bash.
Clone this remote repository with the following command git-clone jjasper skins.
Create the minimal base POM in C:\dev\projects\skins\pom.xml:
In Eclipse create a Working Set Maven Skins and import this Maven Skins project into it with File → Import… → Maven → Existing Maven Projects from Root directory C:\dev\projects\skins and click Finish. E.1. Custom Paper Skin
To create a paperlike skin export a copy of maven-default-skin-1.1:
cd /d C:\dev\projects svn export ^ http://svn.apache.org/repos/asf/maven/skins/tags/maven-default-skin-1.1 ^ maven-default-skin-1.1
And also override the part of the ancestor C:\dev\projects\maven-setup\src\site\site.xml with a simple C:\dev\projects\skins\maven-paper-skin\src\site\site.xml:
Since the banner logo and the favicon are now set in maven-theme.css and site.vm remove the references in C:\dev\projects\skins\maven-paper-skin\src\site\site.xml:
Activate the changes with mvn install and verify them with mvn site in your browser at file:///C:/dev/projects/ skins/maven-paper-skin/target/site/index.html.
315 Appendix E.
Figure E.1. Paperlike Skin Sample
E.2. Custom Fluido Skin
To add some custom imagery repackage the Apache Maven Fluido Skin.
Define a new module maven-fluido-skin in C:\dev\projects\skins\pom.xml:
maven-paper-skin + maven-fluido-skin
Supply the new module C:\dev\projects\skins\maven-fluido-skin\pom.xml: