Performance testing With Apache JMeter? Frank Pientka Where and who is Materna GmbH? . Founded: 1980 Hamburg Bremen Employees: 1,700 Lüneburg Berlin Turnover 2015: € 210 million Wolfsburg FI NOR Dortmund SE . ITK-Full-service provider in the Dresden Düsseldorf premium segment DK Köln UK Bad Vilbel NL PL . Family-owned company in the DE Erlangen CZ Heidelberg ITC sector SK Nürnberg Stuttgart . Target groups: IT organisations München and user departments in both private and public sectors Orlando, Florida, USA © Materna GmbH 2016 www.materna.com 2 Who is Frank Pientka? Dipl.-Informatiker (TH Karlsruhe) Principal Software Architect at Dortmund iSAQB founding member heise.de/developer/Federlesen columne over 20 years IT experience lots of articles and talks © Materna GmbH 2016 www.materna.com Agenda . Why performance testing? . Testing right! . What is Apache JMeter? . What’s new in Jmeter 3.0? . JMeter main elements? . Have a (test-)plan! . Distributed performance testing . Further steps, extensions, challenges . Resume © Materna GmbH 2016 www.materna.com 4 Why performance tests? © Materna GmbH 2016 www.materna.com 5 What went wrong while testing? How secure&safe? The Heisenbug! Volkswagen St. (VW) share © Materna GmbH 2016 www.materna.com 6 Java performance testing tools (selection) JUnitPerf ab Gattling © Materna GmbH 2016 www.materna.com 7 May I introduce Apache JMeter? 18 years of experience . Since 1998 for Apache Tomcat loadtest (Stefano Mazzocchi) . Recent version 3.0 (2016-05-17) . Supported protocols . Web - HTTP, HTTPS, FTP . SOAP/XML-RPC . Database via JDBC, MongoDB, HADOOP . LDAP . JMS . Mail - SMTP(S), POP3(S) and IMAP(S) . Native commands or shell scripts + Highly extensible and configurable, tutorial, example templates, demos - Limited reporting capacities, Short outdated documentation, i18n © Materna GmbH 2016 www.materna.com 8 https://jmeter-plugins.org/stats/ © Materna GmbH 2016 www.materna.com 9 Apache JMeter 3.0 Changes http://jmeter.apache.org/changes.html . Test plan creation and debugging improvements . Reporting improvements (dynamic HTML-Dashboard) . Protocols and Load Testing improvements . UX Improvements . Core improvements . Components updates . Incompatible changes, deprecations © Materna GmbH 2016 www.materna.com 10 A JMeter test plan example © Materna GmbH 2016 www.materna.com 11 How does Apache JMeter simulate a single user? http://jmeter-expert.blogspot.de © Materna GmbH 2016 www.materna.com 12 Typical steps . Create Testplan . Test Testplan . Run Testplan and evaluate results . Document Testplan and results . Extras/schematic.cmd Testplan.jmx Testplan.html . TestPlanCheck --jmx <filename> [ --stats --tree-dump ] . jmeter -n -t <test JMX file> -l <test log file> -e -o <Path to output folder> © Materna GmbH 2016 www.materna.com 13 How does Apache JMeter simulate multiple users? http://jmeter-expert.blogspot.de © Materna GmbH 2016 www.materna.com 14 The main elements of a JMeter testplan http://jmeter-expert.blogspot.de © Materna GmbH 2016 www.materna.com 15 JMeter testplan elements Start Workbench • Test Recorder Listener ConfigTestElement • SummaryReport • Request Defaults • Results Tree • Autorisation Manager • Results Table • Cookie Manager Sampler • HTTP Request Assertion Timer 16 © Materna GmbH 2016 www.materna.com Login testing scenario Thread Group Constant HTTP sampler GET Start Thread: 1000 Timer Hostname/url Ramp-up: 500 300ms RegEx HTTP Status code HTTP sampler POST Hostname/url J_username=$user J_password=$pwd Assertiom Check combination ist OK Constant HTTP sampler GET End Timer logout 300ms Assertiom Check login string © Materna GmbH 2016 www.materna.com 17 Example: Tomcat admin uses manager app © Materna GmbH 2016 www.materna.com 18 Example: Tomcat manager app JMeter testplan (skeleton & meat) © Materna GmbH 2016 www.materna.com 19 JMeter test plan recorder (HTTP Proxy Server) elements Alternative: BlazeMeter's Google Chrome extension, .har file to a JMeter test plan (.jmx file) converter © Materna GmbH 2016 www.materna.com 21 Tips&Tricks . Use user defined variables with same name as in request . Place Login CSV Data Set Config element on login page request . Use Counter element and reference {counter} variable in request i.e. {user} . Use non-GUI mode: jmeter -n -t test.jmx -l test.jtl . Use as few Listeners as possible e.g. disable the “View Result Tree” Use . CSV output rather than XML. Only save the data that you need. Use as few Assertions as possible. Disable all JMeter graphs as they consume a lot of memory . Check, document testplan TestPlanCheck': --jmx <filename> [ --stats --tree-dump ] © Materna GmbH 2016 www.materna.com 22 Four Ways To Launch a JMeter Test without JMeter GUI 1. Command line: jmeter -n -t your_script.jmx 2. JMeter Ant Task ant extras/build.xml 3. JMeter Maven Plugin. pom.xml mvn verify jmeter:jmeter 4. from Java code © Materna GmbH 2016 www.materna.com 25 JMeter distributed testing with Master-Slave Remote start Slave1 Slave1 Remote start All Remote stop System Master-GUI under Slave1 jmeter.properties Test remote_hosts= Slave1,Slave2,Slave3 Remote start Slave3 Slave3 In bin/jmeter.properties "remote_hosts=localhost:1099,localhost:2010" add IP addresses of Slave-Clients, server.rmi.port=1099, server.rmi.localport=1099, client.rmi.localport=2010 On Master: jmeter -n -t script.jmx -R slave1,slave2,slave3. „Remote Start All" distribute and execute testplan and receive results, SET SERVER_PORT=1099, set REMOTE_HOSTS=master:1099 "Remote Stop“, control jmeter-server.log, and have check and assert elements, to handle errors © Materna GmbH 2016 www.materna.com 26 Distributed testing Running Apache JMeter on Amazon EC2 ec2 Script EC2, VPN SSH-Tunneling © Materna GmbH 2016 www.materna.com 27 How to document test plans in HTML? Cold/warm start! Think Time! Bildschirm als Bild speichern cd apache-jmeter-3.0\extras schematic.cmd test-plan.jmx ant -Dtest=test-plan run report TestPlanCheck.bat --jmx testplan.jmx --stats --tree-dump © Materna GmbH 2016 www.materna.com 28 Apache JMeter Dashboard 3.0 Save JPG.image http://jmeter.apache.org/usermanual/generating-dashboard.html © Materna GmbH 2016 www.materna.com 29 Report Dashboard with APDEX (Application Performance Index) table https://www.blazemeter.com/blog/5-key-things-you-need-know-about-jmeter-30 jmeter.bat -n -t WorkBench_JMETER.jmx -l WorkBench_JMETER.log -e -o report http://www.apdex.org © Materna GmbH 2016 www.materna.com 30 https://sense.blazemeter.com © Materna GmbH 2016 www.materna.com 31 Realtime results: Async BackendListener with Graphite/Grafana © Materna GmbH 2016 www.materna.com 32 https://flood.io/har2jmx © Materna GmbH 2016 www.materna.com 33 Continuous integration https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin © Materna GmbH 2016 www.materna.com 34 Useful extensions http://jmeter-plugins.org © Materna GmbH 2016 www.materna.com 35 Further challenges with JMeter . How to integrate JMeter in the CI-Build? . How to document test plans and results? . What further reporting tools are needed? . Extending the Web Test Plan (ForEach Controller with User Defined Variables (UDV), While Controller with StringFromFile Function, RegEx) . How to use JMeter with Selenium/JS/mobile Apps/Cloud? © Materna GmbH 2016 www.materna.com 36 Further info . Gatling http://gatling.io . Grinder3: http://grinder.sourceforge.net . JMeter Info's: http://jmeter.apache.org, http://sourceforge.net/projects/jmeterforwindows . JMeter extensions: http://jmeter-plugins.org, http://www.atlantbh.com/jmeter-components . JMeter Tips: http://blazemeter.com/apache-jmeter, http://jmeter-expert.blogspot.de . JMeter Master-Slave-Test: http://jmeter.apache.org/usermanual/remote-test.html . JMeter MAVEN Plugin: https://github.com/jmeter-maven-plugin, . Online-Reports: https://sense.blazemeter.com/ . https://blazemeter.com/blog/open-source-load-testing-tools-which-one-should-you-use . JMeter-ec2 script: https://github.com/oliverlloyd/jmeter-ec2 © Materna GmbH 2016 www.materna.com 37 Eights Anniversary of the Twitter Fail Whale © Materna GmbH 2016 www.materna.com 38 Fragen? © Materna GmbH 2016 www.materna.com 39 Vernetzt. Kontakt. Materna GmbH Frank Pientka Voßkuhle 37 44141 Dortmund +49 231 5599-8854 [email protected] © Materna GmbH 2016 www.materna.com.