Automate the Testing of JSL Using Hamcrest JMP Discovery Summit Europe 2019 (2019-EU-45MP-061)

Justin Chilton, JMP Senior Associate Test Engineer, SAS Evan McCorkle, JMP Software Developer, SAS

2 Copyright © SAS Institute Inc. All rights reserved. Automated Testing What did we just do? • Determine if our script is performing as expected • Realized it wasn’t working correctly • Fixed the bug • Verified that it was fixed

3 Copyright © SAS Institute Inc. All rights reserved. What is Hamcrest? How did we do it? • Using a new JSL library called JSL-Hamcrest Funny name • By writing tests Unit Testing Framework • Built from assertions about the scripts behavior rd 3 -Generation • And running them Declarative • In an automated way Readable • With clear reporting of success and failure Self-contained

Extension of existing JSL ut test("HangmanGame", "Guessing Non-Letter Fails", Expr( Testing Framework game = new HangmanGame("food", 5); ut assert that(Expr(game:guess letter("@")), Used to test JMP itself ut throws("Invalid Guess!")); )); Available now // Success!

5 Copyright © SAS Institute Inc. All rights reserved. What is Hamcrest? Details • Originally developed for Java and jUnit (~2012) Funny name • Focused on creating easy-to-read declarative Unit Testing Framework assertions 3rd-Generation • Ported to many other languages Declarative • C++, C#, Objective-C, Python, PHP, JavaScript, Erlang, Readable R, Rust, Swift Self-contained • And now JSL Extension of existing JSL • JSL-Hamcrest also comes with a Unit Testing Testing Framework framework similar to jUnit with Used to test JMP itself • Test Cases for consistency Available now • Named tests for human-readable expectations • Customizable reporting

1st Generation assert(x == y) assert(x == y) • Not very informative on failure 2nd Generation • Intent of test writer not always clear assert_equal(x, y) • Imperative programming 3rd Generation assert_that(x, equal_to(y)) FAILURE assert(x == y)

1st Generation assert_equal(x, y) assert_not_equal(x, y) assert(x == y) assert_contained_in(x1, x) 2nd Generation • Informative failures assert_equal(x, y) • Intent is clear • Declarative, not imperative 3rd Generation • Explosion of assertions for different situations assert_that(x, equal_to(y)) • However, often never extended, meaning four or five assertions are abused for all tests, leading back to uninformative failures and unclear intent

1st Generation assert_that(x, equal_to(y)) assert(x == y) assert_that(x, not(equal_to(y))) assert_that(x1, contained_in(x)) 2nd Generation assert_equal(x, y) • Informative failures 3rd Generation • Clear intent assert_that(x, equal_to(y)) • Declarative, not imperative • Reusable/Composable assertions

FAILURE assert_that(x1,contained_in(x)) 5 not contained in {1, 2, 4, 6} 9 Copyright © SAS Institute Inc. All rights reserved. History Generational Progress • Each generation building on the previous 1st Generation • Hamcrest assertions assert(x == y) • Can be written relatively naturally • Can be read relatively natural nd 2 Generation • Provide context on failure assert_equal(x, y) - You know that the test failed - And you know how it failed rd 3 Generation - Often, you needn’t even read the test to know what you broke assert_that(x, equal_to(y)) • Describe what you want to test, not how to extract that information • Can be mixed together to form entirely new assertions

11 Copyright © SAS Institute Inc. All rights reserved. Examples Matchers – The Key to Hamcrest anything assert_that("hamcrest", all of is(anagram_of("matchers")) not equal to contains • Matchers do two things: subset of • Describe their expectations starts with • Determine if those expectations are satisfied by a given missing value close to title is ignoring whitespace throws

… 12 Copyright © SAS Institute Inc. All rights reserved. Examples Matchers – The Key to Hamcrest anything assert_that("ham sandwich", all of is(anagram_of("matchers")) not equal to contains • Matchers do two things: subset of • Describe their expectations starts with • Determine if those expectations are satisfied by a given missing value close to title is ignoring whitespace throws

… 13 Copyright © SAS Institute Inc. All rights reserved. Examples Matchers – The Key to Hamcrest anything assert_that("Copenhagen", all of city_within("Denmark")) not equal to contains • Matchers do two things: subset of • Describe their expectations starts with • Determine if those expectations are satisfied by a given missing value close to title is ignoring whitespace throws

… 14 Copyright © SAS Institute Inc. All rights reserved. Examples Matchers – The Key to Hamcrest anything assert_that("London", all of city_within("Denmark")) not equal to contains • Matchers do two things: subset of • Describe their expectations starts with • Determine if those expectations are satisfied by a given missing value close to title is ignoring whitespace throws

… 15 Copyright © SAS Institute Inc. All rights reserved. Examples Matchers – The Key to Hamcrest anything assert_that("London", all of not(city_within("Denmark"))) not equal to contains • Matchers do two things: subset of • Describe their expectations starts with • Determine if those expectations are satisfied by a given missing value close to title is ignoring whitespace throws

Direct assert_that(x, is_not(equal_to(y))); Leaf Actually Tests • assert_that(value, matcher) equal, less than, has key, starts with • Harness. Is the matcher satisfied by the value? Indirect • is_not(matcher) → Matcher Non-Leaf • Negates any matcher inside of it (Indirect) Delegates Testing • equal_to(value) → Matcher • Only satisfied by values equal to value (Direct) not, contains, as char, ignoring • JSL is unique in that x is an expression whitespace

21 Copyright © SAS Institute Inc. All rights reserved. Possibilities Reporting Success and Failure • Reporter object Log File • Receives success and failure messages Log Window • Can be customized to your liking GUI • JSL-Hamcrest has several builtin Reporters Custom Report • Write failures to the Log Window Send Emails • Write success/failures to a Test Runner GUI … • But you can write your own Reporter to ensure that JSL-Hamcrest presents results exactly how you want them • Used internally when testing JMP itself

22 Copyright © SAS Institute Inc. All rights reserved. Interface Writing your own Matchers • One of the best things about Hamcrest is the << Matches ability to write new Matchers Success/Failure • Domain specific How • Meaningful to you << Describe • Write what you are testing, not how Expectation • Write it in your own words Factory Function • Add any relevant information to failures ut my matcher • If you do it correctly, you can mix your new Matcher with existing ones • Flexible and expressive • Multiply your work • JSL-Hamcrest has a suite of builtin Matchers

24 Copyright © SAS Institute Inc. All rights reserved. How can JSL-Hamcrest help? • Enable • Support • Ease Guiding Hand Regression • Increase Writing Tests Tests Confidence Script Script Script Improvement/ Planning Writing Maintenance

Reports of Safety Net Failures Tests as Documentation

25 Copyright © SAS Institute Inc. All rights reserved. v1.0 Release of JSL-Hamcrest 1.0 Includes • Library with Available Now • Test Cases for JMP 14.1+ • Suite of Matchers • Several Reporters • Documentation Get it on the • Add-In providing JMP User Community • The above library • Test Runner GUI • One-click “Run Tests”

jmp.com