Testing The JCOP Framework Paul Burkimsher October 2010 What is the JCOP Framework? • You should know that by now… ;-) • It’s been around a while but is of course being maintained. – The problem with maintenance is you can fix one bug and introduce another. – Same for upgrading… • => Need for Regression testing – Does the original bit still work like it did yesterday? 2 Regression testing • Is boring, repeating the same tests every day – so we automate it. • We have taken a very top-down approach. We pretend to be a user clicking on buttons, manipulating the DEN, declaring devices… • We write Python programs which fire up PVSS panels, press buttons, type in field values… 3 4 How do we do it? • Use a GUI testing tool called “Squish” from a German company. • We chose Squish because ETM chose Squish for their own GUI testing… 5 Squish is powerful • I agree with the salesman : • It is flexible – You can run Squish on Windows and test • Fw components also running on Windows or • Fw components running remotely on Linux – It supports several programming languages. I chose to use Python (I’m not a Java, Perl or Tcl fan) • It is easy to use – You can record mouse movements and keystrokes to get your initial scripts 6 Powerful, but • Squish itself is being developed intensively – Not ever so stable (presumably they test it :-P ) • PVSS is changing underneath. ETM try very hard to keep the look and feel the same, but their implementation underneath changes – If the computer clicks on qt_go button and its name changes to qt_start, you’re screwed. • Qt is changing underneath (and PVSS is built on top of Qt) • … a lot of running fast to just stay still 7 How does it work? • I defined a sequence of Test Suites (Python programs): – suite_createCleanPut (“ProjectUnderTest”) – suite_installFwInstallationTool – suite_installFwComponents – suite_exerciseAnalogDigitalComponent – suite_exercise CaenComponent • tst_createCrateSy127; tst_createCrateSy403 • tst_createCrateSy527; tst_createCrateSy2525 – suite_exerciseWienerComponent – suite_exerciseTrendEditorNavigator 8 How does it work – II ? N AUT Background (Application Python Script e or t Under Test) w uses Qt Python Script o in IDE r Special qt.dll k get interpreted Squish Runner Squish server by (interpreter - client) 9 • How do you know its worked? 10 • It gets to the end without breaking? • You can put assertions in the Python code • You can inspect the output 11 What is the output of the DEN? – DPs and hence .dpl files – We compare the produced .dpl file with an “expected” .dpl file. – NB You have to remove the timestamps and Dp Ids from the .dpls first. 12 How do we communicate results? • Produce log files at every stage. • Squish produces an XML summary of the assertion results (and provides an html translator) • We create an email with this info and send it to all the JCOP authors whose components are being tested 13 Nightly email 14 Email (cont) 15 Further work • More tests • More software version combinations (PVSS 3.9 etc) • Fully automate the test across to Linux • Code coverage in the scripts 16 Questions ? 17 18.