Testing The JCOP Framework
Paul Burkimsher October 2010 What is the JCOP Framework? • You should know that by now… ;-) • It’s been around a while but is of course being maintained. – The problem with maintenance is you can fix one bug and introduce another. – Same for upgrading…
• => Need for Regression testing – Does the original bit still work like it did yesterday?
2 Regression testing • Is boring, repeating the same tests every day – so we automate it. • We have taken a very top-down approach. We pretend to be a user clicking on buttons, manipulating the DEN, declaring devices…
• We write Python programs which fire up PVSS panels, press buttons, type in field values…
3 4 How do we do it? • Use a GUI testing tool called “Squish” from a German company. • We chose Squish because ETM chose Squish for their own GUI testing…
5 Squish is powerful • I agree with the salesman : • It is flexible – You can run Squish on Windows and test • Fw components also running on Windows or • Fw components running remotely on Linux – It supports several programming languages. I chose to use Python (I’m not a Java, Perl or Tcl fan) • It is easy to use – You can record mouse movements and keystrokes to get your initial scripts
6 Powerful, but • Squish itself is being developed intensively – Not ever so stable (presumably they test it :-P ) • PVSS is changing underneath. ETM try very hard to keep the look and feel the same, but their implementation underneath changes – If the computer clicks on qt_go button and its name changes to qt_start, you’re screwed. • Qt is changing underneath (and PVSS is built on top of Qt) • … a lot of running fast to just stay still
7 How does it work? • I defined a sequence of Test Suites (Python programs): – suite_createCleanPut (“ProjectUnderTest”) – suite_installFwInstallationTool – suite_installFwComponents – suite_exerciseAnalogDigitalComponent – suite_exercise CaenComponent • tst_createCrateSy127; tst_createCrateSy403 • tst_createCrateSy527; tst_createCrateSy2525 – suite_exerciseWienerComponent – suite_exerciseTrendEditorNavigator
8 How does it work – II ?
N AUT Background (Application Python Script e or t Under Test) w uses Qt Python Script o in IDE r Special qt.dll k get interpreted Squish Runner Squish server by (interpreter - client)
9 • How do you know its worked?
10 • It gets to the end without breaking?
• You can put assertions in the Python code
• You can inspect the output
11 What is the output of the DEN?
– DPs and hence .dpl files
– We compare the produced .dpl file with an “expected” .dpl file.
– NB You have to remove the timestamps and Dp Ids from the .dpls first.
12 How do we communicate results?
• Produce log files at every stage.
• Squish produces an XML summary of the assertion results (and provides an html translator)
• We create an email with this info and send it to all the JCOP authors whose components are being tested
13 Nightly email
14 Email (cont)
15 Further work • More tests • More software version combinations (PVSS 3.9 etc) • Fully automate the test across to Linux • Code coverage in the scripts
16 Questions ?
17 18