The Most Suitable Person to Establish Quality Assurance Guidelines for The

The most suitable person to establish quality assurance guidelines for the generation and use of noncommercial clinical software is a medical physicist Diane Kelly, Alan Wassyng, and Colin G. Orton

Citation: Medical Physics 41, 090601 (2014); doi: 10.1118/1.4883877 View online: http://dx.doi.org/10.1118/1.4883877 View Table of Contents: http://scitation.aip.org/content/aapm/journal/medphys/41/9?ver=pdfcov Published by the American Association of Physicists in Medicine

Articles you may be interested in Quality assurance and training procedures for computer-aided detection and diagnosis systems in clinical usea) Med. Phys. 40, 077001 (2013); 10.1118/1.4807642

Report of AAPM TG 135: Quality assurance for robotic radiosurgery Med. Phys. 38, 2914 (2011); 10.1118/1.3579139

Patient-specific quality assurance method for VMAT treatment delivery Med. Phys. 36, 4530 (2009); 10.1118/1.3213085

IAEA Technical Reports Series No. 430: Commissioning And Quality Assurance Of Computerized Planning Systems For Radiation Treatment Of Cancer Med. Phys. 33, 561 (2006); 10.1118/1.2167371

Dosimetric evaluation of the clinical implementation of the first commercial IMRT Monte Carlo treatment planning system at 6 MV Med. Phys. 31, 2771 (2004); 10.1118/1.1786172

POINT/COUNTERPOINT Suggestions for topics suitable for these Point/Counterpoint debates should be addressed to Colin G. Orton, Professor Emeritus, Wayne State University, Detroit: [email protected]. Persons participating in Point/Counterpoint discussions are selected for their knowledge and communicative skill. Their positions for or against a proposition may or may not reﬂect their personal opinions or the positions of their employers.

The most suitable person to establish quality assurance guidelines for the generation and use of noncommercial clinical software is a medical physicist Diane Kelly, Ph.D. Department of Mathematics and Computer Science, Royal Military College, Kingston, Ontario K7K 7B4, Canada (Tel: 613-541-6000 ext. 6171; E-mail: [email protected]) Alan Wassyng, Ph.D. Department of Computing and Software, McMaster University, Faculty of Engineering, Hamilton, Ontario L8S 4L8, Canada (Tel: 905-525-9140 ext. 2607, E-mail: [email protected]) Colin G. Orton, Ph.D., Moderator (Received 14 May 2014; accepted for publication 16 May 2014; published 5 August 2014) [http://dx.doi.org/10.1118/1.4883877]

OVERVIEW crease the trustworthiness of scientific software. She teaches a graduate seminar course to both RMC and Queen’s students Noncommercial software is widely used in radiation ther- that critiques software development and quality assurance ap- apy, especially in academic centers. Such software can take proaches popular in software engineering when specifically many forms, ranging from MU-check spreadsheets; to in- applied to scientific software. Dr. Kelly has a Ph.D. and house built treatment planning and plan-evaluation systems; M.Eng. in Software Engineering, both from RMC, and a B.Sc. or to public domain codes such as EGSnrc or MNCP that in Pure Mathematics and B.Ed. in Mathematics and Computer are interfaced to IMRT planning systems. This noncommer- Science, both from the University of Toronto. She worked in cial software is usually written by in-house medical physi- the nuclear industry for over 20 years as a scientific software cists and, thus, the question arises as to how to establish the developer, technical trainer, and QA advisor. She is a senior integrity of the software they have written. One view has been member of IEEE. that only software professionals are properly equipped to provide quality-assurance guidance on the generation and use of such software. An opposing view is that the medical physi- Arguing against the Proposi- cists themselves are in the best position to determine effective tion is Alan Wassyng, Ph.D. practices to address software quality in their community, and Dr. Wassyng earned his Ph.D. this is the premise debated in this month’s Point/Counterpoint. in Applied Mathematics from the University of the Witwa- tersrand, Johannesburg, South Arguing for the Proposition Africa in 1979. After spend- is Diane Kelly, Ph.D. Dr. ing 14 years as an academic, Kelly is an Associate Profes- first at the University of Wit- sor in the Department of Math- watersrand and then at the Uni- ematics and Computer Science versity of Minnesota, he incor- at the Royal Military Col- porated a computer consulting lege of Canada (RMC). She company in Toronto, Canada. is cross appointed to RMC’s He returned to academia in 2002, joining the Department Department of Electrical and of Computing and Software at McMaster University, Hamil- Computer Engineering and to ton, ON, Canada, where he is currently Associate Profes- the School of Computing at sor and Director of the Centre for Software Certification. Queen’s University. Dr. Kelly’s Dr. Wassyng has published widely on software certifica- research focuses on ways to in- tion and the development of dependable embedded systems.

He is cofounder of the Software Certification Consortium, Kendall and Post, after studying decades of development and is Co-PI on the highly funded “Certification of Safety- of nuclear arsenal simulation software at the Los Alamos Na- Critical Software Intensive Systems” project led by McMaster tional Laboratory, concluded that the best people to draw up a University. list of “best practices” for software development and main- tenance are the members of the code project teams themselves, and that these practices are those “. . . that the teams FOR THE PROPOSITION: Diane Kelly, Ph.D. have judged useful for improving the way they do business”.5 It is the same for medical physicists. They know the best Opening Statement ways to assess their software in order to safely do their Presentations at the Canadian Organization of Medical business. Physicists (COMP) Winter School1 showed that medical physicists are deeply imbued in a safety culture. They re- AGAINST THE PROPOSITION: Alan Wassyng, Ph.D. act instinctively within this culture, pay attention to human- technology interaction, and exhibit due process in the light of Opening Statement safety concerns. I compare this to the environment of a soft- This proposition may make some sense if the guide- ware engineering colleague who specializes in testing: she lines are solely for the use of the software. However, med- lives in a volatile, market-driven, and cost-minimizing envi- ical physicists (MPs) are experts in medical physics, not ronment. Even though she has years of experience in testing the software fundamentals necessary for establishing soft- different products, her instincts and her quality goals are dif- ware quality assurance (SQA) guidelines for software de- ferent from those of a medical physicist. velopment. A history of creating software is not necessar- The software engineering literature does not acknowledge ily a plus. There are lots of people writing software, many the need for the conjunction of computational software de- of whom are not really qualified to do so, and the depend- sign processes with a deep safety culture, which is required ability of software, in general, is dismal.6 Software quality for deployment of software used to support clinical decision in the medical domain should be evaluated on the basis of making. Instead, such software is confused with either con- safety, security, and dependability, and paraphrasing “official” trol software, which directly operates a medical device, or definitions:7 commercial products where patient wellbeing is not directly r affected by the correctness of the output. As a result, there Software safety – under defined conditions, software are no guidelines in the software engineering literature that should not contribute to unsafe behavior or generate re- address the specific characteristics and needs of clinical soft- r sults that can lead directly to harm; ware. Software security – protection afforded the software to When advising on software quality guidelines, a typi- keep it from harm, and from causing harm through users cal software engineer takes a broad-spectrum approach. This maliciously bypassing the software’s designed-in safety approach suffers from two serious flaws. First, it encour- r and dependability; ages the perception that software is correct unless proven Software dependability – in its intended environment, otherwise. This dangerous assumption has been a contrib- the software can be trusted to produce the outputs for utor to several fatal accidents in the safety-critical world.2 which it was designed, with no adverse effects. A recent article3 talks about problems “when a computer Anyone establishing SQA guidelines for the generation of lulls us into a false sense of security”. Second, this broad- software must fully understand fundamental principles of spectrum approach does not use the knowledge of the peo- software that relate to safety, security, and dependability. ple associated with the software, and does not acknowledge Examples are: the specifics of the operational environment that these people r understand. Lack of continuity8 (Moderator: in software engineer- Vessey and Glass4 criticize the software engineering coming, continuity refers to a continuous function for which, munity for their broad, generic solutions, calling them weak intuitively, small changes in the input result in small solutions. They contend that the most effective and strong so- changes in the output) – If a bridge is built to withstand lutions are those that target the specific environment or situ- a force of 100 tonnes, because of the mostly continu- ation. Medical physicists, with their knowledge, can provide ous behavior of physical objects and our mathematical this strong solution. models, the bridge is likely to be safe for loads less than In a 2012 survey of software development and mainte- or equal to 100 tonnes. We do not expect it to collapse nance practices sent to medical physicists across Canada, the if a bicycle goes across it. In contrast, a simple error in medical physics community demonstrated that it already has a software that finds a number within a list of n numbers level of understanding necessary to provide meaningful soft- could easily result in the software working only when ware quality guidance to its own community. The devil is in n is even. This happens if the designer separates the be- the details and the medical physicists who responded to the havior into two cases (n is even and n is odd) and forgets survey understood the details of their environment and the im- to deal with the one case (n is odd). Thus, in this case, plications of problems. This understanding is far beyond what the software will work when n = 1000, but not for all n a software engineer outside the clinic can bring to the table. < 1000 – it fails when n = 19, for example.

Medical Physics, Vol. 41, No. 9, September 2014 090601-3 Kelly and Wassyng: Point/Counterpoint 090601-3 r Information hiding9 (Moderator: information hiding is a achieve it. There are no common activities or “silver bullets” software development technique in which each module’s such as “information hiding” to achieve, for example, “de- interfaces reveal as little as possible about the module’s pendability”. There is no research that demonstrates that par- inner workings and other modules are prevented from ticular code-level activities guarantee any high-level quality. using information about the module that is not in the The best we can do is to understand the particular software module’s interface specification)7 – This is a specific in front of us. In the case of clinical software, medical physi- way of performing modularization so that the resulting cists have the best understanding of what is in front of them in design significantly improves safety and dependability terms of use and the physics embedded in it. By keeping the when changes are made. SQA monitors the development software code very simple (which scientists have a tendency process of the software so that the resulting product is, to do15), medical physicists are in the best position to decide and remains, safe, secure, and dependable. This should what further keeps them out of trouble. My colleague suggests include ways of evaluating the information hiding as- that software engineers have a mandate to ensure that the pub- pects of the design. lic is served and protected. Software engineers do not always live in that culture. Medical physicists live in a safety culture Similar principles are software testing,10, 11 hazard and they have the capacity to fully understand what will best analysis,12 the absolute need for requirements traceability,13 achieve quality software for their own work. and semantics for module interface specifications,14 and numerous others. How can medical physicists take these principles into account when they do not possess this fundamental Rebuttal: Alan Wassyng, Ph.D. software knowledge? The person who does have this knowl- I have been involved in the COMP Winter School1 since edge is a software engineer (or perhaps, a computer scientist). its inception (I missed one year), and give a talk each year The Professional Engineers Act of Ontario [“Professional En- on medical device software. I have been told that the medical gineers Act”, Professional Engineers Ontario, http://www.e- physicists find this talk disturbing because they are surprised laws.gov.on.ca/html/statutes/english/elaws_statutes_90p28_ to find out how much they do not understand about software! e.htm/] states that the mandate of an engineer is “to ensure Their safety culture is, indeed, admirable, but this does not that the public interest may be served and protected.” A soft- make up for a lack of technical (software and system safety) ware engineer is tasked with developing software-dependent knowledge. SQA needs a team of people, including domain systems and protecting the public from harm caused by those experts and software experts. The SQA lead must have both systems. There are definite differences in SQA requirements the software expertise and the requisite safety knowledge. The in different domains – which is why some people think it fact that some software engineers are not immersed in the sys- is appropriate to have an MP establish SQA guidelines. tem safety world should not lead us astray. We see more and However, there are more commonalities across different more software engineers who work in the medical domain, domains than there are differences, so software knowledge is understand software, and have developed expertise in what is of primary importance. SQA is a team effort, with domain required in a regulated domain, in which safety is a primary expertise coming from the MP. However, the core knowledge concern. If there are not enough software engineers with this and guidance must come from a software engineer. safety focus, we should be championing changes to the software engineering curriculum. There are conferences19, 20 tar- Rebuttal: Diane Kelly, Ph.D. geted at software engineering in the medical domain. These My colleague built his argument around a software engi- conferences focus on medical devices and reporting/planning software. IEC standard 62304 focuses on safety of software neering list of software qualities: safety, security, dependabil- 21 ity. Why not a list from scientists: accuracy, trustworthiness, used in medical devices. Just because the standard applies readability, consistency with the physics, and simplicity? Ap- to medical devices does not mean it is irrelevant for other clinical software. Software that can impact the health and safety parently, scientists focus on simplicity far more than software engineers.15 of patients is deemed to be a medical device in most regula- For any set of qualities, we still need to achieve them. My tory regimes. There is a (growing) body of software engineers who do have the specific software expertise required, as well colleague suggests, for example, information hiding and requirements traceability as fundamental principles for anyone. as familiarity with basic safety concepts and regulatory guide- David Parnas, who first published the information hiding prin- lines. They are in a much better position to establish quality assurance guidelines for medical software than are medical ciple, complained in an invited talk16 that most software engineers do not know how to properly apply the principle. Sev- physicists. eral Standish Group surveys reported that only 9%–16% of 1See http://www.medphys.ca/content.php?sec=8 for Canadian Organization of Medical Physicists (Winter School, Quebec City, 2014). software projects are delivered successfully, largely because 2 17 N. Leveson, see http://sunnyday.mit.edu. software engineers do not understand user requirements. 3N. Carr, “All can be lost: The risk of putting our knowledge in the hands But medical physicists do understand their user requirements of machines,” The Atlantic, November 2013, also available at http://www. because they are the users. theatlantic.com/magazine/archive/2013/11/the-great-forgewtting/309516. 4 How do we add quality to a piece of software? It is well I. Vessey and R. Glass, “Strong vs weak approaches to systems develop- 18 ment,” Commun. ACM 41(4), 99–102 (1998). understood that there is a disconnect between the desire for 5D. E. Post and R. P. Kendall, “Software project management and qual- the high-level quality and what low-level activities actually ity engineering practices for complex, coupled multiphysics, massively

Medical Physics, Vol. 41, No. 9, September 2014 090601-4 Kelly and Wassyng: Point/Counterpoint 090601-4

parallel computational simulations: Lessons learned from ASCI,” Int. J. tems expert clinic findings, Part 1,” (2011), see http://pbadupws.nrc.gov/ High Perform. Comput. Appl. 18(4), 399–416 (2004). docs/ML1112/ML111240017.pdf. 6D. Jackson, M. Thomas, and L. I. Millett, Software for Dependable Sys- 14D. L. Parnas and P. C. Clements, “A rational design process: How and why tems: Sufficient Evidence (National Academy of Sciences, Washington DC, to fake it,” IEEE Trans. Software Eng. SE-12(2), 251–257 (1986). 2007). 15B. D. Floyd and S. Bosselmann, “Simplicity research in information and 7ISO/IEC/IEEE 24765, Systems and Software Engineering – Vocabulary communication technology,” IEEE Comput. 46(11), 26–32 (2013). (IEEE, New York, 2010). 16D. L. Parnas, “Software aging,” in Proceedings of the 16th International 8C. Ghezzi, M. Jazayeri, and D. Mandrioli, Fundamentals of Software En- Conference on Software Engineering (IEEE Computer Society Press, Los gineering, 2nd ed. (Pearson/Prentice-Hall, Upper Saddle River, NJ, 2002). Alamitos, 1994), pp. 279–287. 9D. L. Parnas, “On the criteria to be used in decomposing systems into mod- 17S. L. Pfleeger and J. M. Atlee, Software Engineering Theory and Practice, ules,” Commun. ACM 15(12), 1053–1058 (1972). 3rd ed. (Pearson/Prentice-Hall, New Jersey, 2006). 10P. Morgan, A. Samaroo, G. Thompson, and P. Williams, Software testing: 18B. Kitchenham and S. L. Pfleeger, “Software quality: The elusive target,” An ISEB foundation, edited by B. Hambling (The British Computer Soci- IEEE Software 13(1), 12–21 (1996). ety, Swindon, Wiltshire, England, 2007). 19Software Engineering in Health Care (SEHC), see http://www.sehc.info/ 11K. Naik and P. Tripathy, Software testing and quality assurance: Theory index.htm. and practice (John Wiley, New York, 2008). 20Foundations of Health Information Engineering and Systems, see 12C. A. Ericson, Hazard Analysis Techniques for System Safety (John Wiley, http://www.iist.unu.edu/www/workshop/FHIES/. New York, 2005). 21International Electrotechnical Commission, “Medical device software – 13U.S. Nuclear Regulatory Commission, “Research information letter 1001: Software life cycle processes. IEC Standard 62304 1st edition” (2006), see Software-related uncertainties in the assurance of digital safety sys- http://webstore.iec.ch/preview/info_iec62304%7Bed1.0%7Den_d.pdf.

Medical Physics, Vol. 41, No. 9, September 2014