RTEMS SMP qualification - status
Marcel Verhoef, Javier Fernandez Salgado
ESA ESTEC
08/04/2021
ESA UNCLASSIFIED – For ESA Official Use Only 1 Background (1)
• GSTP activity started in February 2019, consortium is composed of • EDISOFT (Portugal – consortium lead) RTEMS qualification experience, strong ties with industry • Embedded Brains (Germany) RTEMS SMP development expertise, strong ties with community • LERO (University of Limerick, Trinity College Dublin, Ireland) formal methods expertise • Jena-Optronik (Germany) end user in space domain, application qualification expertise • CISTER (Portugal) real-time software and software qualification expertise
• Investment: 700 kEuro • Originally planned for 24 months (Feb 2019 – Feb 2021) • Delays incurred due to technical complexity and COVID-19 Dec 2021
• Activity is executed in close collaboration with the RTEMS community and end-users in the space domain
2 Background (2)
• To complement many completed ESA sponsored R&D for RTEMS • EDISOFT RTEMS (http://rtemscentre.edisoft.pt) • Based on RTEMS 4.8.0, qualified to DAL-B, applied in many space missions • Open source, but qualification data pack is licensed • Available for ERC32, LEON2, LEON3 (single core only, no LEON4 support) • This product is actively maintained by EDISOFT (latest is R15, issued in Oct 2020) and will remain available, the new activity will not replace this product • Contact EDISOFT on license cost and support contracts
• RTEMS-SMP, available in the RTEMS mainline, part of 5.x (release) / 6.x (dev) • Co-developed with the RTEMS community, with significant ESA investment, now production ready for LEON3 dual-core and LEON4 quad-core (final reports of this activity available at http://microelectronics.esa.int/NGMP) • Several device drivers made SMP compliant by Cobham Gaisler (own investment)
3 Objectives of the qualification activity (1)
• Production of a (pre-) qualification toolkit that allows end-users to qualify their (space) application software on bespoke (space-qualified) hardware • Primary focus is on qualifying the SMP elements of the RTEMS super core, target application area is payload (instrument) data processing, exact scope is defined in “space subset” • Qualification of RTEMS 5.x /6.x on LEON2/3 single core is not a priority ( RTEMS EDISOFT) • Base-line target platforms are the Cobham Gaisler GR712RC (LEON3 dual core) and GR740 (LEON4 quad core) System-On-Chips and associated Gaisler reference boards • The pre-qualification toolkit uses the GCC-based cross-compiler provided by the RTEMS Source Builder as baseline (RSB - currently at GCC v10.2.1, but this may evolve during the project) • Alignment with the (qualified) Mathematical Library for Flight Software (MLFS) see https://essr.esa.int/project/mlfs-mathematical-library-for-flight-software and (under development, qualified Q4/2021) LibmCS (complete libm for critical systems) • Currently out-of-scope (not fitting with project time and budget constraints): LLVM / clang compiler support and other multi-core SoC architectures (i.e. ARM based, RISC-V)
• In principle self-sustaining after contract close (fully community supported) except for ESA PA assessment
4 Objectives of the qualification activity (2)
• The Qualification toolkit contents • Curated version of the source code and related documentation, including all resources needed (i.e. compiler, build scripts) to build RTEMS itself • All verification and validation evidence, obtained from analysis, testing and proof, for the identified target configurations, in the form of documents required by ECSS-E-ST-40C and ECSS-Q-ST-80C Rev 1 • Curated test suite and all supporting resources required to automatically execute the test suite and reproduce the test evidence for each identified target configuration • Set of instructions of how to use the qualification toolkit (user manual)
• The challenge is to keep this activity as close as possible to the RTEMS main-line evolution • The qualification toolkit will be fully open source and free of charge • Other standards (i.e. DO-178C, IEC 61508, ISO 26262) have been investigated, and ECSS qualification approach is in line with those standards but formal compliance is out of scope for the ESA contract (but might be community contributed)
5 Objectives of the qualification activity (3)
In summary
• Lower the cost of qualifying by providing a pre-qualified RTEMS and ready-to-use supporting eco-system
• Sustainable long-term solution by leveraging community support, enabling change and innovation while maintaining quality and stability
• Easy to adopt, use and maintain
The qualification data pack significantly reduces the qualification effort needed by the end-user but there is still work left to do!
6 RTEMS SMP Qualification – status (1)
• The RTEMS SMP functionality is already fully functional and tested (on both GR712 and GR740) and is commonly available as part of the current RTEMS 5.x / 6.x main-line
• Only minor API changes have been made as part of the qualification activity (available in 6.x development) - it can be used already for software development as per today
• The qualification scope (subset of the RTEMS eco-system) is well-defined and has been discussed with and reviewed by the community in March 2019 and has not changed significantly since July 2019 (Technical Note: Space Profile, Release 2, 18 July 2019); latest release (Release 3, March 2021, final) covers only minor API changes.
• Minimal driver support included: GPIO, UART and SpW. (no MIL-STD-1553 support)
• The space profile for RTEMS 5.x / 6.x will be pre-qualified to ECSS SW criticality category-C Caveat: a clause-by-clause compliance matrix for both ECSS E-40 and Q-80 is provided!
7 RTEMS SMP Qualification – status (2)
• The qualification data pack will contain all verbatim RTEMS APIs – also those outside of the qualification perimeter are included (to remain in-line with community RTEMS – nothing is removed); these are split over two libraries to clearly distinguish the qualified from the non-qualified parts
• The qualification data packs are available in open source (permissive license model) at no cost to the end users – no geographic restrictions
• The qualification data packs (for GR712RC and for GR740) will be issued to the community at large via new portal web-site https://rtems-qual.io.esa.int/ (under approval)
• Regular community releases (growing only in coverage completeness) of the qualification data packs will be made until the project passes AR - to allow early adoption by end-users
• Full (dockerised) continuous integration pipe-line has been set-up in ESA TEC-SW software lab, to enable quick turn-around times for delta-certification of (any) change to the RTEMS eco-system.
8 RTEMS SMP Qualification Data Packs
• CDR data pack: available March 2021 (SMP versions only) • QR1 data pack: available July 2021 (all configurations) • QR2 data pack: available Oct 2021 (all configurations) • AR data pack: available Dec 2021 (all configurations)
• Four types of qualification data packs are available • LEON4 – SMP (qualified on GR740 quad-core) • LEON4 – classic RTEMS (qualified on GR740 using a single core only) • LEON3 – SMP (qualified on GR712RC dual-core) • LEON3 – classic RTEMS (qualified on GR712RC using a single core only)
9 Pre-qualification – what does that mean?
• Only a subset of the RTEMS APIs is taken into account in the qualification activity, if features are used that are outside the agreed space profile, they still need to be qualified in the scope of the end-user project typically this applies to drivers (legacy / reuse / heritage)
• The RTEMS pre-qualification runs on two Gaisler reference boards under a limited number of configurations (i.e. CPU speeds, memory and cache configuration settings,…) this requires the tests to be repeated on the bespoke hardware to reproduce the test evidence applicable to the end user project
• RTEMS is statically linked (as a library) to the application code using a flat memory model without protection (no user / kernel space distinction); this implies that ASW behavior may influence RTEMS and the other way around RTEMS pre-qualification does not reduce the need on qualification of RTEMS and ASW combined. RTEMS is like any other re-used sw library w.r.t. qualification and ISVV needs!
• The RTEMS ECSS documentation provided in the qualification data pack will be partial (i.e. RB, VTS-RB, VTR-RB and dependability analysis is application / system specific) this must be complemented by the end-user (guidelines, examples and templates are provided in the qualification data pack)
10 RTEMS QUALIFICATION DATA PACK CONTENTS CONSORTIUM PRESENTATION
11 RTEMS SMP Ready to Fly RTEMS Qualification Data Package Content & Usage
CISTER Research Centre in Real-Time & Embedded Computing Systems Sebastian Huber embedded brains GmbH What is in the QDP?
● Delivery in two parts: SCF + archive file
● Binaries of needed tools (compiler, linker, provided “as is”, ready to distribute)
● Pre-qualified part of RTEMS operating system
● Extra (not pre-qualified) part of RTEMS
● Documentation (RTEMS, ECSS, technical notes)
● Other stuff (sources, tests, Dockerfile)
CISTER 2021-04-08 Research Centre in 2/11 Real-Time & Embedded Computing Systems What do you need to get your application qualified?
● Hardware: based on Gaisler GR712RC or GR740
● System requirements (what shall your application do?)
● Application software realizing those requirements – Uses pre-qualified parts of RTEMS – Built by tools from QDP
● Documents showing that ECSS standards are met by your application (reference documents provided by QDP)
● Qualification authority to escort the development of your application (usually ESA)
CISTER 2021-04-08 Research Centre in 3/11 Real-Time & Embedded Computing Systems How do your parts integrate with the QDP?
Your Your software qualification application RTEMS documentation & tests Tools
Compiler, Linker, Pre-qualified Extra part etc. part RTEMS RTEMS Pre-qualification (Space Profile, (POSIX API) documentation Classic API) QDP Deployed on your qualified hardware (Gaisler GR712RC or GR740)
CISTER 2021-04-08 Research Centre in 4/11 Real-Time & Embedded Computing Systems Background Information: How is a QDP built?
● QDP content and build process is defined by configuration items (shares the infrastructure also used for the RTEMS specification)
● Main QDP build steps – Clone necessary Git repositories (rtems, rsb, rtems-tools, rtems-docs, etc.) – Build the tools using the RSB – Build RTEMS in various configurations (debug, coverage, pre-qualified, with extras) – Run the tests of various configurations on the reference board or simulator – Generate documents (also using the raw output obtained from test runs to get for example the code coverage data and the test results for the Software Validation Report with Respect to TS) – Package the deployed artifacts in an archive – Generate the Software Configuration File (SCF) using the built QDP (the output of “make” and pkg-config included in the SCF is generated from a real execution of the programs, this is a partial acceptance test)
CISTER 2021-04-08 Research Centre in 5/11 Real-Time & Embedded Computing Systems Which documentation will be provided?
● Software Configuration File – accompanies QDP, overview, content, tutorial
● Space Profile – what has been pre-qualified?
● Standard RTEMS Documentation – RTEMS Classic API Guide – description of RTEMS and its API – RTEMS User Manual – how to use RTEMS? – RTEMS Software Engineering – how to maintain RTEMS?
● Technical Note QT-109 – Core document which includes the planning & content of documents – ECSS tailoring and compliance matrix – Analysis of other standards such as IEC 61508
● Interface Control Document & Software Requirement Specification – requirements
● Software Verification Report & SPAMR – manual qualification activities, static code analysis, metrics, test results, code coverage, traceability, etc.
CISTER 2021-04-08 Research Centre in 6/11 Real-Time & Embedded Computing Systems Which verification activities have been performed?
Verification checks that all project activities meet ECSS requirements
● Documents: svr.pdf and spamr.pdf in QDP
● Automated verification where possible
● Static Analyzer: Coverity, Clang Static Analyzer, Cppcheck
● Anticipated for 2022: Independent verification to meet Criticality Category B requirements
CISTER 2021-04-08 Research Centre in 7/11 Real-Time & Embedded Computing Systems Formal Techniques to support Qualification
● Exploration – Looked at theorem provers and model-checkers ● Isabelle/HOL, Frama-C, TLA+, Spin, ... ● Chosen Technology – Promela/SPIN – Strong track-record, and good fit to RTEMS community – Focus: test generation ● Status – Proof-of-concept tests currently in QDP (Events Manager) ● rtems-6-sparc-gr7*-smp-1/src/rtems/testsuites/validation/tr- model-*.c ● rtems-6-sparc-gr7*-smp-1/sparc-rtems6/gr*-qual-only/tests/ ts-model-0.exe – Now looking at Thread Queue implementations
CISTER 2021-04-08 Research Centre in 8/11 Real-Time & Embedded Computing Systems How to use the QDP
● Start with the Software Configuration File (SCF) – Getting started guide – Hints to ease the migration from previous RTEMS versions – Document overview with links ● Required host system – Native Debian 10 (Buster) or other Linux distribution with a compatible glibc – Docker container (FROM: debian:buster, see SCF) ● Unpack to /opt ● Create directory ~/hello, copy /opt/rtems-6-sparc-*/src/example there ● Run “make” to compile and link ● Run on simulator ● Run on hardware
CISTER 2021-04-08 Research Centre in 9/11 Real-Time & Embedded Computing Systems What is NOT included in QDP?
● Training
● Support services More efficient ● Expert knowledge to than self-exploration
– Customize the QDP Consultants / Service providers – Support new architectures/BSPs
– Extend the pre-qualified scope e.g. (e.g. POSIX, OpenMP)
● Long term maintenance
CISTER 2021-04-08 Research Centre in 10/11 Real-Time & Embedded Computing Systems Contact
● QDP tools and project lead provided by EDISOFT: [email protected]
● RTEMS engineering provided by embedded brains GmbH: [email protected]
● Formal methods expert: [email protected]
CISTER 2021-04-08 Research Centre in 11/11 Real-Time & Embedded Computing Systems Summary
• First public QDPs will be made available via https://rtems-qual.io.esa.int/ to community at large (including these slides)
• ESA gitlab issue tracker will be enabled for Q&A
• Complete QDP (Cat-C) is expected in Dec 2021
• Complementary ISVV activity (to lift to Cat-B pre-qualification) is has been approved by IPC (Nov 2020) ITT is expected to appear in April 2021
• Contact points: • [email protected] (new technical officer of the GSTP contract) • [email protected] (head of flight software section – TEC-SWF) • [email protected] (head of software quality assurance section – TEC-QQS)
12