Challenges in Iot Testing

CHALLENGES IN IOT TESTING

Makhorov © Vadim Vadim ©

Ina Schieferdecker, Axel Rennoch DSTB, June 9th, 2016 IOT MARKET FORECAST

2 © Fraunhofer FOKUS FURTHER FORECASTS

Connected Mobiles worldwide

Source: Cisco Global Mobile Traffic Forecast Update, Gartner

Global data streams in the Internet per Second in Terabyte

Source: ITU ICT Facts and Figures 2015-2020 3 © Fraunhofer FOKUS IOT REFERENCE MODEL (ONE OF MANY)

4 © Fraunhofer FOKUS NEW ARCHITECTURAL PARADIGM

Today Upcoming

Hierarchical Orchestrated

ERP

MES

SCADA

PLC

I/O

Openess, Dynamicity, Scalability

5 ANYTHING NEW IN IOT TESTING ?! Similar - Protocol stacks - Protocol testing - IETF-based: CoAP, MQTT, etc. - Conformance - IEC-based: OPC-UA - Interoperability - ITU-based: M2M - Performance

- Application frameworks - Software testing - Eclipse: Kura, Scada, etc. - Component testing - Many others - Integration testing - System testing Different - Security - Security testing - ISO: common criteria - Risk-oriented testing - Mitre: CWE list - Fuzz testing - Others - Online testing

- Data - Data quality - Semantic real-time data 6 © Fraunhofer FOKUS FURTHER ASPECTS

IoT solutions often are … IoT test solutions need to … 1. in harsh, unreliable environments - Integrate simulators for environmental conditions

- Systematically determine reference 2. in highly dynamic configurations with configurations large number of – typically diverse – - sensors and actuators with open Adjust and scale test configurations interfaces and dynamically

- 3. In resource-constrained Be a real-time system by itself environments - Support test scenarios for hybrid systems (both events and streams)

à Test platform for the Internet of Things

à à embedded virtualized 7 © Fraunhofer FOKUS INTEGRATION OF SEVERAL TESTING APPROACHES

Software System Testing Testing IoT Protocol Testing Testing Security Testing Test Automation

8 © Fraunhofer FOKUS CHALLENGE TEST AUTOMATION

- TTCN-3 is the Testing and Test Control Notation - Internationally standardized testing language for formally defining test scenarios. Designed purely for testing

testcase Hello_Bob () { p.send(“How do you do?“); alt { []p.receive(“Fine!“); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } }

9 DESIGN PRINCIPLES OF TTCN-3

- One test technology for different tests - Distributed, platform-independent testing - Integrated graphical test development, documentation and analysis - Adaptable, open test environment

- Areas of Testing - Regression testing - Conformance and functional testing - Interoperability and integration testing - Real-time, performance, load and stress testing - Security testing

- Used for system and product qualification and certification, e.g. for GCF/PTCRB certification of handsets

10 TTCN-3 EXECUTION

API / Communication Software / Invocations

Tester SUT

System TTCN-3

API / Communication Software / Invocations

11 A TTCN-3 TEST SYSTEM

Test System User

TM: Management TL: Logging TE – TTCN-3 Executable TCI

TM – Test Management

TL – Test Logging TE Component

CD – Codec Handling CD: Codec CH: CH – Component Handling TRI SA – System Adapter SA: System Adapter PA: Platform Adapter PA – Platform Adapter

System Under Test (SUT) SUT – System Under Test

ETSI ES 201 873-1 TTCN-3 Core Language (CL) ETSI ES 201 873-5 TTCN-3 Runtime Interface (TRI) ETSI ES 201 873-6 TTCN-3 Control Interfaces (TCI)

12 IMPLEMENTATION

ATS Test System

TE +

Communication / SUT Invocation

13 TTCN-3 DOMAINS: TELECOM

- Industrial use - Big companies with hundreds of TTCN-3 engineers: Ericsson, Nokia, Siemens, Motorola - large distribution among SMEs

- Standardization bodies - Standardized test suites: ETSI / 3GPP (LTE)/ OMA / TETRA and its members - IMS performance benchmarking: Intel, HP, BT and others

- Test tool manufacturer: - Commercial Tektronix, Catapult, Nexus, R&S, Spirent, … - Free tools by Eclipse and academics

- Certification program based on TTCN-3: e.g. WiMax forum

14 TTCN-3 DOMAINS: AUTOMOTIVE

- Cockpit systems CD Changer - Edutainment Amplifier - Head units Head Unit / Tuner

Test MOST Bus

- Car-to-X communication er - Car-to-car, car-to-roadside, car-to-backbone Speaker - Autonomic driving

Telematics Applications in the Cockpit • Audio (CD / Radio), Video • Telephone, SMS • Navigation • Speech recognition • User interface for body electronic 15 CHALLENGE EMBEDDED SYSTEMS • Interfaces to support distributed • Concepts to deal with synchronous scheduling of Test System User time and continuous components signals • Interfaces to support transmission of • Concept that allow continuous signals between TM: Test Management TL: Test Loggingadvanced control flow components TCI for hybrid system

testingC t D n g e : n

i n : C l o d H o

p TE: Test Executable n D C a m e o H c C

TRI

SA: System Adapter PA: Platform Adapter

System Under Test (SUT) • Interfaces to support • Interfaces to support access to time and stimulation with and sampling evaluation of continuous signals 16 TTCN-3 EMBEDDED MODES

accelerate constant brake

Xn until [g1] until [g2] yn

SIGNAL GENERATION BUILDING BLOCKS

testcase signal_generation() runs on mtcType{ seq{ apply_noise(Throttle, 5.0, 5.0); apply_noise(Throttle, 10.0, 5.0); apply_ramp(Throttle, 10.0, 10.0, 2.0, 3); …}

17 AUTOMATED V2X TEST BED

Test Execution

V2X Test Bed and Tool Suite

Test Data Generation (e.g. Traffic Log File Analysis Simulation Data) and Visualization

18 V2X TEST BED ARCHITECTURE

template V2XMessage c2xMessage(in id := sid, payload := { protocolVersion := 1, actionID := 100, ICT cancelationFlag := false, LSS generationTime := {t, 0}, validityDuration := 100, IRS referencePosition := {longitude := {isE, protocolMsg := pm } } IVS send traffic jam traffic jam Test Control

Elements IVS • Test control with TTworkbench and V2X framework (allow for synchronized stimulation and evaluation of V2X system interaction) • Currently up to 4 IVS and 2 IRS systems to be connected to the test control over Ethernet • Optional integration with ICT and other hardware possible HMI

19 DRIVE C2X REFERENCE TESTS

• Compatible with ETSI Standards Example: Traffic Jam Ahead Warning • Virtualized Test Environment Tests with different jam configurations by varying: • number of vehicles in jam Tests available for: • velocity of vehicles • Stationary vehicle warning • distance to EGO • Road works warning • velocity of EGO • Slow vehicle warning

• Traffic jam ahead warning JAM is simulated by injecting CAM messages • In vehicle signage for the individual vehicles • Emergency vehicle warning, • Emergency electronic brake lights

20 CHALLENGE SECURITY TESTING

Security testing solutions for six industrial domains http://www.itea2-diamonds.org/

Risk Analysis (Compositional CORAS)

Functional Fuzz Testing Testing (Randomized Model-Based testing for Security Testing (Testing of unknowns) Contermeasures)

Test Automation

(TTCN-3 based)

Ina Schieferdecker, Model Based Security Testing: Selected Considerations (Keynote) Sectest 2011, Workshop on the 4th IEEE International Conference on Software Testing, Verification and Validation Berlin, Germany 21 FUZZ TESTING

1. Fuzzing originally describes the random generation of test vectors (Miller et. al. in the early 1990s).

2. Fuzzing is about injecting invalid or random inputs in order • to reveal unexpected behaviour • to identify errors and expose potential vulnerabilities.

negave input 3. Ideally, fuzzers generate semi-valid input data, space i.e. input data that is invalid only in small portions. speciﬁed (unlimited), target of random funconality fuzzing

4. Depending on fuzzer’s knowledge about the protocol, fuzzers can generate totally invalid to target of semi-valid input data model-based fuzzing

à Developed in DIAMONDS: new behavior-fuzzing approaches

see also: Takanen, DeMott, Miller: Fuzzing for Software Security and Quality Assurance. Artech House, 2008. 22 © Fraunhofer FOKUS G&D Case Study Banknote Processing Machines

23 © Fraunhofer FOKUS G&D Case Study Methodology

Risk Test Test code Test analysis Modeling generation generation execution

24 EVALUATION AND OPTIMIZATION

How evolved is my security testing process ?

How can I Improve my security testing process ?

25 CASE STUDY RESULTS

1. Collection of the experiences and results for all case studies • Case study experience sheets (www.itea2-diamonds.org) • Case study experience report (ETSI document)

2. STIP Evaluation • Shows progress in all case studies

26 The RACOMAT Tool

Combines component based, low level risk assessment with security testing - Risk analysis for component-based testing - Reusable risk assessment artifacts - Automated analysis of system components - Integrates with external data bases like MITRE CAPEC and MITRE CWE

- Risk-Based Security Testing, Test-Based Risk Assessment and automation with the help of Security Test Patterns and Security Testing Metrics - Semi-automated derivation of tests - Automated execution of tests

- Test environments as part of test setups

Using ComponentsUsing Components

Supporting System Networking ComponentsSupporting underSystem Test ComponentsNetworking Components under Test Components

In: Testumgebungen für eingebettete Systeme im Griff. Controlling Carsten Weise, SIGS Datacom Online Testing Issue, 2012

ComponentsControlling Components

- Combinations of real, virtualized and simulated components - Integration of monitors and impairment components - Management of test environments (configurations, versions, connections)

28 © Fraunhofer FOKUS IPV6 TESTBED INFRASTRUCTURE Hybrid infrastructure running virtualized images and real physical devices: - IPv6 Linux/FreeBSD/NetBSD/OpenBSD soft routers – XORP, Quagga, Zebra - Physical vendors‘ hardware (e.g. Cisco Routers) - Virtualization and Virtualization Management - VMware ESXi, Virtual Box, Xen and OpenStack/CloudStack (in the pipeline) - Test automatization and reporting based on scripting and various tools (tcpdump, wireshark, pcap, Perl, Python, bash, TTCN-3)

29 FURTHER EXAMPLES

- HL7/IHE testing in eHealth

- TCMS testing in transport

- Performance testing in mobile communication

- Data platform testing in open data

- etc.

30 © Fraunhofer FOKUS CHALLENGE IOT-INFRASTRUCTURE oneM2M: Horizontal layer of functions commonly needed across different segments

• It is software/middleware between application and data processing & communication HW • Integrated into devices/gateways/servers, e.g. sensors, actors, things, routers, cloud • Connects data producers and consumers in secure manner • Hides complexity of NW usage from apps • Controls when communication happens • Increases efficiency of data transport • Stores and shares data • Supports access control • Notifies about events • Talks to groups of things • Manages devices on large scale Source: onem2m.org 32 © Fraunhofer FOKUS ONEM2M ABSTRACT PROTOCOL TESTER

Test System System Under Test Test configurations

ATS IUT TEST SYSTEM IUT

AE Mca CSE

HTTP/CoAP/MQTT HTTP/CoAP/MQTT TCP/UDP TCP/UDP IP IP

Lower Layers Lower Layers TEST SYSTEM IUT AE Mca

CSE

CSE Mcc

Lower Layers link

Source: oneM2M; TS-0015-V-0.5.1; Testing Framework, 2016-May-13 33 © Fraunhofer FOKUS ONEM2M TTCN-3 TEST SPECIFICATION PUBLIC AVAILABLE

Group 1: AE(AE); Group 2: CSE(CSE) Subgroup - Registration(REG) Subgroup - Data Management and Repository(DMR) Subgroup - Subscription and Notification(SUB) Subgroup - Group Management(GMG) Subgroup - Discovery(DIS) Subgroup - Location(LOC) Subgroup - Device Management(DMG) Subgroup - Communication Management and Delivery Handling(CMDH) Subgroup - Security(SEC)

Main modules Supplementary modules C T EL-ITP E EL-TM L

AL-TM C T Test Industrial Consumer Security A AL-TA Autom IoT ?! IoT ?! L AL-TTA

Software C Foundation Auto T Agile MBT Usability Mobile IoT ?! motive F Embedded L Systems Foundation ?!

35 © Fraunhofer FOKUS is the network of excellence is a leading provider of for the soware cerﬁcaon examinaons development industry in all over the world, head- German-speaking European quartered in Germany with countries. subsidiaries in London, Boston and Amsterdam. 1.400 globally acve compa- Focusing on IT professions, iSQI nies, specialists, instuons of plays a large role in cerfying higher educaon and research the know-how of professionals instutes are members of in over 90 countries on 6 ASQF and share the commit- connents in 10 languages. In ment to guarantee quality 2015, iSQI examines more than standards in ICT. 22.000 individuals.

Working group on IoT Quality Engineering

- Challenges and risks - Concepts, methods and approaches - Syllabus, exams, training material

- Mainly members from industry (e.g. Festo, Siemens, SAP)

- If you are interested in status/results, please drop an email: [email protected]

SUMMARY: CHALLENGES IN IOT TESTING

1. Combination of software, system, protocol and security testing

2. Need for high-degree of test automation

3. Management of distributed, flexible and/or virtualized test environments including test, simulation, SUT components and devices

4. Testing new IoT infrastructure (service layers)

5. Development of expertise and experiences in IoT Testing

38 Thank you for your attention! www.fokus.fraunhofer.de (System Quality Center)

39 CONTACTS

Fraunhofer FOKUS Kaiserin-Augusta-Allee 31 10589 Berlin, Germany www.fokus.fraunhofer.de

Axel Rennoch Ina Schieferdecker Project Manager Director [email protected] [email protected] Phone +49 30 3463-7344 Phone +49 30 3463‐7241