CHALLENGES in IOT TESTING © Vadim © Vadim Makhorov
Total Page:16
File Type:pdf, Size:1020Kb
CHALLENGES IN IOT TESTING © Vadim © Vadim Makhorov Ina Schieferdecker TestNet, May 11, 2016 TALKING PLANTS, ANIMALS AND MORE http://www.iot-a.eu/public 3 © Fraunhofer FOKUS IOT MARKET FORECAST 4 © Fraunhofer FOKUS FURTHER FORECASTS Connected Mobiles worldwide Source: Cisco Global Mobile Traffic Forecast Update, Gartner Global data streams in the Internet per Second in Terabyte Source: ITU ICT Facts and Figures 2015-2020 5 © Fraunhofer FOKUS IOT REFERENCE MODEL (ONE OF MANY) 6 © Fraunhofer FOKUS NEW ARCHITECTURAL PARADIGM Today Upcoming Hierarchical Orchestrated ERP MES SCADA PLC I/O Openess, Dynamicity, Scalability 7 CRITICALITY IMPLY HIGH QUALITY REQUIREMENTS »Implementation of real-time enabled CPS solutions will place high demands on the availability of services and network infrastructure in terms of space, technical quality and reliability.« In: Securing the future of German manufacturing industry. Recommendations for implementing the strategic initiative INDUSTRIE 4.0, Forschungsunion, acatech, Apr. 2013. Priorities of Quality, Time and Costs In: Stand und Trends der Qualitätssicherung von vernetzten eingebetteten Systemen, Fraunhofer FOKUS Studie, Aug. 2014 8 © Fraunhofer FOKUS ANYTHING NEW IN IOT TESTING ?! Similar Protocol stacks Protocol testing IETF-based: CoAP, MQTT, etc. Conformance IEC-based: OPC-UA Interoperability ITU-based: M2M Performance Application frameworks Software testing Eclipse: Kura, Scada, etc. Component testing Many others Integration testing System testing Different Security Security testing ISO: common criteria Risk-oriented testing Mitre: CWE list Fuzz testing Others Online testing Data Data quality Semantic real-time data 9 © Fraunhofer FOKUS FURTHER ASPECTS IoT solutions often are … IoT test solutions need to … 1. in harsh, unreliable environments Integrate simulators for environmental conditions Systematically determine reference 2. in highly dynamic configurations with configurations large number of – typically diverse – sensors and actuators with open Adjust and scale test configurations interfaces and dynamically 3. In resource-constrained Be a real-time system by itself environments Support test scenarios for hybrid systems (both events and streams) Test platform for the Internet of Things embedded virtualized 10 © Fraunhofer FOKUS INTEGRATION OF SEVERAL TESTING APPROACHES Software System Testing Testing Protocol IoT Testing Security Testing Testing Test Automation 11 © Fraunhofer FOKUS CHALLENGE TEST AUTOMATION TTCN-3 is the Testing and Test Control Notation Internationally standardized testing language for formally defining test scenarios. Designed purely for testing testcase Hello_Bob () { p.send(“How do you do?“); alt { []p.receive(“Fine!“); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } } 12 DESIGN PRINCIPLES OF TTCN-3 One test technology for different tests Distributed, platform-independent testing Integrated graphical test development, documentation and analysis Adaptable, open test environment Areas of Testing Regression testing Conformance and functional testing Interoperability and integration testing Real-time, performance, load and stress testing Security testing 13 TTCN-3 EXECUTION API / Communication Software / Invocations Tester SUT System TTCN-3 API / Communication Software / Invocations 14 A TTCN-3 TEST SYSTEM Test System User TM: Management TL: Logging TE – TTCN-3 Executable TCI TM – Test Management TL – Test Logging TE CD – Codec Handling CD: Codec CH: Component CH – Component Handling TRI SA – System Adapter SA: System Adapter PA: Platform Adapter PA – Platform Adapter System Under Test (SUT) SUT – System Under Test ETSI ES 201 873-1 TTCN-3 Core Language (CL) ETSI ES 201 873-5 TTCN-3 Runtime Interface (TRI) ETSI ES 201 873-6 TTCN-3 Control Interfaces (TCI) 16 IMPLEMENTATION ATS Test System TE + Communication / SUT Invocation 17 TTCN-3 DOMAINS: TELECOM Industrial use Big companies with hundreds of TTCN-3 engineers: Ericsson, Nokia, Siemens, Motorola large distribution among SMEs Standardization bodies Standardized test suites: ETSI / 3GPP (LTE)/ OMA / TETRA and its members IMS performance benchmarking: Intel, HP, BT and others Test tool manufacturer: Commercial Tektronix, Catapult, Nexus, R&S, Spirent, … Free tools by Eclipse and academics Certification program based on TTCN-3: e.g. WiMax forum 19 TEST SYSTEM EXAMPLE: OMA SUPL Secure User Plane Location Protocol Single MTC controls e.g.: UlpPort (Lup interface) IpcPort (IP configuration) TTCN-3 MTC MTC BSF DNS smsPort used for SMS Executable UtpPort for upper tester commands ssc nwcipi ipc ulpbsf dns sms sms utp IpiPort (IP information, e.g. release) NwcPort: network bearer control, SUT e.g. handover trigger Adapter SscPort: satellite simulation control, e.g. scenario trigger MT_SMS, WAP_Push, SIP_Push TLS (PSK) SMS Upper Satellite Tester TCP/UDP/IPTCP/UDP/IP Simulator Adapter e.g. text Network Bearer SUT Upper IUT ((SUPLSUPL Tester (SUPL Implementation) Terminal)Terminal) 20Server TTCN-3 DOMAINS: AUTOMOTIVE Cockpit systems CD Changer Edutainment Amplifier Head units Head Unit / Tuner Tester MOST Bus Car-to-X communication Car-to-car, car-to-roadside, car-to-backbone Speaker Autonomic driving Telematics Applications in the Cockpit • Audio (CD / Radio), Video • Telephone, SMS • Navigation • Speech recognition • User interface for body electronic 22 CHALLENGE EMBEDDED SYSTEMS • Interfaces to support distributed • Concepts to deal with synchronous scheduling of Test System User time and continuous components signals • Interfaces to support transmission of • Concept that allow continuous signals between TM: Test Management TL: Test Loggingadvanced control flow components TCI for hybrid system testing CoDec CD: TE: Test Executable CH: Handling Component TRI SA: System Adapter PA: Platform Adapter System Under Test (SUT) • Interfaces to support • Interfaces to support access to time and stimulation with and sampling evaluation of continuous signals 23 TTCN-3 EMBEDDED MODES accelerate constant brake Xn until [g1] until [g2] yn SIGNAL GENERATION BUILDING BLOCKS testcase signal_generation() runs on mtcType{ seq{ apply_noise(Throttle, 5.0, 5.0); apply_noise(Throttle, 10.0, 5.0); apply_ramp(Throttle, 10.0, 10.0, 2.0, 3); …} 24 PedalPositions v_act 2 1 <phi_Acc, phi_Brake> <phi_Acc, v_act ped_min <phi_Brake> <phi_Acc> <phi_Brake> <phi_Acc> <phi_Acc> T_100_Drive(v) T_70_Drive(v) >= >= BrakePedal AccPedal 70 <phi_Acc> INTEGRATION IN ML/SL T_max_Brake boolean boolean 1/30 1/70 BrakePedal AccPedal 1/100 <AccPedal, BrakePedal> <AccPedal, T_des_Brake Mu selection f(u) T_des_Drive <T_des_Drive, T_des_Brake> <T_des_Drive, PedalFlags 2 DesiredTorques // accelerate vehicle until 35 1 // ms and activate ACCS cont{ onentry{v_other.value:= 25.0} phi_acc.value:=80.0; } until{ 1. Introduce a vehicle ahead [v_ego.value > 35.0] { 2. Accelerate the ego vehicle until its phi_acc.value:=0.0; velocity rises to more than 35 m/s. lever_pos.value:= MIDDLE; 3. Activate the cruise control. } 4. The ACC shall switch to the velocity } control mode and the vehicle shall hold the current velocity // wait for several seconds 5. When the vehicle reaches the safe wait(now+10.0); distance to the vehicle ahead the ACC // evaluate shall switch to distance control mode. cont{ 6. The ACC should now adjust the running assert(v_ego.value <= 38.0); } velocity according to the vehicle ahead until{ 7. The safe distance to the vehicle ahead [d_other.value < sd] { ... must be guaranteed. 25 25 AUTOMATED V2X TEST BED Test Execution V2X Test Bed and Tool Suite Test Data Generation (e.g. Traffic Log File Analysis Simulation Data) and Visualization 26 THE SIMTD SET UP IN THE LAB 27 Hier steht die Fußzeile V2X TEST BED ARCHITECTURE template V2XMessage c2xMessage(in id := sid, payload := { protocolVersion := 1, actionID := 100, ICT cancelationFlag := false, LSS generationTime := {t, 0}, validityDuration := 100, IRS referencePosition := {longitude := {isE, protocolMsg := pm } } IVS traffic jam traffic Test Control Elements IVS • Test control with TTworkbench and V2X framework (allow for synchronized stimulation and evaluation of V2X system interaction) • Currently up to 4 IVS and 2 IRS systems to be connected to the test control over Ethernet HM • Optional integration with ICT and other hardware possible I 28 EXAMPLE: WEATHER WARNING • IVS1: Generate situation WiperSystem := { Front := "normal", Rear := "idle" } WiperSystem := { Front := "fast", Rear := "idle" } • IVS2: Check message reception DENM message received ? • IVS2: Check HMI interaction 29 DRIVE C2X REFERENCE TESTS • Compatible with ETSI Standards Example Traffic Jam Ahead Warning (TJAW): • Virtualized Test Environment Tests TJAW with different jam configurations by varying: Tests available for: • number of vehicles in jam • Stationary vehicle warning • velocity of vehicles • Road works warning • distance to EGO • Slow vehicle warning • velocity of EGO • Traffic jam ahead warning • In vehicle signage JAM is simulated by injecting CAM messages • Emergency vehicle warning, for the individual vehicles • Emergency electronic brake lights 30 SIMTD REFERENCE TESTS • 40 Communication tests and test variants CAM variants CAM frequencies, message life time handling etc. DENM variants • 20 Application tests testing event detection, propagation, handling and user Improve Quality notification for several V2X applications • Reference circuit event handling and user notification for several V2X applications • Reference circuit with load