ENHANCE FUEGO TEST EFFICIENCY BY APPLYING ADDITIONAL SOFTWARE AND HARDWARE SOLUTIONS OPEN SOURCE SUMMIT JAPAN 2017 2017/05/31 KHIEM NGUYEN STAFF ENGINEER, SOFTWARE ENGINEERING DIVISION, RENESAS DESIGN VIETNAM © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. AGENDA . Self introduction Page 03 . Fuego as a test automation solution Page 06 . Why Fuego . Known efforts of applying Fuego . A FUNCTIONAL automation test solution based on Fuego Page 13 . Feedback to Fuego upstream Page 24 . Future work Page 27 © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 2 BIG IDEAS FOR EVERY SPACE SELF INTRODUCTION © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. BIG IDEAS FOR EVERY SPACE ABOUT RENESAS AND RENESAS DESIGN VIETNAM Sales Companies Manufacturing and Renesas Electronics America Engineering Renesas Electronics Canada Service Companies Renesas Electronics Brasil-Servicos Renesas Electronics Europe (UK) Renesas Semiconductor Manufacturing Renesas Electronics Europe (Germany) Renesas Semiconductor Package & Test Solutions Renesas Electronics (China) Renesas Semiconductor (Beijing) Renesas Electronics (Shanghai) Renesas Semiconductor (Suzhou) Renesas Electronics Hong Kong Renesas Semiconductor (Malaysia) Renesas Electronics Taiwan Renesas Semiconductor (Kedah) Renesas Electronics Singapore Renesas Semiconductor Technology Renesas Electronics Malaysia (Malaysia) Renesas Electronics India Renesas Semiconductor KL Renesas Electronics Korea Design and Application Technologies Companies Renesas System Design Renesas EngineeringServices Renesas Design Vietnam Renesas Semiconductor Design (Beijing) Renesas Semiconductor Design (Malaysia) . Renesas Design Vietnam Co., Ltd. (RVC) was founded in October 2004, as one of the main design centers in Renesas group. Business Corporation Intersil Corporation . Business line: Design of semiconductor for both hardware and software. © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 4 BIG IDEAS FOR EVERY SPACE WHO AM I ? . Name：Khiem Trong. Nguyen (KHIEM Nguyen - キム グエン ) . Company：Renesas Design Vietnam . Career：09 years experiences in embedded software development . Development and verification for Mobile software platform . Development and verification for In-vehicle software platform . Development for test automation solutions of In-vehicle software platform . Email: [email protected] © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 5 BIG IDEAS FOR EVERY SPACE FUEGO AS A TEST AUTOMATION SOLUTION © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. BIG IDEAS FOR EVERY SPACE ABOUT FUEGO [1/2] 2014 2016 2017 Fuego v1.1 LTSI Test Fuego v1.0 “Blaze” Fuego next project (JTA) release release 3 important milestones © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 7 BIG IDEAS FOR EVERY SPACE ABOUT FUEGO [2/2] 2014 2016 2017 Fuego v1.1 LTSI Test Fuego v1.0 “Blaze” Fuego next project (JTA) release release . 1st release by May . Released by Nov. 2016 . Released by April 2017 . LAVA integration 2014 . More engagement from . Upgraded to Jenkins . etc . Adopted by AGL and community 2.32.1 community . Function enhancement . New Fuego directories . Run on Docker from from community layout 2015 . Wiki, Mailing list . New Fuego script system discussion . New transport for board communication © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 8 BIG IDEAS FOR EVERY SPACE PITFALLS OF SOFTWARE TEST ACTIVITIES P1 P2 P3 P4 Much human Hard to Hard to port Unstable effort due to modify test cases test results manual tests un-unified & from one among & repetitive close-source solution to testers test cycle test solution another common need © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 9 BIG IDEAS FOR EVERY SPACE SOLVE PITFALLS - WHY FUEGO ? Jenkins-based open source test automation solution Can Solved P1 P3 Easily expand the supported features, Portable by running Fuego utilizing the Jenkins on top of Docker plugins Can solved P3 P4 P2 P3 Can solved Able to utilize the existed features to Easily add more test cases optimize test execution individually or in groups P4 Can solved P1 Can solved © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 10 BIG IDEAS FOR EVERY SPACE USE FUEGO - THE KNOWN EFFORT [1/2] • Fujitsu contributed new ethtool test cases. 2014 • Mitsubishi showed how to run LTP efficiently with LTSI test project. 2015 • AGL deployed it as official AGL CIAT solution. • Toshiba proposed improvement on Post-processing tool to 2016 summarize LTP test result. © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 11 BIG IDEAS FOR EVERY SPACE USE FUEGO - THE KNOWN EFFORT [2/2] . Renesas has also put many efforts to apply Fuego into test automation solutions. Building up automation test for R-Car platform which could execute selected test cases without external hardware’s hotplug operation. Summarize the test results (logs) automatically and output them as a report (excel file). [1] http://elinux.org/images/1/12/The_Test_Activities_In_Renesas_Using_Fuego.pdf © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 12 BIG IDEAS FOR EVERY SPACE A FUNCTIONAL AUTOMATION TEST SOLUTION BASED ON FUEGO © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. BIG IDEAS FOR EVERY SPACE WHAT IS A FUNCTIONAL AUTOMATION TEST SOLUTION ? .Typical requirements for a functional automation test solution: Able to perform the tests Automatically notify status of R1 automatically as submitted R4 schedule test execution to testers Have efficient solution for Able to automatically trigger R2 device management R5 the test based on source code update (CIAT) Support managing test cases, Able to complete the test case R3 test results and test report R6 creation without human intervention © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 14 BIG IDEAS FOR EVERY SPACE FUEGO VS FUNCTIONAL AUTOMATION TEST SOLUTION Fuego Able to perform the test automatically as R1 supported submitted schedule Local and remote Have efficient solution for device Fuego board control service R2 management partly supported Need scalable test Support managing test case, test result and R3 database service test report creation Email Automatically notify status of test execution R4 to testers Fuego notification has not Git event Able to automatically trigger the test based supported R5 hook service on source code update (CIAT) yet Need hardware solution Able to complete the test case without R6 for board control human intervention © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 15 BIG IDEAS FOR EVERY SPACE BRIEF IMAGE ABOUT FUEGO INTEGRATED SYSTEM Host/TFTP PC Database Serial Connection Email Notification User_A Board A Adruino Network Switch User_B Fuego PC Board B Gerrit Fuego integrated system in our view point. User_C © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 16 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING SOFTWARE SOLUTIONS Host/TFTP PC . Lava as Fuego’s “board management” facility. Strong open-source solution for Serial Connection board management. Local boards and remote boards Support interface to extend board management capabilities, like Board A Adruino power supply ON/OFF, etc Switch Fuego PC Board B © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 17 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING SOFTWARE SOLUTIONS . Gerrit as Fuego’s “test event trigger” facility. Strong open-source solution for source code review and source User_A code management. Well-known combination to realize Network CI infrastructure. User_B Fuego PC User_C Gerrit © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 18 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING SOFTWARE SOLUTIONS Database . Testlink as Fuego’s “test case and test result database” facility. Fuego PC . Strong open-source solution for test case management. User_A Manage test specification. Manage test plan -> execution. Generate test execution matrix report. User_B Network Import/export . Jenkins plugin is available. User_C © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 19 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING HARDWARE SOLUTIONS . What features hardware solutions can support ? . Power supply control . Boot from power-on-reset (cool boot) or from hardware reset (warm boot) . Firmware update . Suspend-to-RAM’s resume trigger . External connections’ hotplug © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 20 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING HARDWARE SOLUTIONS Power supply control RSTMODE_SW=L EX_PWRONn=H/L Boot from power-on-reset (cool boot) or hardware reset (warm boot) From schematics of R-Car Starter Kit Pro © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 21 BIG IDEAS FOR EVERY SPACE REALIZE FUEGO INTEGRATED SYSTEM INTEGRATING HARDWARE SOLUTIONS Pin number Pin name 8 1 DAT3 7 2 CMD I/O 6 3 VSS 5 4 SD 4 VDD 3 CARD 5 CLK 2 6 VSS 1 7 DAT0 9 8 DAT1 9 DAT2 Control VDD pin to hotplug SDCard Control Vbus pin to hotplug USB Maintain the quality of GPIO signal is crucial for the hot-plug feature. Control pin19 to hotplug HDMI © 2017 Renesas Design Vietnam Co., Ltd. All rights reserved. Page 22 BIG IDEAS FOR EVERY SPACE IMAGE ABOUT FUEGO INTEGRATED SYSTEM Host/TFTP PC Database Serial Connection Email Notification User_A Board A Adruino Network Switch User_B Fuego PC Board B With