Standalone applications testing and automation Desktop QA team Red Hat, Inc. 2014-02-02 WhoWho wewe areare Red Hat's Desktop QE team Quality assurance for: Desktop hardware Wireless, graphics, audio... Desktop application stack Desktop en ironments !UI management tools $%ce applications 2 FOSDEM 2014 02 February 2014 DistributionsDistributions underunder testtest RHEL7.0 Desktop RHEL )*+ updates Fedora Rawhide/RHEL next &atest versions testing: /etwork0anager Evolution 3 FOSDEM 2014 02 February 2014 DistributionsDistributions underunder testtest RHEL7.0 Desktop RHEL )*+ updates Fedora Rawhide/RHEL next &atest versions testing: /etwork0anager Evolution 4 FOSDEM 2014 02 February 2014 DistributionsDistributions underunder testtest RHEL7.0 Desktop RHEL )*+ updates Fedora Rawhide/RHEL next &atest versions testing: /etwork0anager Evolution 5 FOSDEM 2014 02 February 2014 DistributionsDistributions underunder testtest RHEL7.0 Desktop RHEL )*+ updates Fedora Rawhide/RHEL next &atest versions testing: /etwork0anager Evolution 6 FOSDEM 2014 02 February 2014 InfrastructureInfrastructure bitsbits Beaker manages availa1le machines and distri1utions Simple Test Harness task fetches test automation code and runs required tests Behave controls automation execution Dogtail/pexpect e.ecutes actions on "#-3&# 3ode review in Gerrit 4est case management - Nitrate 7 FOSDEM 2014 02 February 2014 InfrastructureInfrastructure bitsbits Beaker manages availa1le machines from the pool and distributions Simple Test Harness task fetches test automation code and runs required tests Behave controls automation execution Dogtail/pexpect e.ecutes actions on "#-3&# 3ode review in Gerrit 4est case management - Nitrate 8 FOSDEM 2014 02 February 2014 InfrastructureInfrastructure bitsbits Beaker manages availa1le machines from the pool and distributions Simple Test Harness task fetches test automation code and runs required tests Behave controls automation execution Dogtail/pexpect e.ecutes actions on "#-3&# 3ode review in Gerrit 4est case management - Nitrate 9 FOSDEM 2014 02 February 2014 InfrastructureInfrastructure bitsbits Beaker manages availa1le machines from the pool and distributions Simple Test Harness task fetches test automation code and runs required tests Behave controls automation execution Dogtail/pexpect e.ecutes actions on "#-3&# 3ode review in Gerrit 4est case management - Nitrate 10 FOSDEM 2014 02 February 2014 InfrastructureInfrastructure bitsbits Beaker manages availa1le machines from the pool and distributions Simple Test Harness task fetches test automation code and runs required tests Behave controls automation execution Dogtail/pexpect e.ecutes actions on "#-3&# 3ode review in Gerrit 4est case management - Nitrate 11 FOSDEM 2014 02 February 2014 BeakerBeaker http://beaker-project.org Automation and task execution system for la1s of test computers 70 / 1are-metal machines support 0ultiarchitecure support Flexi1le permissions system 12 FOSDEM 2014 02 February 2014 BeakerBeaker 0ost nota1le tasks used: /desktop/rhel7/install #nstalls the desktop components 8!/$0E-9DE: /desktop/simpletestharness fetches automation code from git-gzip sources starts speci<ed tests to 1e executed stores test output, reports and artifacts 13 FOSDEM 2014 02 February 2014 BeakerBeaker 0ost nota1le tasks used: /desktop/rhel7/install #nstalls the desktop components 8!/$0E-9DE: /desktop/simpletestharness fetches automation code from git-gzip sources starts speci<ed tests to 1e executed stores test output, reports and artifacts 14 FOSDEM 2014 02 February 2014 DogtailDogtail =ython !"I automation framework taking advantage of Accessi1ility technologies Based on 64*?=# @ toolkit-neutral technology, used 1y !4K, Qt, 0ozilla, &i1reOffice 15 FOSDEM 2014 02 February 2014 DogtailDogtail Dogtail-1ased upstream test suites Pi4i7i !/$0E ?oftware Evince 16 FOSDEM 2014 02 February 2014 SniffSniff ?niA is a !"# for 64*?=# structure 17 FOSDEM 2014 02 February 2014 ScriptsScripts andand unittestsunittests ?cripts *> "nittests *B BDD BDD @ Behaviour Driven Development 18 FOSDEM 2014 02 February 2014 GherkinGherkin !herkin @ >usiness Reada1le Domain ?peci<c &anguage 4est scenarios are human*reada1le list of steps to 1e performed ?teps are matched to python procedures, called step definitions 19 FOSDEM 2014 02 February 2014 GherkinGherkin Core keywords: Feature, Background, ?cenario ?cenarios can 1e grouped 1y feature or using tags 20 FOSDEM 2014 02 February 2014 BehaveBehave https://pypi.python.org/pypi-1ehave Python implementation of >DD approach Flexi1le output formatters (HT0&, plain etc.: ?etup / 1reakdown is implemented via 1efore_D / after_D procedures 21 FOSDEM 2014 02 February 2014 BehaveBehave 22 FOSDEM 2014 02 February 2014 TipsTips andand TricksTricks HTM& report with screenshots after each step and logs from journalctl ?creencast recording Detect app crashes via a1rt 23 FOSDEM 2014 02 February 2014 BenefitsBenefits ofof BDDBDD approachapproach 6utomated test scenarios are human-readable 3an be used as a instructions for manual tests Easy to update / enhance Scenarios can be written by designers draft new features document app behaviour Steps can be reused across several projects: Gnome Online Accounts handling Working with !NOME open / save file dialogs 24 FOSDEM 2014 02 February 2014 BenefitsBenefits ofof BDDBDD approachapproach 6utomated test scenarios are human-readable 3an be used as a instructions for manual tests Easy to update / enhance Scenarios can be written by designers draft new features document app behaviour Steps can be reused across several pro5ects: Gnome Online 6ccounts handling Working with GNOME open / save file dialogs 25 FOSDEM 2014 02 February 2014 BenefitsBenefits ofof BDDBDD approachapproach 6utomated test scenarios are human-readable 3an be used as a instructions for manual tests Easy to update / enhance Scenarios can be written by designers draft new features document app behaviour Steps can be reused across several pro5ects: Gnome Online 6ccounts handling Working with GNOME open / save file dialogs 26 FOSDEM 2014 02 February 2014 BenefitsBenefits ofof BDDBDD approachapproach "# a1straction in scenarios: Same scenarios can 1e used to test arious frontends: e.g. /etwork0anager's nmcli - nmtui E olution tests with minimal adjustments can 1e used for RHE& 6, RHE& 7, Fedora 20 and Fedora Rawhide Due to grouping scenarios 1y feature we can easily run regression check for aAected feature 27 FOSDEM 2014 02 February 2014 BenefitsBenefits ofof BDDBDD approachapproach "# a1straction in scenarios: Same scenarios can 1e used to test arious frontends: e.g. /etwork0anager's nmcli - nmtui E olution tests with minimal adjustments can 1e used for RHE& 6, RHE& 7, Fedora 20 and Fedora Rawhide Due to grouping scenarios 1y feature we can easily run regression check for aAected feature 28 FOSDEM 2014 02 February 2014 SuccessSuccess stories:stories: NM,NM, EvolutionEvolution Evolution testing across availa1le distri1utions Proposed patch testing /etwork 0anager "ses pe.pect instead of dogtail Scenarios can 1e re-used to test arious /0 frontends: nmcli, nmtui (ncurses-based: Sharing steps with !/$0E Control Center 29 FOSDEM 2014 02 February 2014 SuccessSuccess stories:stories: NM,NM, EvolutionEvolution Evolution testing across availa1le distri1utions Proposed patch testing /etwork 0anager "ses pe.pect instead of dogtail Scenarios can 1e re-used to test arious /0 frontends: nmcli, nmtui (ncurses-based: Sharing steps with !/$0E Control Center 30 FOSDEM 2014 02 February 2014 FutureFuture plansplans Working with Fedora QA to have a similar process for ,edora Execute tests directly in upstream 3#: !/$0E*continuous Build.kde.org 31 FOSDEM 2014 02 February 2014 FutureFuture plansplans Working with Fedora QA to have a similar process for ,edora Execute tests directly in upstream 3#: !/$0E*continuous Build.kde.org 32 FOSDEM 2014 02 February 2014 Questions 2014-02-02 33.