UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering QtWebKit layout tests DRT and Tools András Bécsi (bbandix) <[email protected]> 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Outline Monitoring the buildbot Update toQt 4.6 on theBuildBot ■ ■ ■ ■ ■ ■ BuildBotMiner BuildBot logs manual-tester layout tests Progress of the port measured by passing DumpRenderTree issues Font metric issues » » Brokentests Flakeytests AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 2 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Qt 4.6 issues Issueswith fonts ■ ■ metricsdo only work on our BuildBot howeverwe got feedback thatthese After the update of the expected files tests (~ 500) experienced pixel differencesin many After the update to Qt 4.6 we » are correct on the bot layout on developers machines but More than 300tests have incorrect AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 3 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Qt 4.6 font metric issues the 306 failing tests found onlyminor differences(5-10 tests) On my laptop managedI to reproduce We testedother major distributions, but ■ the same usewe on the buildbot We tried the default configuration of Qt 4.6, – buildbot and and my machine: buildbot differencesthe between package Suspicious to Default /usr/localinstall echo yes|./configure–opensource-no-webkit » » » libxext1.0.4→ 1.1.1 libfreetype2.3.7 2.3.11→ fontconfig2.6.0→ 2.8.0 AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 4 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Qt 4.6 Whythese differences? Howshold wefix this ■ ■ ■ Find the cause in Qt andfix that? Change the packages to older versions? dependent problems? that could cause these environment- Was something changed in the 4.6 release ~1000 now skipped tests skipped seem now to~1000 which have issue is solved issue if metric thisonly be differencesenabled could AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 5 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Proposal Proposal for new tests follow this proposal test dumpAsText() needed Lobbying to change current tests to Using Using render-tree dumps only reallyif – – platforms may differentneed results In this fontcase differences rendering between in the test E.g. if font metrics are important for some reason getComputedStyle() AndrásBécsi (bbandix) @ Nokia WebKit Code Camp if possible to validate the with 6 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering crashesDRT after change 4.6 Qt to Some tests seemed tocorrupt follow-ups ■ ■ ■ ■ ■ tests involved in notifyDone()DRT's problem. Fix landed, which reveled that there are more javascript correctly. did'nDRT clean up windows opened by Further investigation showed no crashes No stderr output run-webkit-tests reported crash DRT » Instead there was a non-blocking Instead I/Othere was a non-blocking EAGAIN/EWOULDBLOCK issue AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 7 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Other problems with DRT Phonon relatedcrashes Similaras before ■ ■ ■ ■ Only run-webkit-tests reports crash No stderr output No real crash because they cause the “crashes” Two tests are involvedand are skipped » » http/tests/security/local-video-source-from-remote.html fast/runin/nonblock-runin.html AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 8 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering features Missing DRT DRT was in a really bad shape There arestill many missing features ■ ■ Especially in jsobjects.h and jsobjects.cpp All components of DumpRenderTree were in » » » » » Wewould needsome restructuring tomatch the developmentofmissing features progresses refactoringAfter these toseparate filesthe TextInputController EventSender LayoutTestController implementationsofother ports AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 9 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Progress of Progress the port QtWebKit Tests we pass: Tests altogether: ■ ■ ■ ■ A A tool for making updates a bit easier: 5400 tests show layout problems or are new More than 6300, slowlyincreasing 11720 – → differences, or and correctare new tests. testshave font only related which pixel thereAsmentioned arealready approx. 1000 http://github.com/bbandix/manual-tester AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 10 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED $./manual-tester pathToQtLauncher ~/testResultsDir $ ./generateTestFile.sh ~/testResultsDir \ $ run-webkit-tests -o ~/testResultsDir Department of Softwar e Engineering manual-tester visually check new or pixel-different tests A not-so-userfriendly Qt 4 applicationto git://github.com/bbandix/manual-tester.git ~/path_to_WebKit/LayoutTests AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 11 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED manual-tester Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 12 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering The The time-zone issue searchingis hard features at the same timeis hard Skipping testsand implementing missing our bot in US time-zone Most of the work on the Mac port isdone The logs of the buildbotare short, Keeping the BuildBot green without Some developers didn't keep an eyeon AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 13 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Monitoring the buildbot more information month ago to have our buildbot one We started tomonitor ■ ■ information system a web-capable We started to design 1000 revisions data on more than Now havewe raw – BuildBotMiner AndrásBécsi (bbandix) @ Nokia WebKit Code Camp TIMEOUT=0 CRASH=24 FAIL=19 NEW=26 OK=6158 RUN=6227 JSCREGFIX=0 JSCREGFAIL=0 JSCREGRUN=1119 DATE="2009-11-16" PATCH_LLOC=744 FILES_CHANGED=11 Christiansen" AUTHOR="Kenneth Rohde Silva" REVIEWER="Gustavo Noronha REVISION=51044 BUILD=3890 SLAVE=Qt_Linux_Release MSG=SUCCESS 14 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering BuildBotMiner presentinthe data need to be doneyet make the information publiclyavailable What isit good for? Sophisticatedqueries and formsfor Weplan to fully monitor our new bots and Python Django/ based web application Script gathers theraw info and log files ■ ■ Loading the information to a database Parsing the information and logfiles » Additionalinformation toour benchmark site AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 15 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering BuildBotMiner Planned features Connect thebuildbot and the svn trac ■ ■ ■ ■ ■ Creating fancy charts andgraphs Gathering information about Logging build issues failing tests Pointing out the cause of over-the-night Monitoring and pointing out flakey tests – – Layout test debugging progress Layout test debugging process progress and Development AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 16 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 17 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 18 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 19 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 20 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 21 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering BuildBotMiner More soon on our blog: http://webkit.sed.hu AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 22 09/12/09 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Thanks for yourpatience! AndrásBécsi (bbandix) @ Nokia WebKit Code Camp 23.