UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering
QtWebKit layout tests DRT and Tools
András Bécsi (bbandix)
g
E
n
i
G
r
e Outline
E
e
Z
n
i
S
g
n
F Update to Qt 4.6 on the BuildBot
E
O
e
r
a
Y
w ■
T
t Font metric issues
f
I
S
I
o
S
S
S
f
R
N
o
E
E ■ DumpRenderTree issues
t
I
V
n
D
I
e
E
m
N
t
G
r ■ U Progress of the port measured by passing
E
a
Z
p
e
S layout tests
D
M
U
R
A ■ I manual-tester
T
N
E I C Monitoring the buildbot
S
S
A
T ■ BuildBot logs
I
S
R » Flakey tests
E
V
I
N » Broken tests
U ■ BuildBotMiner
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 2 D
g
E
n
i
G
r
e Qt 4.6 issues
E
e
Z
n
i
S
g
n
F
E
O
e
r Issues with fonts
a
Y
w
T
t
f
I
S
I
o
S
S S ■After the update to Qt 4.6 we
f
R
N
o
E
E
t
I
V
n
D
I experienced pixel differences in many
e
E
m
N
t
G
r
U
E a tests (~ 500)
Z
p
e
S
D
M
U ■After the update of the expected files
R
A
I
T however we got feedback that these
N
E
I
C metrics do only work on our BuildBot
S
S
A
T »More than 300 tests have incorrect
I
S
R
E
V layout on developers machines but
I
N U are correct on the bot
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 3 D
g
E Qt 4.6 font metric issues
n
i
G
r
e
E e
Z n We tested other major distributions, but
i
S
g
n
F
E
O e found only minor differences (5-10 tests)
r
a
Y
w
T
t
f
I
S
I o ■ We tried the default configuration of Qt 4.6,
S
S
S
f
R
N
o
E
E
t I the same we use on the buildbot
V
n
D
I
e
E
m
N
t
G
r echo yes | ./configure –opensource -no-webkit
U
E
a
Z
p
e
S Default install to /usr/local
D
M
U
R
A I On my laptop I managed to reproduce
T
N
E I the 306 failing tests
C
S
S – A Suspicious package differences between the
T
I
S buildbot and my machine:
R
E
V
I » fontconfig 2.6.0 → 2.8.0
N
U » libfreetype 2.3.7 → 2.3.11 » libxext 1.0.4 → 1.1.1 09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 4 D
g
E
n
i
G
r
e Qt 4.6
E
e
Z
n
i
S
g
n
F
E
O
e
r Why these differences?
a
Y
w
T
t
f
I
S
I
o
S
S
S
■ Was something changed in the 4.6 release
f
R
N
o
E
E
t
I
V
n
D
I e that could cause these environment-
E
m
N
t
G
r
U
E
a
Z p dependent problems?
e
S
D M ~1000 now skipped tests which seem to have
U
R
A
I only metric differences could be enabled if this
T
N
E issue is solved
I
C
S
S A How shold we fix this
T
I
S
R ■ E Change the packages to older versions?
V
I
N
U ■ Find the cause in Qt and fix that?
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 5 D
g
E
n
i
G
r
e Proposal for new tests
E
e
Z
n
i
S
g
n
F E Using render-tree dumps only if really
O
e
r
a
Y
w
T t needed
f
I
S
I
o
S
S
S
f
R
N
o – E.g. if font metrics are important for some reason
E
E
t
I
V
n
D
I
e
E in the test
m
N
t
G
r
U
E
a
Z
p
e –
S In this case font rendering differences between
D
M
U platforms may need different results
R
A
I T N Using getComputedStyle() with
E
I
C
S dumpAsText() if possible to validate the
S
A
T
I
S test
R
E
V I
N Lobbying to change current tests to U follow this proposal
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 6 D
g
E
n
i
G
r
e DRT crashes after change to Qt 4.6
E
e
Z
n
i
S
g
n
F
E
O
e
r Some tests seemed to corrupt follow-ups
a
Y
w
T
t
f
I
S
I
o
S
S
S
■ run-webkit-tests reported DRT crash
f
R
N
o
E
E
t
I
V
n
D
I
e
E ■ No stderr output
m
N
t
G
r
U
E
a
Z
p
e
S
■ Further investigation showed no crashes
D
M
U
R » Instead there was a non-blocking I/O
A
I
T EAGAIN/EWOULDBLOCK issue
N
E
I
C
S ■ DRT did'n clean up windows opened by
S
A
T javascript correctly.
I
S
R
E
V ■ Fix landed, which reveled that there are more
I
N
U tests involved in DRT's notifyDone() problem.
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 7 D
g
E
n
i
G
r
e Other problems with DRT
E
e
Z
n
i
S
g
n
F
E
O
e
r Phonon related crashes
a
Y
w
T
t
f
I
S
I
o
S
S
S
■ Two tests are involved and are skipped
f
R
N
o
E
E
t
I
V
n
D
I e because they cause the “crashes”
E
m
N
t
G
r
U
E a » fast/runin/nonblock-runin.html
Z
p
e
S
D » http/tests/security/local-video-source-from-remote.html
M
U
R
A
I
T
N
E
I Similar as before
C
S
S
A ■ No real crash
T
I
S
R
E ■ No stderr output
V
I
N
U ■ Only run-webkit-tests reports crash
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 8 D
g
E
n
i
G
r
e Missing DRT features
E
e
Z
n
i
S
g
n
F
E
O
e
r DRT was in a really bad shape
a
Y
w
T
t
f
I
S
I
o
S
S
S
■ All components of DumpRenderTree were in
f
R
N
o
E
E
t
I
V
n
D
I e jsobjects.h and jsobjects.cpp
E
m
N
t
G
r
U
E a » After refactoring these to separate files the
Z
p
e
S development of missing features progresses
D
M
U
R » We would need some restructuring to match the
A
I
T implementations of other ports
N
E
I
C
S
There are still many missing features
S
A
T I ■
S Especially in
R E » LayoutTestController
V
I
N
U » EventSender » TextInputController
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 9 D
g
E
n
i
G
r
e Progress of the QtWebKit port
E
e
Z
n
i
S
g
n
F
E Tests altogether:
O
e
r
a
Y
w
T
t
f
I
S
I o ■11720
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I e Tests we pass:
E
m
N
t
G
r
U
E
a
Z
p
e ■ S More than 6300, slowly increasing
D
M
U
R ■ 5400 tests show layout problems or are new
A
I
T
N – As already mentioned there are approx. 1000
E
I
C tests which only have font related pixel
S
S
A differences, or are new and correct tests.
T
I
S
R
E ■ A tool for making updates a bit easier:
V
I
N
U → http://github.com/bbandix/manual-tester
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 10 D
g
E
n
i
G
r
e manual-tester
E
e
Z
n
i
S
g
n
F
E
O
e
r A not-so-userfriendly Qt 4 application to
a
Y
w
T
t
f
I
S
I
o
S
S S visually check new or pixel-different tests
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D git://github.com/bbandix/manual-tester.git
M
U
R
A
I
T
N $ run-webkit-tests -o ~/testResultsDir
E
I
C
S
$ ./generateTestFile.sh ~/testResultsDir \
S
A
T ~/path_to_WebKit/LayoutTests
I
S
R
E $./manual-tester pathToQtLauncher ~/testResultsDir
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 11 manual-tester
D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 12 D
g
E
n
i
G
r
e The time-zone issue
E
e
Z
n
i
S
g
n
F
E
O
e
r Most of the work on the Mac port is done
a
Y
w
T
t
f
I
S
I
o
S
S
S
in US time-zone
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m N Some developers didn't keep an eye on
t
G
r
U
E
a
Z
p
e
S our bot
D
M
U
R
A
I Keeping the BuildBot green without
T
N
E
I Skipping tests and implementing missing
C
S
S
A features at the same time is hard
T
I
S
R
E The logs of the buildbot are short,
V
I
N
U searching is hard
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 13 D
g
E
n Monitoring the buildbot
i
G
r
e
E
e
Z n MSG=SUCCESS
i
S
g
n SLAVE=Qt_Linux_Release
F
E
O e
r We started to monitor BUILD=3890
a
Y
w
T
t REVISION=51044
f
I
S
I
o
S our buildbot one
S S REVIEWER="Gustavo Noronha
f
R
N
o
E
E
t Silva"
I
V
n
D
I month ago to have
e
E AUTHOR="Kenneth Rohde
m
N
t
G
r
U
E
a Christiansen"
Z p more information
e
S
FILES_CHANGED=11
D
M
U ■ Now we have raw PATCH_LLOC=744
R
A DATE="2009-11-16"
I T data on more than JSCREGRUN=1119
N
E I JSCREGFAIL=0
C 1000 revisions
S JSCREGFIX=0
S
A RUN=6227 T ■ We started to design
I
S OK=6158
R E a web-capable NEW=26
V
I
N FAIL=19
U information system CRASH=24 – BuildBotMiner TIMEOUT=0
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 14 D
g
E
n
i
G
r
e BuildBotMiner
E
e
Z
n
i
S
g
n
F E Script gathers the raw info and log files
O
e
r
a
Y
w
T
t
f
I S
I Python / Django based web application
o
S
S
S
f
R
N
o
E
E
t
I
V n ■ Parsing the information and logfiles
D
I
e
E
m
N
t
G
r
U
E a ■ Loading the information to a database
Z
p
e
S
D
M
U We plan to fully monitor our new bots and
R
A
I
T
N make the information publicly available
E
I
C » Additional information to our benchmark site
S
S
A
T
I
S Sophisticated queries and forms for
R
E
V
I presentin the data need to be done yet
N U What is it good for?
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 15 D
g
E
n
i
G
r
e BuildBotMiner
E
e
Z
n
i
S
g
n
F E Connect the buildbot and the svn trac
O
e
r
a
Y
w
T
t
f
I S
I
o Planned features
S
S
S
f
R
N
o
E
E
t
I
V n ■ Monitoring and pointing out flakey tests
D
I
e
E
m
N
t
G
r
U
E a ■
Z p Pointing out the cause of over-the-night
e
S
D M failing tests
U
R
A
I
T ■ Logging build issues
N
E
I
C ■
S Gathering information about
S
A
T – Development process and progress
I
S
R
E – Layout test debugging progress
V
I
N
U ■ Creating fancy charts and graphs
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 16 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 17 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 18 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 19 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 20 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 21 D
g
E
n
i
G
r
e BuildBotMiner
E
e
Z
n
i
S
g
n
F
E
O
e
r More soon on our blog:
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r
U
E a http://webkit.sed.hu
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 22 D
g
E
n
i
G
r
e
E
e
Z
n
i
S
g
n
F
E
O
e
r
a
Y
w
T
t
f
I
S
I
o
S
S
S
f
R
N
o
E
E
t
I
V
n
D
I
e
E
m
N
t
G
r Thanks for your patience!
U
E
a
Z
p
e
S
D
M
U
R
A
I
T
N
E
I
C
S
S
A
T
I
S
R
E
V
I
N
U
09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 23