UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of

QtWebKit layout tests DRT and Tools

András Bécsi (bbandix) D

g

E

n

i

G

r

e Outline

E

e

Z

n

i

S

g

n 

F Update to Qt 4.6 on the BuildBot

E

O

e

r

a

Y

w ■

T

t Font metric issues

f

I

S

I

o

S

S

S

f

R

N

o

E

E ■ DumpRenderTree issues

t

I

V

n

D

I

e

E

m

N

t

G

r ■ U Progress of the port measured by passing

E

a

Z

p

e

S layout tests

D

M

U

R

A ■ I manual-tester

T

N

E I  C Monitoring the buildbot

S

S

A

T ■ BuildBot logs

I

S

R » Flakey tests

E

V

I

N » Broken tests

U ■ BuildBotMiner

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 2 D

g

E

n

i

G

r

e Qt 4.6 issues

E

e

Z

n

i

S

g

n

F

E

O

e

 r Issues with fonts

a

Y

w

T

t

f

I

S

I

o

S

S S ■After the update to Qt 4.6 we

f

R

N

o

E

E

t

I

V

n

D

I experienced pixel differences in many

e

E

m

N

t

G

r

U

E a tests (~ 500)

Z

p

e

S

D

M

U ■After the update of the expected files

R

A

I

T however we got feedback that these

N

E

I

C metrics do only work on our BuildBot

S

S

A

T »More than 300 tests have incorrect

I

S

R

E

V layout on developers machines but

I

N U are correct on the bot

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 3 D

g

E Qt 4.6 font metric issues

n

i

G

r

e

E e 

Z n We tested other major distributions, but

i

S

g

n

F

E

O e found only minor differences (5-10 tests)

r

a

Y

w

T

t

f

I

S

I o ■ We tried the default configuration of Qt 4.6,

S

S

S

f

R

N

o

E

E

t I the same we use on the buildbot

V

n

D

I

e

E

m

N

t

G

r echo yes | ./configure –opensource -no-

U

E

a

Z

p

e

S Default install to /usr/local

D

M

U

R

A  I On my laptop I managed to reproduce

T

N

E I the 306 failing tests

C

S

S – A Suspicious package differences between the

T

I

S buildbot and my machine:

R

E

V

I » fontconfig 2.6.0 → 2.8.0

N

U » libfreetype 2.3.7 → 2.3.11 » libxext 1.0.4 → 1.1.1 09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 4 D

g

E

n

i

G

r

e Qt 4.6

E

e

Z

n

i

S

g

n

F

E

O

e

r Why these differences?

a

Y

w

T

t

f

I

S

I

o

S

S

S

■ Was something changed in the 4.6 release

f

R

N

o

E

E

t

I

V

n

D

I e that could cause these environment-

E

m

N

t

G

r

U

E

a

Z p dependent problems?

e

S

D M ~1000 now skipped tests which seem to have

U

R

A

I only metric differences could be enabled if this

T

N

E issue is solved

I

C

S

S  A How shold we fix this

T

I

S

R ■ E Change the packages to older versions?

V

I

N

U ■ Find the cause in Qt and fix that?

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 5 D

g

E

n

i

G

r

e Proposal for new tests

E

e

Z

n

i

S

g

n

F  E Using render-tree dumps only if really

O

e

r

a

Y

w

T t needed

f

I

S

I

o

S

S

S

f

R

N

o – E.g. if font metrics are important for some reason

E

E

t

I

V

n

D

I

e

E in the test

m

N

t

G

r

U

E

a

Z

p

e –

S In this case font rendering differences between

D

M

U platforms may need different results

R

A

I T  N Using getComputedStyle() with

E

I

C

S dumpAsText() if possible to validate the

S

A

T

I

S test

R

E

V I 

N Lobbying to change current tests to U follow this proposal

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 6 D

g

E

n

i

G

r

e DRT crashes after change to Qt 4.6

E

e

Z

n

i

S

g

n

F

E

O

e

r Some tests seemed to corrupt follow-ups

a

Y

w

T

t

f

I

S

I

o

S

S

S

■ run-webkit-tests reported DRT crash

f

R

N

o

E

E

t

I

V

n

D

I

e

E ■ No stderr output

m

N

t

G

r

U

E

a

Z

p

e

S

■ Further investigation showed no crashes

D

M

U

R » Instead there was a non-blocking I/O

A

I

T EAGAIN/EWOULDBLOCK issue

N

E

I

C

S ■ DRT did'n clean up windows opened by

S

A

T javascript correctly.

I

S

R

E

V ■ Fix landed, which reveled that there are more

I

N

U tests involved in DRT's notifyDone() problem.

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 7 D

g

E

n

i

G

r

e Other problems with DRT

E

e

Z

n

i

S

g

n

F

E

O

e

r Phonon related crashes

a

Y

w

T

t

f

I

S

I

o

S

S

S

■ Two tests are involved and are skipped

f

R

N

o

E

E

t

I

V

n

D

I e because they cause the “crashes”

E

m

N

t

G

r

U

E a » fast/runin/nonblock-runin.html

Z

p

e

S

D » http/tests/security/local-video-source-from-remote.html

M

U

R

A

I

T

N

E

I Similar as before

C

S

S

A ■ No real crash

T

I

S

R

E ■ No stderr output

V

I

N

U ■ Only run-webkit-tests reports crash

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 8 D

g

E

n

i

G

r

e Missing DRT features

E

e

Z

n

i

S

g

n

F

E

O

e

r DRT was in a really bad shape

a

Y

w

T

t

f

I

S

I

o

S

S

S

■ All components of DumpRenderTree were in

f

R

N

o

E

E

t

I

V

n

D

I e jsobjects.h and jsobjects.cpp

E

m

N

t

G

r

U

E a » After refactoring these to separate files the

Z

p

e

S development of missing features progresses

D

M

U

R » We would need some restructuring to match the

A

I

T implementations of other ports

N

E

I

C

S 

There are still many missing features

S

A

T I ■

S Especially in

R E » LayoutTestController

V

I

N

U » EventSender » TextInputController

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 9 D

g

E

n

i

G

r

e Progress of the QtWebKit port

E

e

Z

n

i

S

g

n

F

E Tests altogether:

O

e

r

a

Y

w

T

t

f

I

S

I o ■11720

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I e Tests we pass:

E

m

N

t

G

r

U

E

a

Z

p

e ■ S More than 6300, slowly increasing

D

M

U

R ■ 5400 tests show layout problems or are new

A

I

T

N – As already mentioned there are approx. 1000

E

I

C tests which only have font related pixel

S

S

A differences, or are new and correct tests.

T

I

S

R

E ■ A tool for making updates a bit easier:

V

I

N

U → http://github.com/bbandix/manual-tester

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 10 D

g

E

n

i

G

r

e manual-tester

E

e

Z

n

i

S

g

n

F

E

O

e

r A not-so-userfriendly Qt 4 application to

a

Y

w

T

t

f

I

S

I

o

S

S S visually check new or pixel-different tests

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D ://.com/bbandix/manual-tester.git

M

U

R

A

I

T

N $ run-webkit-tests -o ~/testResultsDir

E

I

C

S

$ ./generateTestFile.sh ~/testResultsDir \

S

A

T ~/path_to_WebKit/LayoutTests

I

S

R

E $./manual-tester pathToQtLauncher ~/testResultsDir

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 11 manual-tester

D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 12 D

g

E

n

i

G

r

e The time-zone issue

E

e

Z

n

i

S

g

n

F

E

O

e

r Most of the work on the Mac port is done

a

Y

w

T

t

f

I

S

I

o

S

S

S

in US time-zone

f

R

N

o

E

E

t

I

V

n

D

I

e

E 

m N Some developers didn't keep an eye on

t

G

r

U

E

a

Z

p

e

S our bot

D

M

U

R

A 

I Keeping the BuildBot green without

T

N

E

I Skipping tests and implementing missing

C

S

S

A features at the same time is hard

T

I

S

R

E The logs of the buildbot are short,

V

I

N

U searching is hard

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 13 D

g

E

n Monitoring the buildbot

i

G

r

e

E

e

Z n MSG=SUCCESS

i

S

g

n SLAVE=Qt_Linux_Release

F

E

O e 

r We started to monitor BUILD=3890

a

Y

w

T

t REVISION=51044

f

I

S

I

o

S our buildbot one

S S REVIEWER="Gustavo Noronha

f

R

N

o

E

E

t Silva"

I

V

n

D

I month ago to have

e

E AUTHOR="Kenneth Rohde

m

N

t

G

r

U

E

a Christiansen"

Z p more information

e

S

FILES_CHANGED=11

D

M

U ■ Now we have raw PATCH_LLOC=744

R

A DATE="2009-11-16"

I T data on more than JSCREGRUN=1119

N

E I JSCREGFAIL=0

C 1000 revisions

S JSCREGFIX=0

S

A RUN=6227 T ■ We started to design

I

S OK=6158

R E a web-capable NEW=26

V

I

N FAIL=19

U information system CRASH=24 – BuildBotMiner TIMEOUT=0

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 14 D

g

E

n

i

G

r

e BuildBotMiner

E

e

Z

n

i

S

g

n

F  E Script gathers the raw info and log files

O

e

r

a

Y

w

T

t

f

I S 

I Python / Django based web application

o

S

S

S

f

R

N

o

E

E

t

I

V n ■ Parsing the information and logfiles

D

I

e

E

m

N

t

G

r

U

E a ■ Loading the information to a database

Z

p

e

S

D

M

U We plan to fully monitor our new bots and

R

A

I

T

N make the information publicly available

E

I

C » Additional information to our benchmark site

S

S

A

T

I 

S Sophisticated queries and forms for

R

E

V

I presentin the data need to be done yet

N U What is it good for?

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 15 D

g

E

n

i

G

r

e BuildBotMiner

E

e

Z

n

i

S

g

n

F  E Connect the buildbot and the svn trac

O

e

r

a

Y

w

T

t

f

I S 

I

o Planned features

S

S

S

f

R

N

o

E

E

t

I

V n ■ Monitoring and pointing out flakey tests

D

I

e

E

m

N

t

G

r

U

E a ■

Z p Pointing out the cause of over-the-night

e

S

D M failing tests

U

R

A

I

T ■ Logging build issues

N

E

I

C ■

S Gathering information about

S

A

T – Development process and progress

I

S

R

E – Layout test debugging progress

V

I

N

U ■ Creating fancy charts and graphs

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 16 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 17 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 18 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 19 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 20 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 21 D

g

E

n

i

G

r

e BuildBotMiner

E

e

Z

n

i

S

g

n

F

E

O

e

r More soon on our blog:

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r

U

E a http://webkit.sed.hu

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 22 D

g

E

n

i

G

r

e

E

e

Z

n

i

S

g

n

F

E

O

e

r

a

Y

w

T

t

f

I

S

I

o

S

S

S

f

R

N

o

E

E

t

I

V

n

D

I

e

E

m

N

t

G

r Thanks for your patience!

U

E

a

Z

p

e

S

D

M

U

R

A

I

T

N

E

I

C

S

S

A

T

I

S

R

E

V

I

N

U

09/12/09 András Bécsi (bbandix) @ Nokia WebKit Code Camp 23