Multi-project multi-product OpenJDK builder

● Name matters ● Projects matter ● Storage matters

● Not selling product, sharing reasons

Multi-project multi-product OpenJDK builder

● we have infrastructure ● ready to open ● ready to deploy ● not as competitor, ● as complement.

● Now stalled, waiting for this meeting ● in its best, it can be joined

Name matters

● build #xyz is good...but... but not here

● we took Fedoras' Name-Version-Release (.Architecture) (NVR/NVRA) – n-v-r.a ● dashes from behind, still surprisingly... – evolved, proven – nicely readable – works just fine – date is very bad idea

Name matters

Name matters

Name matters

● N – java-X- ● quite fedora bound ● probably not best for adopt openjdk ● V – tag ● the only thing going through all projects merging from main forest

● R – repo (stripped from common parts) – practically anything ● number of changesets since last tag ● 0 is release candidate – keywords ● static, slowdebug, fastdebug, zero, openj9 ● A – src or real platform (i686, x86_64, aarch64, ppc64le...)

Why is naming so important and so hard?

● OpenJDKs is not only (now) jdk9 and/or jdk8 – the projects (EA of EA/tech preview) have even same importance as stable releases

● jigsaw

● shenandoah

● ports – all those were/are as much awaited as main jdk – when project is built, it nothing more then jdk of version it is based on with additional features ● multiple projects(repos) built into single products(jdks of specific version) ● new release system might change this a bit

● multiple repos to provide single product and we (like we right here) *care* about *all* forests (?)

● mayor jdk must be clear from name

project to product mapping

● product == java compliant => – java-9-openjdk, java-1.8.0-openjdk, java-1.7.0-openjdk

● project == repo => – icedtea7-forest, icedtea7-2.6forest, jdk7u => java-1.7.0-oenjdk – jdk8u, jdk8u-aarch64, jdk8u-aarch64-shenandoah, jdk8u-shenandoah, jdk8u-jigsaw => java- 1.8.0-openjdk – jdk9, jdk9u, jdk9u-shenandoah, jdk9u-jigsaw => java-9-openjdk ... – iced tea really should be considered in – same for -web – windows repos? – forest approval process

● name must clearly show the identification of project it is from

pull

● forests will be with us for some time – To add forest must be automated ● Any user can add own forest – The whole setup will probably need management anyway ● Keep single developer usecase in mind

● pull on forest is far from being straight-forward

● multi-project multi-product environment is not making it easier

● hg incoming on all trees is decision point for pull in case of pooling – Git alternative?

● official builder can be (should be) triggered by hook

pull

● we currently semi-pool, so – granularity is often more then one changeset – we use hooks in case of internal builds ● one patch can cover more trees ● jenkin's url build-now timeout serves well here ● this fact (more changesets in single patch) enforces the observer logic anyway

● hg incoming+hg pull+hg update later – all three provides very important logs ● Easy to generate “the patch” – all mentioned logic have easy maintenance ● shared script about 100lines over all of this – ignoring windows git repos :(

src snapshot

● pull can bring both release candidate and testing version(s) – more variants are build from same revision ● More src snapshots o changeset: 2000:c6c15bd2fa20 | tag: tip ● ☠☠☠☠ merge ☠☠☠☠ | o changeset: 1999:af980289e0df – |\ tag: jdk8u162-b02 can bring in even more | | | o changeset: 1998:01c1e35a6ade candidates -> | | | o changeset: 1997:7a25d12cd94f | | tag: jdk8u152-b16 – Should bring in also releases | o changeset: 1996:d680e12deacb | |\ tag: jdk8u152-b15 ● If not yet processed in past .... - tip is u162-b02+X changeset, however - u152-b16+1 changesets is there too (01c1e35a6ade) - our top one is 4 src snapshots from one pull - not most easy task, especially in forests - may not be applicable on low-tagging rate projects src snapshot

● we know – what to pull – when to pull – what jdk is an result – we already know some artifacts ● hg incoming, pull and update logs – other important artifacts are ● logs of individual trees ● heads of trees ● combined patch (difference from old heads) [webrev?] – because more changesets in patch – enough info to drop .hg (120mb->60mb) ● logs &comp should be both in snapshot tarball and jenkions artifacts

● we know all targets of snapshot (fastdebug, slowdebug, zero, openj9...) – multiplying of src snapshots

● we guess already name, but what to do with it?

src snapshot

● N – java-X-openjdk ● quite fedora bound – probably not best for adopt openjdk

● V – tag ● the only thing going through all projects merging from main forest

● R – repo (stripped from common parts) – practically anything ● number of changesets since last tag ● 0 is release candidate – keywords ● static, slowdebug, fastdebug, zero, openj9

● A – src – src.tarxz ● tarxz is to keep the dots adapted with .rpm, .zip and other

src snapshot

● pooling x hooks x other triggers

● remember – pull is per project – build per product ● ok...project might affect build ● Surprisingly complicated setup file

● Build 165 and up – External scheduler is checking changes, and lunches job only if chaged – unstable build=changed

● 164 and down. Normal pooling – There is row of 50 stable builds – Not much interesting view... src snapshot

● src snapshot (and logs) is jenkins artifact ● src snapshot is arch-independent operation – implementation have nothing to do with following builds – still the src snapshot must be tight with builds ● jenkins is really bad in this ● even worse with multi-project multi-product environment

● upload to well organized shared space – current awesome front-page is subset of web page reading from this storage

src+builds(+logs) shared storage fake-koji

● inspired by fedora koji – Real koji doesn’t have products, projects and variants

● well readable for humans

● well readable for machines – jenkins koji-scm plugin - https://github.com/judovana/jenkins-scm-koji-plugin

● grouping product->project

● grouping project->product

● variants (slowdebug, fastdebug, zero, openj9...)

● listing(s) – with good listing and grouping and jenkins binding, custom search not necessary

Fake koji human readable part (1)

←its not nice, it really needs front-page :) ←listing products ←Listing product’s projects ← Latest build of project first ● With variants

← Product→Project→archs+variants→ bits

– But missing: ● Stable versions

Fake koji human readable part (2)

– project’s Full listing with variants → ● Even with variants focused listing ● And failed (partial) builds – Still missing: ● Stable versions

Fake koji human readable part transiting with machine readable part

● Http directory listing – All containing, reducible full text search – Grouping builds, srcs and logs in well organized form

Fake koji machine readable part

● Queries – Curl api is great, but might soon lack readability ● Json replies are not well readable from bash – Last N builds of product – Last N builds of project * – Last N stable builds of product – Last N stable builds of project * – Give me failed build(s) – Giive me passed builds on some arch only – product’s projects * – Architectures of build – Variants of build ** – Logs of build – Files of builds – Status ● Finished (since all primary archs pass) ● Failed (since first primary arch fail) ● Running (since src upload to last secondary arch) – ….

● We have xmlrpc to be compatible with original koji

* not done, not missing for now, still nice to have ** workarounded in current implementation

Build

● Once we have – Source snapshot ● With all info about the change – Proper name ● Major jdk and project-specific stuff – Variants and architectures description – Place where to get it all – Place where to upload it

– The build itself is easy ● Reporting is not

Build

● How to build matters a LOT – Forest contributors must be able to contribute to the build part ● If they don’t care, project get disabled ● Have `Is release “button”` – Build part must be simple and unified

● Build differs only between major JDK releases – But small changes might change form project to project – Variants affects build in minor way, but the result is absolutely different

● Must be reproducible – Final configure_call+make_call+whatever_else_call is generated and stored as build artifact as important a logs and bits – VM image should be accessible

● If possible, build with full bootstrap

● Src.tarxz → ARCH.tarxz – Build and build call is uploaded to its place in fake-koji

● Surprisingly easy to achieve in our proof of concept environment

Build

● Logs of build matters a LOT – Forest contributors must be able to receive failed build reports ● Not only contributors ● subscribing for projects?

● Logs are big – Separated x joined stdout/err can help – Simple grepping of fail, error or warning with reasonable -A -B – Html coloring of above, is helping extraordinary well – Access to full build log x reproducible build

● Repeated failures – Once the logs are truncated to shard of its size while kept useful – It is easy to make a pattern matching pinpointing know failures ● Beta now at our side

● All the logs are uploaded to its place in fake koji (and kept as jenkins artifact as well)

● Still surprisingly easy to achieve in our proof of concept environment

Primary and secondary architectures

● Build is running since src upload ● Build is failed, when one of primary architectures fails ● Build is finished, when all primary architectures are finished ● Build is unstable when some secondary architecture (or variant?) failed – The second is hard to achieve – In current schema, variant is simply different build ● Build for itself only (zero, newer osses)

● Secondary and primary architectures set is based on project, not product!

e.g.: ● jdk7u – primary arches intel 32+64, win64. Secondary arches: ppc64, ppc32, s390, s390x ● IcedTea7 – aarch64 and ppc64le added to secondary arches, win64 removed ● Jdk8u-aarch64 project – aarch64 moved to primary arches, win64 added to primary arches ● Jdk9 – ppc64le added to primary arches, aarch32 added to secondary arches

● Secondary architectures are presented more like variant builds, then regular builds

Testing the build

● Must be separated form build (and of course src snapshoting) – Must be easily connectable – NVRA is the connecting point

● JTREGs are must – Results must be tangled to at least to release candidates – Project maintainers should have all builds’ results – Jtregs are obtained from src.tarxz ● NVRA instead of build number ● Links to binaries ● Links to sources – What variants? ● Fastdebug have sense – TCK ● Would be awesome to have, I would drop 60% of my workload :) – https://github.com/judovana/jenkins-report-jck – Many other free and very useful testsuites ● Lucene ● jmh ● Jcstress ● ... – Testing everything everywhere ● We do that now – HW limitations – tck+jtreg jenkins plugin (next screens:) :

Most powerful is watch dog over regression

Jenkins koji-scm plugin:

… todo ... ● keep front page ● Is absolutely fabulous ● “drop” current jenkins jobs ● drop(?) current build scripts ● over complexes, not transparent, and well.. not doing enough ● allow multi repo environment ● setup "koji like" storage ● setup NVRA viewing jenkins ● Adapt major-tag-changes+repo naming

● What is variant?

● get rid of tags

● Get rid of keywords &comp Repo1 Repo2 RepoN Overview: Pull? Pull? Pull?

How NVRA went? pull+update

Src snapshots(!), logs... upload Build primary arch1 Testsuite 1 Testsuite 1 arch1 archN src uild B

, nly) way cts o Storage one rtifa ( ns a Testsuiteenki N Testsuite N as j Build, logs… (for builds ) ains rem arch1 archN ults Res src

Build primary archN

src src Build Logs... answers, queries binaries

Build Build

secondary arch1 primary archN Pull, update, build and testing (Green, blue and yellow) Are schedule dby jenkins Pull can be pooling, semi pooling or trigered

Magneta nd white queries are external tools Queying storage or jenkins Scheduler

● This slide was added after London meeting, as reaction to nightly builds, and as my comments are missing :)

● We have two fair schedules.

● This scheduler is jenkins job, – First scheduler is highPushRate-lowInterest – run 3x per week ● java-1.8.0-openjdk java-1.8.0-openjdk-dev java-9-openjdk java-9-openjdk-dev ● So repos RH have low interest XOR have high push rate – Second is lowPushRate-highInterest – run 6x per week ● java-1.7.0-openjdk java-1.7.0-openjdk-forest java-1.8.0-openjdk-aarch64 java-1.8.0-openjdk-aarch64- shenandoah java-9-openjdk-shenandoah java-1.7.0-openjdk-forest-26 ● So repos where RH have interest XOR have low frequency of push

● The scheduler iterates over repos, and if there is hg incomming, it starts this forest’s pull job (another jenkins job) . Next time time is starts on next repo. – So we have snapshots and tests on all “interesting” repos at least once per week – And of all “not interesting” repos aprox once per month