Multi-project multi-product OpenJDK builder ● Name matters ● Projects matter ● Storage matters ● Not selling product, sharing reasons Multi-project multi-product OpenJDK builder ● we have infrastructure ● ready to open ● ready to deploy ● not as competitor, ● as complement. ● Now stalled, waiting for this meeting ● in its best, it can be joined Name matters ● build #xyz is good...but... but not here ● we took Fedoras' Name-Version-Release (.Architecture) (NVR/NVRA) – n-v-r.a ● dashes from behind, still surprisingly... – evolved, proven – nicely readable – works just fine – date is very bad idea Name matters Name matters Name matters ● N – java-X-openjdk ● quite fedora bound ● probably not best for adopt openjdk ● V – tag ● the only thing going through all projects merging from main forest ● R – repo (stripped from common parts) – practically anything ● number of changesets since last tag ● 0 is release candidate – keywords ● static, slowdebug, fastdebug, zero, openj9 ● A – src or real platform (i686, x86_64, aarch64, ppc64le...) Why is naming so important and so hard? ● OpenJDKs is not only (now) jdk9 and/or jdk8 – the projects (EA of EA/tech preview) have even same importance as stable releases ● jigsaw ● shenandoah ● ports – all those were/are as much awaited as main jdk – when project is built, it nothing more then jdk of version it is based on with additional features ● multiple projects(repos) built into single products(jdks of specific version) ● new release system might change this a bit ● multiple repos to provide single product and we (like we right here) *care* about *all* forests (?) ● mayor jdk must be clear from name project to product mapping ● product == java compliant => – java-9-openjdk, java-1.8.0-openjdk, java-1.7.0-openjdk ● project == repo => – icedtea7-forest, icedtea7-2.6forest, jdk7u => java-1.7.0-oenjdk – jdk8u, jdk8u-aarch64, jdk8u-aarch64-shenandoah, jdk8u-shenandoah, jdk8u-jigsaw => java- 1.8.0-openjdk – jdk9, jdk9u, jdk9u-shenandoah, jdk9u-jigsaw => java-9-openjdk ... – iced tea really should be considered in – same for icedtea-web – windows github repos? – forest approval process ● name must clearly show the identification of project it is from pull ● forests will be with us for some time – To add forest must be automated ● Any user can add own forest – The whole setup will probably need management anyway ● Keep single developer usecase in mind ● pull on forest is far from being straight-forward ● multi-project multi-product environment is not making it easier ● hg incoming on all trees is decision point for pull in case of pooling – Git alternative? ● official builder can be (should be) triggered by hook pull ● we currently semi-pool, so – granularity is often more then one changeset – we use hooks in case of internal builds ● one patch can cover more trees ● jenkin's url build-now timeout serves well here ● this fact (more changesets in single patch) enforces the observer logic anyway ● hg incoming+hg pull+hg update later – all three provides very important logs ● Easy to generate “the patch” – all mentioned logic have easy maintenance ● shared script about 100lines over all of this – ignoring windows git repos :( src snapshot ● pull can bring both release candidate and testing version(s) – more variants are build from same revision ● More src snapshots o changeset: 2000:c6c15bd2fa20 | tag: tip ● ☠☠☠☠ merge ☠☠☠☠ | o changeset: 1999:af980289e0df – |\ tag: jdk8u162-b02 can bring in even more | | | o changeset: 1998:01c1e35a6ade candidates -> | | | o changeset: 1997:7a25d12cd94f | | tag: jdk8u152-b16 – Should bring in also releases | o changeset: 1996:d680e12deacb | |\ tag: jdk8u152-b15 ● If not yet processed in past .... - tip is u162-b02+X changeset, however - u152-b16+1 changesets is there too (01c1e35a6ade) - our top one is 4 src snapshots from one pull - not most easy task, especially in forests - may not be applicable on low-tagging rate projects src snapshot ● we know – what to pull – when to pull – what jdk is an result – we already know some artifacts ● hg incoming, pull and update logs – other important artifacts are ● logs of individual trees ● heads of trees ● combined patch (difference from old heads) [webrev?] – because more changesets in patch – enough info to drop .hg (120mb->60mb) ● logs &comp should be both in snapshot tarball and jenkions artifacts ● we know all targets of snapshot (fastdebug, slowdebug, zero, openj9...) – multiplying of src snapshots ● we guess already name, but what to do with it? src snapshot ● N – java-X-openjdk ● quite fedora bound – probably not best for adopt openjdk ● V – tag ● the only thing going through all projects merging from main forest ● R – repo (stripped from common parts) – practically anything ● number of changesets since last tag ● 0 is release candidate – keywords ● static, slowdebug, fastdebug, zero, openj9 ● A – src – src.tarxz ● tarxz is to keep the dots adapted with .rpm, .zip and other src snapshot ● pooling x hooks x other triggers ● remember – pull is per project – build per product ● ok...project might affect build ● Surprisingly complicated setup file ● Build 165 and up – External scheduler is checking changes, and lunches job only if chaged – unstable build=changed ● 164 and down. Normal pooling – There is row of 50 stable builds – Not much interesting view... src snapshot ● src snapshot (and logs) is jenkins artifact ● src snapshot is arch-independent operation – implementation have nothing to do with following builds – still the src snapshot must be tight with builds ● jenkins is really bad in this ● even worse with multi-project multi-product environment ● upload to well organized shared space – current awesome front-page is subset of web page reading from this storage src+builds(+logs) shared storage fake-koji ● inspired by fedora koji – Real koji doesn’t have products, projects and variants ● well readable for humans ● well readable for machines – jenkins koji-scm plugin - https://github.com/judovana/jenkins-scm-koji-plugin ● grouping product->project ● grouping project->product ● variants (slowdebug, fastdebug, zero, openj9...) ● listing(s) – with good listing and grouping and jenkins binding, custom search not necessary Fake koji human readable part (1) ←its not nice, it really needs front-page :) ←listing products ←Listing product’s projects ← Latest build of project first ● With variants ← Product→Project→archs+variants→ bits – But missing: ● Stable versions Fake koji human readable part (2) – project’s Full listing with variants → ● Even with variants focused listing ● And failed (partial) builds – Still missing: ● Stable versions Fake koji human readable part transiting with machine readable part ● Http directory listing – All containing, reducible full text search – Grouping builds, srcs and logs in well organized form Fake koji machine readable part ● Queries – Curl api is great, but might soon lack readability ● Json replies are not well readable from bash – Last N builds of product – Last N builds of project * – Last N stable builds of product – Last N stable builds of project * – Give me failed build(s) – Giive me passed builds on some arch only – product’s projects * – Architectures of build – Variants of build ** – Logs of build – Files of builds – Status ● Finished (since all primary archs pass) ● Failed (since first primary arch fail) ● Running (since src upload to last secondary arch) – …. ● We have xmlrpc to be compatible with original koji * not done, not missing for now, still nice to have ** workarounded in current implementation Build ● Once we have – Source snapshot ● With all info about the change – Proper name ● Major jdk and project-specific stuff – Variants and architectures description – Place where to get it all – Place where to upload it – The build itself is easy ● Reporting is not Build ● How to build matters a LOT – Forest contributors must be able to contribute to the build part ● If they don’t care, project get disabled ● Have `Is release “button”` – Build part must be simple and unified ● Build differs only between major JDK releases – But small changes might change form project to project – Variants affects build in minor way, but the result is absolutely different ● Must be reproducible – Final configure_call+make_call+whatever_else_call is generated and stored as build artifact as important a logs and bits – VM image should be accessible ● If possible, build with full bootstrap ● Src.tarxz → ARCH.tarxz – Build and build call is uploaded to its place in fake-koji ● Surprisingly easy to achieve in our proof of concept environment Build ● Logs of build matters a LOT – Forest contributors must be able to receive failed build reports ● Not only contributors ● subscribing for projects? ● Logs are big – Separated x joined stdout/err can help – Simple grepping of fail, error or warning with reasonable -A -B – Html coloring of above, is helping extraordinary well – Access to full build log x reproducible build ● Repeated failures – Once the logs are truncated to shard of its size while kept useful – It is easy to make a pattern matching pinpointing know failures ● Beta now at our side ● All the logs are uploaded to its place in fake koji (and kept as jenkins artifact as well) ● Still surprisingly easy to achieve in our proof of concept environment Primary and secondary architectures ● Build is running since src upload ● Build is failed, when one of primary architectures fails ● Build is finished, when all primary architectures are finished ● Build is unstable when some secondary architecture