Using Redmine and Hudson for Wendelstein 7-X CODAC Software Development

Jahre Using redmine and hudson for Wendelstein 7-X CODAC software development

Christine Hennig

Agenda

• Wendelstein 7-X (W7-X) and it’s CODAC group • Software development managed by Redmine • Continuous integration with Hudson • Migration stories • AOB

27/04/2010 [email protected] / [email protected] Wendelstein 7-X assembly http://www.ipp.mpg.de/ippcms/eng/for/publikationen/w7xletters/download/no_4_03_2010.pdf

27/04/2010 [email protected] / [email protected]

Wendelstein 7-X CODAC group

• Work started in ~1999 as 2 independent groups • XDV Experimental Data Acquisition Project (java software) joint project Garching/Greifswald • W7-X Control Group (C++ for VxWorks) part of W7-X project from beginning • 2009 fusion into W7-X CODAC group • IDA Integrated Data Analysis group (empty) added • Already 2007 AUG CODAC group fusion • Stellarator and Tokamak CODAC still separated

27/04/2010 [email protected] / [email protected] Location of people, hardware and software

DAQ- W7-X Redmine HW CODAC Hudson Data (prod) Base SVN XDV (prod)

Weißwurst-Äquator [humorous term describing XDV the very different culture of Data the southern part of Bavaria AUG Base as opposed to its other parts] CODAC (test) DAQ- HW (test)

27/04/2010 [email protected] / [email protected]

W7-X CODAC

• continuous (30 minutes) Control&DAQ • ~ 15 people • ~ 10 years of development • mixed age (but starting getting old together) • some 100.000 lines of code • W7-X expected to start operation in May 2014 • CODAC testbed – 40 year old small stellarator/tokamak hybrid machine (WEGA -Wendelstein Experiment in Greifswald for Apprenticeship) – for production database, hardware, software

27/04/2010 [email protected] / [email protected] Redmine + Hudson

• tools dedicated for software development • software developers playground • no replacement of official project management tools • group-only use • daily working horse • agile • results must go into the official project management tools (bidirectional)

27/04/2010 [email protected] / [email protected]

Redmine

Heard about trac? • Redmine is a flexible project management web app • Ruby on Rails based, cross-platform and cross-database • open source (GPL) • main features – issue tracking – News, documents & files management – Feeds & email notifications – wiki – forums – Time tracking – SCM integration (SVN, CVS, Git, Mercurial, Bazaar and Darcs)

27/04/2010 [email protected] / [email protected] Redmine: main page

Selfdefined nagivation

27/04/2010 [email protected] / [email protected]

Redmine: per project overview page

overview tickets files options

activity

27/04/2010 [email protected] / [email protected] Redmine: Support of multiple projects

27/04/2010 [email protected] / [email protected]

Redmine: Ticket system

27/04/2010 [email protected] / [email protected] Redmine: A bug in a project

27/04/2010 [email protected] / [email protected]

Redmine: activity page

27/04/2010 [email protected] / [email protected] Redmine: source code repository

SCM

27/04/2010 [email protected] / [email protected]

Redmine: Subversion integration

27/04/2010 [email protected] / [email protected] Redmine: Subversion – whom to blame?

27/04/2010 [email protected] / [email protected]

Redmine: GANTT

27/04/2010 [email protected] / [email protected] Comparison with JET tools

27/04/2010 [email protected] / [email protected]

Hudson

Heard about CruiseControl? • Hudson is an open source tool to perform Continuous Integration. • Continuous integration is a process in which all development work is integrated at a predefined time or event and the resulting work is automatically tested and build. • The idea is that development errors are identified very early in the process. • Fully web based configuration

27/04/2010 [email protected] / [email protected] Hudson: project list

weather forecast

27/04/2010 [email protected] / [email protected]

Hudson: a fortran code

27/04/2010 [email protected] / [email protected] Hudson: test and other trends

27/04/2010 [email protected] / [email protected]

Hudson: red is bad

Last failed build

27/04/2010 [email protected] / [email protected] Hudson: checkstype, findbugs, PMD

Someone improved the code!

27/04/2010 [email protected] / [email protected]

Hudson: junit test results

Nothing red is quite good (if there are sufficient tests)

27/04/2010 [email protected] / [email protected] Hudson: checkstype, findbugs, PMD (cont.)

Automatic start due to configured code dependencies

27/04/2010 [email protected] / [email protected]

Hudson: cobertura

Are there sufficient tests?

27/04/2010 [email protected] / [email protected] Hudson: cobertura (cont.)

Are there sufficient tests? red is bad

27/04/2010 [email protected] / [email protected]

Comparison with JET tools

27/04/2010 [email protected] / [email protected] Migration

1999 2009 time Trac (CO) migration redmine (CODA) connection Trac (DA) SVN (indiv.) SVN (physics) Trac SVN (CODA) CVS CVS (DA) (DA) (DA) SVN (CO) Hudson

27/04/2010 [email protected] / [email protected]

consolidation

redmine (CODA) connection SVN (indiv.) SVN (physics) SVN (DA) SVN (CO) Hudson

27/04/2010 [email protected] / [email protected] Migration CVS to SVN

• urgent need – Java refactoring (file rename) impossible (loss of history) – incredibly slow – using cvs was error prone (once or twice discarded repositories and started from scratch) – poor branching, merging etc. • used a migration tool • CVS history kept in SVN • Putted existing stuff in legacy folder • restructuring to allow new structure • based on best efford : – every time code is touched for the first time Ł move to the new directory structure (some people avoided doing this) • nobody ever had a look back at CVS

27/04/2010 [email protected] / [email protected]

trac ticket structure was

Trac (CO) #413 #278 ... #13

SVN (indiv.) SVN (physics) SVN (DA) ... SVN #198 #278 #513 (CO) Trac (DA)

27/04/2010 [email protected] / [email protected] Migration of the trac wiki to redmine

• First created a common trac project for the merged team – hoping trac would support multiple projects soon – nobody wanted to move all single pages manually – soon gave up all 3 trac‘s and migrated into a single multiproject redmine system • 3 trac projects into ~20 redmine projects – projects for main java projects (code centric) – additional projects for meetings, user docu – actual number of projects is larger than the number of people • 2 (3) trac wiki‘s into ~10 new redmine wikis into 1 final redmine wiki – ~ 2* 150 wiki pages moved by tool into a legacy wiki – CamelCase links were converted to ordinary trac links – corrected when touched i.e. moved to proper position – could not find pages any more (which project?) – finally restructuring back into a single wiki – only overview project has a wiki attached

27/04/2010 [email protected] / [email protected]

Migration of the trac tickets to redmine

• 2 ticket systems (including numbers, interconnections and references to code fixes in SVN) – 2 * ~500 tickets – many references between tickets • within project • between the projects (complete URL) – had to define mapping of level and category – 1st project kept numbers – 2nd project got new numbers • consistency of internal references kept • SVN references lost – spanning references between 2 systems lost

27/04/2010 [email protected] / [email protected] Outlook for planned improvements

• add software deployment • more and better tests • fixing all the bugs • constant improvement of code and docu • create user docu from wiki page • Improve connection to – official project mangement – intranet

27/04/2010 [email protected] / [email protected]

What to include/exclude?

• Fluid information Ł agile tools – changes rapidly – less reliable – Ideas – work in progress • Solid information Ł EDMS – changes slowly or never – changes have huge impact – reliable – accurate

27/04/2010 [email protected] / [email protected] What to include/exclude? (cont. 1)

• Product Requirements Ł EDMS • Product Concept Ł EDMS – Context – First-level functional and structural descriptions • Detailed Designs – Functional and Structural descriptions for each code/data/object Assembly Ł redmine Ł EDMS – Source code / data Ł SCM • Installation Ł redmine • Commissioning Ł EDMS • Procurements Ł SAP • Planning & Progress Reporting Ł MSProject (possibly each with its own issue tracking and discussion forum) Ł redmine

27/04/2010 [email protected] / [email protected]

What to include/exclude? (cont. 2)

• hardware configuration history Ł Database • software configuration history Ł SCM • services configuration history Ł SCM • operational configuration history Ł Database • consumables history (sources, disposals) Ł SAP • how-to's Ł redmine • fault / issue tracking Ł redmine • summary reports (e.g. availability analysis, failure mode analysis) Ł EDMS (all documents, source code/data, etc. should held in version-controlled-repository) Ł SCM

27/04/2010 [email protected] / [email protected] 5 steps to a successfull knowledge wiki

• Incorporate employees • Do not put an empty wiki online • Nominate knowledge editors • Stress the benefit • Reward the knowledge work

source: http://www.computerzeitung.de/articles/fuenf_schritte_zum_erfolgreichen_unternehmens-wiki:/2009009/31845540_ha_CZ.html

27/04/2010 [email protected] / [email protected]

Learning 2.0

• blogs and wikis eliminate hierarchy • encouragement people to imporve their bosses contribution • if people know each other there is less fear of expose • sensibilization, motivation, get use to use wikis • exchange of experience about best practice • convince technology-affine people is like preach to the converted • decentralize knowledge work • don‘t make any targets how many entries some has to write • high individual effort • treat content carefully • don‘t create a data grave • search needs success within 1-2 minutes • sub-wikis for structuring • no universal remedy source: http://www.computerzeitung.de/articles/hierarchien_bremsen_das_lernen_20:/2009009/31840980_ha_CZ.html

27/04/2010 [email protected] / [email protected] Take-home message well-known software engineering problems

appropriate tools Learning curve is deep / barrier to entry less routine work

27/04/2010 [email protected] / [email protected]

Take-home message (revised)

stand on the shoulder of giants

27/04/2010 [email protected] / [email protected] Links

• Redmine: – http://andrewpeters.net/2008/07/17/redmine-trac-without-the- suck/ – http://www.redmine.org/ – http://www.soebes.de/files/RedmineCLT2009.pdf (Installation) • Hudson: – http://www.vogella.de/articles/ContinuousIntegration/article.html – http://hudson-ci.org/ • Trac: – http://trac.edgewall.org/ • CI – „Continous Integration for One“ by Jazz Higgins in {cvu} Vol. 22 Issue 1 Mar 2010 – http://martinfowler.com/articles/continuousIntegration.html

27/04/2010 [email protected] / [email protected]