A Silicon Detector Measurements DB

A silicon detector measurements DB

FeasibilityFeasibility studystudy && progressprogress reportreport MatthiasMatthias Bergholz,Bergholz, WolfgangWolfgang FriebelFriebel CERN, Nov 25.2009 Outline of the talk

RequirementsRequirements andand designdesign principlesprinciples LayoutLayout ofof thethe databasedatabase ComponentsComponents ofof thethe frameworkframework TheThe useruser interfaceinterface StatusStatus ofof thethe projectproject NextNext stepssteps

Nov 25 2 Requirements for the DB

FirstFirst measurementsmeasurements comingcoming soonsoon DB needed with at least limited functionality early in 2010 Ambitious schedule for development AccessibilityAccessibility ofof thethe DBDB As simple as possible with few or no restricttions BeBe preparedprepared forfor designdesign changeschanges Due to the nature of an R&D project flexibility is important UserUser interfaceinterface shouldshould bebe easyeasy toto useuse andand powerfulpowerful Upload of data needs to be simple and efficient Complicated queries should be possible Nov 25 3 Design principles

RapidRapid prototypingprototyping essentialessential duedue toto timetime constraintsconstraints Working solutions can be discussed, fast iterations, proof of concepts Alternatively reuse of existing solutions LayeredLayered conceptconcept toto hidehide implementationimplementation detailsdetails fromfrom useruser Allows for design changes without user visible effects Access to the DB possible but discouraged WebWeb applicationapplication insteadinstead ofof GUIGUI No need to install software for using the DB Web access almost everywhere possible (no restrictions)

Nov 25 Established scalable solution with many options (auth, 4 proxy, ...) Design principles (2)

FlexibilityFlexibility inin thethe selectionselection ofof aa databasedatabase Use SQL without advanced or DB specific constructs if possible Migration to a different DB should be easily possible AutomationAutomation ofof taskstasks shouldshould bebe possiblepossible As little language dependency as possible for easy tasks Access to the project internals should be possible (language and learning curve matters) MaintainabilityMaintainability Modularity is important to decrease bug hunting time Nov 25 Automated testing procedures need to ensure quality 5 Design choices

SeveralSeveral solutionssolutions forfor rapidrapid prototypingprototyping forfor thethe webweb For many languages, all inspired by Ruby on Rails (2004) e.g. Django (Python) Implementation using a Model-View-Controller (MVC) design i.e. separation of the data model from the program logic and visualisation CatalystCatalyst (perl)(perl) hashas beenbeen chosenchosen Huge user base, vast number of perl modules worldwide known as comprehensive perl archive network, CPAN Nov 25 Familiarity with the language 6 Many design principles can be easily fulfilled (eg. DB choice) Highly automated generation of web user interfaces Design choices (2)

StartingStarting withwith mysqlmysql asas DBDB backendbackend Also tried: migration to sqlite3 as proof of concept PutPut asas muchmuch infoinfo intointo thethe datadata modelmodel (constraints,(constraints, rules,rules, transformations)transformations) insteadinstead ofof writingwriting programprogram logiclogic UsingUsing thethe TemplateTemplate ToolkitToolkit 22 forfor webweb pagespages layoutlayout e.g. used in mailing list management software “sympa” WillWill needneed toto enhanceenhance webweb interfaceinterface byby AJAXAJAX (usability)(usability) Most promising candidate: prototype library Nov 25 7 Communication with the web server then using JSON AlternativesAlternatives toto HTMLHTML outputoutput possible:possible: e.g.e.g. XMLXML Layout of the database

PrincipalPrincipal tablestables relatedrelated toto taskstasks toto bebe performedperformed withwith thethe DBDB history - contains all actions recorded in the DB meas_action – describes what measurements are available mod_action – actions that modify the detector (e.g. irradiation) ship_action – transfer of detectors between centers sim_action – simulation results for e.g. field maps etc. free_action – everything else person, center – tables related to places, people and accounts person_role, role – additional role based authorisation parameter, meas_value – measured parameters Nov 25 More tables for parameters, measurement methods and 8 related values existing, simplified here Person management

person

name center prename phone id email person_role name center_id person_id country accountname role_id town password

street created

postcode updated

expirydate

role

role Nov 25 9 Structure management

type

name substructure wafer structure id id name id name type_id name wafertype structure_id wafer_id technology status_id

number status

name

Nov 25 10 Action mgmnt, e.g. measurement

parameter

name

actiontype_id

actiontype unit meas_action_value id required id name derivated meas_action_id

parameter_id history meas_action value

id id

date uploaddate

substructure_id datalocation

annotation person_id filename

id actiontype_id errorflag

name annotation_id history_id

Nov 25 11 Database flexibility

AvoidAvoid schemaschema changeschanges ifif possiblepossible UseUse separateseparate tablestables holdingholding listslists ofof thingsthings thatthat maymay needneed toto bebe modifiedmodified later,later, e.g.e.g. parameter – just the name, the units and a pointer to a method in a table instead of fixed parameter columns meas_value –– a value, a pointer to the parameter (name), a pointer to the actual measurement DoDo notnot relyrely onon advancedadvanced SQLSQL functionalityfunctionality Can be compensated by writing perl code Can be compensated by more complicated templates

Nov 25 12 Components of the framework

DatabaseDatabase mysql,mysql, provideprovide aa schemaschema onlyonly ObjectObject relationalrelational mappermapper Tables become classes, columns become methods Additional functionality by enhancing methods WebWeb contentcontent defineddefined usingusing templatestemplates (TT2)(TT2) WebWeb layoutlayout (Colors,(Colors, TextText sizesize etc)etc) byby CSSCSS AdditionalAdditional classesclasses forfor programprogram logiclogic (controller)(controller) AJAXAJAX forfor easiereasier useruser interactioninteraction withwith thethe WebWeb GUIGUI AJAX = “Asynchronous JavaScript and XML” e.g. autocompletion of names, drag and drop etc.

Nov 25 13 The object relational mapper

package CecDB::Schema::Result::Role;

use base 'DBIx::Class';

__PACKAGE__->load_components( "InflateColumn::DateTime", "TimeStamp", "EncodedColumn", "Core", ); __PACKAGE__->table("role"); __PACKAGE__->add_columns( "id", { generated by mapper data_type => "INTEGER", default_value => undef, is_nullable => 0, size => undef, }, "role", { data_type => "varchar", default_value => "NULL", relation between tables is_nullable => 1, size => 15, }, ); __PACKAGE__->set_primary_key("id");

# Created by DBIx::Class::Schema::Loader v0.04006 @ 2009-09-15 12:01:31

# You can replace this text with custom content, and it will be preserved on regeneration use overload '""' => sub {$_[0]->id}, fallback => 1; __PACKAGE__->utf8_columns(qw/id role/); __PACKAGE__->has_many(map_user_roles => 'CecDB::Schema::Result::PersonRole', 'role_id');

Nov 25 14 Enhancing the data model

package CecDB::Schema::Result::Person; ... "password", { data_type => "VARCHAR", default_value => undef, size => 63, }, ... # overwrite some of the definitions above __PACKAGE__->add_columns( "password", instead of storing plain { data_type => "VARCHAR", passwords only an encrypted default_value => undef, version can enter the DB size => 31, encode_column => 1, encode_class => 'Digest', additional methods, e.g. encode_args => {salt_length => 10}, checks for min length and encode_check_method => 'check_password', content may be added }, ... Nov 25 15 Enhancing the data model (2)

NewNew pseudopseudo columnscolumns cancan bebe createdcreated thatthat dodo notnot existexist inin thethe DBDB Note: columns become methods Additional methods can be defined InIn templatestemplates methodsmethods areare calledcalled Makes no difference whether method corresponds to a real table column or is a newly added method The “has_many” method does generate such new methods dynamically

Nov 25 16 Flexibility by using roles

PersonsPersons getget assignedassigned aa rolerole Can be used to restrict access rights Can be used to generate different views to the data example: edit functionality for tables may be added OtherOther informationinformation fromfrom thethe DBDB andand thethe GUIGUI useableuseable Location dependent display of information Location is obtained from REMOTE_ADDR, i.e. Calling host example: phone number 79454 displayed as 79454 if host from cern.ch 022/7679454 if host from ch +41-022/7679454 if host from outside ch Nov 25 ThereforeTherefore loginlogin withwith accountaccount namename usefuluseful 17 A sample template

[% # Provide a title -%] [% META title = 'Detector List' -%]

[% # Display each detector in a table row %] [% FOREACH detector IN detectors -%] [% END -%]

Name	Detail	Type	Links
[% detector.name %]	[% detector.subname.name %]	[% detector.type.name %]	[% # Add a link to delete a detector %] Delete

For any other action than "list detectors" you need to Login

Roles for [% c.user.email %]:

[% role %]

...

Nov 25 18 Program logic and API

#!/usr/bin/env perl

use CecDB::Schema; Load the schema my $schema = CecDB::Schema->connect('dbi:mysql:test_wf;host=mysqlsrv.ifh.de'); my @users; if ($ARGV[0]) { # email address provided SQL Query if ( $ARGV[0] =~ /\@/ ) { @users = $schema->resultset('Person')->find({email => $ARGV[0]}); # select user by id } elsif ( $ARGV[0] =~ /^\d+$/ ) { @users = $schema->resultset('Person')->find($ARGV[0]); # surname given } else { @users = $schema->resultset('Person')->find({name => $ARGV[0]}); } print "please select precisely one user\n" if @users > 1; print "no user selected\n" if ! @users; } # display all users if @users empty, exit, if more than one user @users = $schema->resultset('Person')->all if ! @users; if ( @users != 1 ) { exit; }

# do the password change my $user = $users[0];

# the account name is the email print "updating password for ", $user->email, "\n"; # create password according to the specs given in the user_model $user->password('my_secret'); $user->update; SQL update exit;

Nov 25 19 The user interface

InteractionInteraction withwith applicationapplication byby sendingsending HTTPHTTP requestsrequests DefaultDefault behaviourbehaviour URLs lead to responses that can be interpreted by the web browser Web GUI, see screenshot (design study, not really useful) OtherOther formatsformats By giving other URL's the reply could consist of CSV or XML files Useful in scripts by using wget or similar tools AdvancedAdvanced interfaceinterface functionalityfunctionality Exchange parts of the displayed page in the browser Nov 25 (AJAX) 20 Usually done by asynchronously sending data (JSON format) Webgui screenshot (demo only)

Nov 25 21 Status

InitialInitial databasedatabase designdesign donedone WebWeb GUIGUI basedbased onon thatthat designdesign donedone Authentication, authorisation tested and working Upload and download functionality tested and working Display and modification of individual tables working (CRUD) Combined display of related tables tested and for a few views done Demonstrated flexibility in adding New parameters for a given measurement method New measurement methods

Proposals

NewNew DBDB designdesign readyready forfor discussiondiscussion http://www-zeuthen.desy.de/~bergholz/CEC/dbSchema_mysql.sql startedstarted creatingcreating GUIGUI basedbased onon thatthat schemescheme ProposalsProposals forfor measurementmeasurement datadata formatformat http://www-zeuthen.desy.de/~bergholz/CEC/FILEschema.pdf andand samplesample numberingnumbering readyready forfor discussiondiscussion http://www-zeuthen.desy.de/bergholz/CEC/NAMEschema.pdf

Next steps

IterateIterate DBDB schemaschema afterafter discussiondiscussion inin thethe groupgroup ProduceProduce aa moremore realisticrealistic WebWeb GUIGUI Ready to use in measurement campaign Should accept mass uploads RefineRefine WebWeb GUIGUI byby addingadding Style sheets (CSS) AJAX methods (fast page alterations without resending whole pages) WriteWrite testtest suitesuite toto verifyverify correctnesscorrectness ofof softwaresoftware Stubs for testing already present

Nov 25 24 Demo

http://www-zeuthen.desy.de:3000http://www-zeuthen.desy.de:3000

WillWill bebe availableavailable onlyonly duringduring mymy talktalk

Nov 25 25