Canada - France - Hawaii Telescope Corporation Société du Télescope Canada - France - Hawaii P.O. Box 1597 Kamuela, Hawaii 96743 USA Telephone (808) 885-7944 FAX (808) 885-7288

The QSO Project: Detailed Design of the Phase II Tool

Prepared by: Renaud Savalle QSO-008/Version 1.3/FINAL RELEASE Table of Contents The QSO Project: Detailed Design of the Phase II Tool 1 1 History 3 2 Introduction 3 2.1 Context 3 2.2 Relevant Documentation 3 3 Big Picture 3 3.1 PH2: an extranet web application 3 3.2 Mirroring 3 4 Application Architecture 4 4.1 Overview 4 4.2 The ColdFusion Technology 4 4.3 Functional Flowchart 5 4.4 Form List 6 4.5 General architecture of ColdFusion templates 7 4.5.1 Form 7 4.5.2 Form checking 7 4.5.3 Callbacks 7 4.5.4 Database queries 7 4.5.5 Database Stored Procedures 8 4.5.6 Runtime error management 8 4.5.7 JavaScript code 8 4.5.8 Session variables 8 4.5.9 The WDDX technology 9 4.6 Java Applets 9 4.7 Event logging 9 4.8 Issues 9 4.8.1 Usernames 9 4.8.2 WDDX and HTML tables 10 4.8.3 Database interaction with strings 10 4.8.4 Navigation 10 5 Implementation 10 5.1 Summary of data read and modified by forms 10 5.2 CFML Forms 12 5.2.1 Application.CFM 12 5.2.2 Forms 12 Algorithm 13 Events 13 5.2.3 Java components 13 5.2.4 The FOV Tool 14 Algorithm 14 5.2.5 The Pattern Creation Tool 14 5.2.6 Other components 14 6 Build Plan 14 6.1 Current State of the application 14 6.2 Future 14 7 Glossary 14 8 Bibliography 15 1 History

Version Date Comments v1.0 June 1, 2000 Initial Revision v1.1 June 8, 2000 Updated flowchart, corrections in section Implementation v1.2 June 14, 2000 Version updated for Detailed Design Meeting v1.3 June 15, 2000 Corrections after Detailed Design Meeting, ok for publication

2 Introduction

2.1 Context

This document describes the detailed design of the Phase 2 web tool, called “PH2”, which is the interface by which the Principal Investigators of programs to be observed in QS/SO mode submit the details of the requested observations.

2.2 Relevant Documentation

· The Specifications and General design of PH2 is available in the document QSO-006. · The Specifications and Design of the Phase 2 database are available in the document QSO-007.

3 Big Picture

3.1 PH2: an extranet web application

According to the specifications, PH2 is a web application and as such, relies on HTTP connections to work. HTTP is served by a web server and is carried over TCP/IP through the Internet to the CFHT users. For PH2, we specifically address the Principal Investigators of the programs to be run in QS/SO mode. Since the Internet link from Hawaii to France does not provide the necessary bandwidth to allow the use of an interactive tool, we plan to set up a mirror in France. This mirror could possibly be installed at CDS in Strasbourg, a site well connected to the astronomical French institutes, which already hosts a mirror of our web site. The link from Hawaii to Canada has proved fast enough in the previous uses of Poopsy and we are not mirroring in Canada.

3.2 Mirroring

Each site features two servers on the same LAN:

1. A dedicated Database Server maintains a copy of the PH2 Database, which holds all the data of the application. 2. A Web Server, together with a ColdFusion module, provides the intelligence of the application. This server can be used for other services (therefore in could be the same as the existing machine which holds the mirror of the CFHT web site) Canadian Users

French Users

Hawaian Users HTTP Traffic

CDS Web+CF Server HTTP Traffic

Transatlantic Link

French DB Machine

PH2 DB Mirror #2 HTTP Traffic Summit DB Machine Replication

PH2 DB

Waimea DB Machine Replication CFHT Web+CF Server

PH2 DB Mirror #1

Figure 1: The PH2 Big Picture

4 Application Architecture

4.1 Overview

The PH2 Tool is made of several ColdFusion templates or CFM files, which interact with the database to provide the web user interface in HTML format. Two graphical tools, the Field-Of-View (FOV) Tool and the Pattern Creation Tool, are Java Applets. According to the specifications, the use of those tools is not mandatory for Principal Investigators to complete the Phase 2.

4.2 The ColdFusion Technology

ColdFusion is a complete Web application server for developing and delivering scalable e-business applications. By providing powerful server-side scripting embedded into web pages CF allows to build web applications. Although what ColdFusion provides could be achieved using a more traditional approach with CGI scripts, this would be very time consuming.

The key concepts of the ColdFusion architecture are:

· Rapid Development · Session environment to build cooperative pages (HTTP is sessionless) · Application environment to handle security and authentication · The WDDX technology to integrate web languages and build offline data browsing and editing · Open integration with databases, XML · Scalable deployment, clusters

Figure 2: The ColdFusion Architecture describes the interaction between the Web Server, the ColdFusion Server and the Client Browser.

Web Browser Web Server Space http://www/form.html GET

HTML Form Web Browser FORM ACTION=SCRIPT.CGI Retrieve HTML Form Database METHOD=POST FORM ACTION=SCRIPT.CGI METHOD=POST SUBMIT HTML

POST Call

CGI Script Perl SQL http://www/script.cgi HTML

WebRESULTS Browser

HTTP Web Application HTTP Server Netscape Server Web Browser Apache...

http://www/form. cfm CFML Form GET / POST Call CFML=HTML + Server-side script

FORM ACTION=FORM.CFM CFML Form CFML METHOD=POST Web Browser HTML+JavaScript HTML RESULTS... IF POSTED... ELSE...

WDDX Database Server

JavaScript FORM ACTION=FORM.CFM METHOD=POST

SUBMIT

ColdFusion Server

Files, System Calls...

Web Server Java Applet Java

End-User SQL

Figure 2: The ColdFusion Architecture

4.3 Functional Flowchart

This chart describes the organization of the application. In the center lies the database and its tables. The database is central to a sessionless environment like the web. The application consists on a succession of web pages containing forms, the first one being the login page (top right). Individual shapes identify the functional modules. Arrows show the control flow and larger colored arrows show the data flow. PH2 Tool Functional Flowchart Renaud Savalle & Pierre Martin Version 1.2 - June 5, 2000

Login Page

Build Monitoring OBs Proceed DAD Summary [similar to OBs] HelpDesk Proceed CF Page Get User Info

SELECT ... Java Applet Bad Valid Login ? No TBD login !

Yes JavaScrit UPDATE PI address Window

User UPDATE Program User Active ? No Inactive ! SELECT row(s) from DB

INSERT row(s) into DB Update Yes OBs+SOBs SELECT User UPDATE Obs UPDATE OBs UPDATE Prg Calc. I-Time (Priority,Tra ck, Notes) User UPDATE row(s) into DB User Logged ? Yes already Logged ! DELETE Obs DELETE row(s) from DB DELETE Error No Delete DELETE Obs Sequence Bad Correct Pw ? No DB Op Pw ! FAILED DELETE Error Proceed DELETE OB Observation Yes Delete INSERT Error OB Del Seq Usr

INSERT Error INSERT Obs Program Selection

INSERT Obs Delete OB Create Sequence Proceed

Yes SELECT Programs Proceed Del Seq Create Seq Create Sequence I-Time Left ? OB INSERT OB Program Presentation Create Yes OB Create OB I-Time Left ? Del OB Create OB SELECT Program No No Prg Build Observing SELECT Prg I-Time Proceed SELECT Obs Blocks UPDATE Program

SELECT OBs Update Filter Not SELECT Targets Prg Data enough I-Time ! SELECT ICs OK Targets UPDATE Program Program Constraints UPDATE failed Go Back ! Pattern Proceed

SELECT Constraints Instrument Config Proceed IQ GSC Stars Adjust Calculated I-Time (Session SELECT Targets Update Prg variable) Airmass UPDATE Error Pointing Constraints

Sky Bg SELECT Target SELECT Pointings

OK DELETE Targets

Init Constraint Instrument INSERT Target Session Vars Save Form Data to Reload INSERT Target Session Vars Reload SELECT Constraints SELECT Filters SELECT Pointings UPDATE Targets SELECT Targets INSERT User Pattern Targets INSERT Constraints SELECT Patterns SELECT ICs Reinit FOV SELECT Target Init Session INSERT ICs Vars SELECT IQ Del Add Proceed Upload UPDATE ICs Delete Marked SELECT Airmass DELETE ICs SELECT Sky Bg Add n Add Constraints Delete n Targets

Add n INSERT Pointing Upload From File ICs DELETE Constraints OK Add n Delete n Proceed ICs Targets Delete n OK Save Form Constraints SELECT Field Stars Add Data to INSERT Error Add Parse OK Session Vars Save Form Data to File Delete Marked Session Vars INSERT Error

Delete Marked SELECT InstrumentDELETE GeometryError UPDATE Constraints Save Form Data to Session Vars Update Reload Targets From Add Del Form Start FOV Tool Init Reload Moving Targets Add Del Instrument Configurations Session [similar to Targets] UPDATE Error Init Vars Update Constraints Session Proceed Proceed Pattern Reinit ICs INSERT Error Vars Update Proceed Proceed Reinit Constraints Parse Error DELETE Error

Save Reinit DB Op User INSERT Error FAILED DELETE Error Save UPDATE Error Pointing Go Back ! User INSERT Error UPDATE Error Pattern Start Pattern Tool Reinit

DB Op FAILED File Format Go Back ! Error Retry Upload ! DB Op FAILED Save Pointing Go Back ! Save Pattern

User Pattern Creation Close FOV Tool Tool

Close No No

Pointing Saved ? Pattern Saved ?

Yes Close Window Yes Close Window

Figure 3: PH2 Functional Flowchart

4.4 Form List

The following forms constitute the PH2 application:

1. Login 2. Program Selection 3. Program Presentation 4. Fix Targets 5. Moving Targets 6. Instrument Configurations 7. Constraints 8. Observing Blocks and Sequence of OBs 9. Monitoring Observing Blocks 10. Archiving 11. Summary 12. HelpDesk

Each form leads to the next one via the PROCEED button. Moreover, all forms are available from a menu, which always remain available in a separate frame.

4.5 General architecture of ColdFusion templates

A common architecture applies to all the ColdFusion Templates of the application. Each CF template contains the code for both the form and the scripts, which are executed when a button of the form is pressed. ColdFusion provides both the read/write database interaction through SQL requests, as well as the code of the PH2 application itself. Once the CF template has been processed by the CF Server, the code sent to the client consists only of HTML and JavaScript code.

In addition, CF provides Session variables which are used to manage data which is not yet saved into the database, thus avoiding unnecessary updates when the user has not validated the form data but it is necessary to reload the page (e.g. to display a newly created row)

The main implementation choices for these CF templates are described below.

4.5.1 Form

The form is an usual HTML form, possibly using CF improved versions of the well-known HTML form tags (INPUT etc).

4.5.2 Form checking

Form checking allows to catch the errors in input data before they are caught by the database via the enforcement of constraints during the INSERT or UPDATE queries, or before they cause a database error if such constraints are not implemented.

The check of input parameters is done using JavaScript functions on the client side. For simple cases, those JS functions are handled by the ColdFusion Server from parameters of the CFINPUT tag, an improved INPUT HTML tag which automatically creates JavaScript functions on-the-fly)

4.5.3 Callbacks

When a button is pressed, the same form is reloaded, and branching allows to perform only the necessary actions. Then the display is updated. If the action to be performed features a change in the database which should be reflected in the interface, the form has to be reloaded. In that case, Session variables keep all non-saved data.

4.5.4 Database queries

There are four types of database queries: SELECT (retrieve data) INSERT (create new object) DELETE (delete existing object) and UPDATE (update the fields of an existing object). Those queries, associated with the strong potential of SQL, as well as the referential integrity of the database, are enough to perform all the necessary operations while maintaining data integrity.

4.5.5 Database Stored Procedures

Complex queries which are often executed can be encapsulated into compiled Stored Procedures (SP) written in T-SQL, which execute on the database server. Those SP carry the “business logic” of the application and can benefit all the clients accessing the database. Stored Procedures can be called directly from CF1

4.5.6 Runtime error management

Due to wide range of values allowed, concurrent access that may generate DB load etc. runtime error condition may occur.

· Request (error 404 etc) · Validation · DB queries

It is expected that most errors will come from DB queries, whether caused by timeouts, load, or server error. We plan to take the following action upon errors:

· Display a detailed message, allow to send error report to the QSO team · Allow to retry (in case of load/bandwidth problem) with the same form kept,.

Error management and reporting will uses the same look-and-feel (using CFERROR).

NB: The use of a JavaScript Alert Window is practical but does not allow to print/send the result to the support team.

4.5.7 JavaScript code

In addition to form checking, JavaScript code is used to give more interactivity to the pages in order to avoid unnecessary reloads when all the data has been sent and loaded into JS objects (e.g. the dynamic Instrument Configuration menu in the Observing Block form),

4.5.8 Session variables

In the above templates, the new information is saved only upon request when the user clicks on the Proceed button. However the page needs to be reloaded when a row is added or deleted from an array (since this implies redrawing the page and not changing a control, it cannot be easily done with JavaScript alone). Therefore, the form data is saved into Session variables whenever the page must be reloaded.

Session variables are also used to enforce the “user only logged once at a time” paradigm.

CF maintains a list of Session variables in a separate database, but stored on the same database server. Thus mirroring the database server allows to detect two concurrent users (two users using the same userid at the same time) provided that the replication link is working. If this link is down, we may revert to

1 cf [BF1] p 429 allowing each P.I. to connect only from a selected server (defaulting to the one closer to the country of affiliation of his program, but which can be different, e.g. for CFHT resident astronomers)

4.5.9 The WDDX technology

The Web Dynamic Data Exchange is a new technology introduced by Allaire in ColdFusion. It allows to serialize data into an XML-compliant packet format, in order to allow exchange of data between all the components of a web application written in different languages (CFML, JavaScript but also COM objects). WDDX allows to make more intelligent forms by providing the necessary function to map the CFML variables to JavaScript variables. All the JavaScript code can then work in offline mode.

In particular, one interest of WDDX for PH2 is the possibility to design offline data-browsing applications2. If the forms are written this way, they behave like small independent JavaScript spreadsheets which do not require a server connection to work. Only the final submission of data will need a connection to the web server.

Serialization of data also easily allow users to save and reload their work on their local machine, without affecting the server, until they are are totally done with everything.

The combination of those features would make it possible for users to work in a semi-offline mode where they could, once a form has been downloaded, work offline, save their work, close the browser, until the next time they connect on the same form, to complete the submission. Another advantage is that this approach puts less pressure on the server by avoiding unnecessary INSERT/DELETEs.

For example, if the Targets form is written this way, the Add and Delete buttons do not create new rows in the target table but only affect the form page. Only the Proceed button will actually update the database.

4.6 Java Applets

When graphics and interactivity is needed we use Java Applets. Since their use is optional and requires an interaction with the main form they are launched from, those applets are displayed in a separate window.

The LiveConnect technology is used to link Applets and web pages via JavaScript. LiveConnect makes available to JS the public methods of the Java Applets and allow to control and exchange data with them from a web page.

4.7 Event logging

The event table of the Phase 2 database will record every major event happening during the application (user login/logout, data modification). This will allow troubleshooting and make user support easier.

4.8 Issues

Most of the above technology has been validated on prototypes. The main issues identified are listed below:

4.8.1 Usernames

2 c.f. [BF2] p 313 DB queries can be executed using always the same generic www DB login (www_ph2) or using the user’s PH2 login. Here are the pros and cons of both approaches:

· User login: ColdFusion allows a USERNAME in each CFQUERY : although more complicated to manage, this allow debugging and reporting by logging a user_insert column for each table. In addition we need to manage all the PH2 users as database users (this can be automated with scripts) · One generic web login for all users: this approach is simpler to implement, but less troubleshooting capabilities

4.8.2 WDDX and HTML tables

Since the specifications include the use of arrays to display data, and since changing the contents of an array needs the page to be rewritten, the use of WDDX in our application will certainly require some dynamic HTML form rewriting. Specifically, in order to process the Add and Delete events this way, we need to validate the integration of CFML, WDDX, and JavaScript with self-generating HTML Forms containing HTML tables. This aspect needs further testing to be validated.

4.8.3 Database interaction with strings

In the CF code, queries may need string pre/postprocessing to avoid syntax errors, since some characters like ‘ must be backslashed in the database. ColdFusion provide the necessary functions to do so.

4.8.4 Browser navigation

As a web application, PH2 should enforce only client navigation which is compatible with the application control flow.

In particular the back and forward buttons events of the client browser should be handled so that it is not allowed to go back once a form is submitted, because the data shown would be inconsistent with the state of the database and subsequent submissions could lead to impredictable results.

5 Implementation

See document QSO-006 for the appearance of pages. Only a few details were changed (ability to delete n marked rows instead of only one, some text added). In the following we describe how we plan to implement PH2 using CFML and JavaScript.

5.1 Summary of data read and modified by forms

The following table summarize the data SELECTed and UPDATEd by form, along with the Session data to keep in each form. We also describe where the default data come from (Phase 1 database, TAC, a default or a NULL value)

Form Table Column Condition Init Read Writ Session e Login usr userid Ph1 Y N N password Ph1 Y N N Program prg runid where usr.id = prg.usr_id Ph1 Y N N Selection Program prg title Ph1 Y Y N Presentation abstract Ph1 Y Y N agency Ph1 Y N N type Ph1 Y N N runid Ph1 Y N N itime_alloc TAC Y N N tac_rank TAC Y N N tac_grade TAC Y N N usr fname where usr.id = prg.usr_id Ph1 Y N N lname Ph1 Y N N institute Ph1 Y Y N email Ph1 Y Y N Program prg is_monitor F Y Y N Constraints is_photometric F Y Y N is_moving_fields F Y Y N ph2_comment NULL Y Y N Fixed Targets target label Auto Y Y N name Ph1|NULL Y Y Y magv NULL Y Y Y is_pi_checked F Y Y Y target_fix req_ra where Ph1|NULL Y Y Y target_fix.id=target.id req_dec Ph1|NULL Y Y Y req_equinox 2000.0 Y Y Y

pointing name where 1 Y Y Y pointing.id=target.poi_id ra_offset 00:00.0 Y N N dec_offset 00:00.0 Y N N Moving Targets target label Auto Y Y N name NULL Y Y Y magv NULL Y Y Y is_pi_checked F Y Y Y target_oe longitude where NULL Y Y Y target_fix.id=target.id smajor_axis NULL Y Y Y excentricity NULL Y Y Y inclination NULL Y Y Y ascnode_long NULL Y Y Y peri_long NULL Y Y Y oe_equinox NULL Y Y Y elong_min NULL Y Y Y elong_max NULL Y Y Y pointing name where 1 Y N Y pointing.id=target.poi_id ra_offset 00:00.0 Y N N dec_offset 00:00.0 Y N N Instr Config iconfig label Auto Y Y N name NULL Y Y Y filter NULL Y Y Y exptime NULL Y Y Y pradius 15.0 Y Y Y pattern name where Single Y Y N pattern.id=iconfig.pat_id description … Y N N

binning name where 1 Y Y Y binning.id=iconfig.bin_id

Constraints cons name NULL Y Y Y iq name where cons.iq_id=iq.id IQ<=0.55 Y N Y skybg name where Dark Y N Y cons.sky_id=skybd.id airmass name where <1.2 Y N Y cons.air_id=airmass.id OBs and SOBs ob label Auto Y N N comment NULL Y Y Y itime from iconfig Y N N target label previously Y N N entered values name Y N N iconfig label Y N N name Y N N cons label Y N N name Y N N obs label Auto Y N N type Auto Y Y N number Auto Y Y N Mon OBs Idem OB plus: ob mon_period NULL Y Y Y mon_niter NULL Y Y Y mon_niter_min NULL Y Y Y

5.2 CFML Forms

We here describe common implementation details for the application forms

5.2.1 Application.CFM

This file is implicitly included by all the CF Template of the application and holds most of the shared code:

· Authentication · Error management (CFERROR) · Application, Client and Session variables (global variables with a fixed scope)

5.2.2 Forms PH2 forms are designed along the same model:

Algorithm

1. Handle different types of submissions · Add row: add a new row into main table(s), re-affect labels to all rows to enforce no-gaps constraint, save form fields into session variables, reload page · Delete row: delete row(s) from main table(s), re-affect labels to all rows to enforce no-gaps constraint, save form fields into session variables, reload page · Proceed: update database, go to next form · Upload File: Parse file, unserialize, overwrite session vars · Save to File (only with WDDX) : serialize and write file to client disk · Reinit: reinitializes form from database, canceling modifications

1. Load accessory tables which hold data to be shown on the form Example for Targets Form, load table poinrting with condition: is_user=F or (is_user=T and usr_id=current_userid), then store columns label,ra_offset,dec_offset into temporary array

2. Load main table (or tables with a join) Example for Targets Form: join on target (condition: prg_id = current_prg_id) and target_fix (condition: id=target_id)

3. Initialize session variables if form not submitted 4. Show form with default from session vars 5. Handle events

Events

Three kinds of events must be handled:

1. SUBMIT Events (C.f. beginning of Algorithm above)

2. JavaScript Events Example in Targets Form Menu selected: update offsets accordingly

3. Launch Applets Events Example in Targets Launch Applet FOV with selected object

Issues

· Discriminate SUBMIT events ([BF1] p 239) · Split/unsplit coordinates (RA,DEC) fields · Immediate validation of input forms with JS events onBlur vs CF does validation onSubmit only

5.2.3 Java components

The use of Java is limited to two optional tools in the application. 5.2.4 The FOV Tool

The FOV tool is described in QSO-006, section 5.4.1. The goal is to allow the P.I. to position the mosaic precisely on the target, avoiding bright stars.

Algorithm

1. Input: positions entered in the Targets Form 2. Get the GSC stars by (call a DB Stored Procedure linked to getgsc or calling getgsc directly) 3. Calculate projection for display 4. Display Stars 5. Display the mosaic 6. Handle user events to move the mosaic, update fields

NB: The existing FOV tool for CFH12K “TFM” could be integrated or used as a starting point.

5.2.5 The Pattern Creation Tool

5.2.6 Other components

The CFH12K Exposure Time Calculator, which is developed by Elixir (project DIET), must be integrated with PH2.

6 Build Plan

6.1 Current State of the application

At the time of writing the database has been implemented and populated with test data. A prototype of the Targets Form has been written to validate the ColdFusion technology and the intergration with the database. The WDDX technology is being tested to provide semi-offline forms. Authentication has been tested and needs to be integrated. A prototype of the Java FOV applet (without the GSC stars) has been made. All the other forms, as well as the Pattern Creation Java applet need to be implemented.

6.2 Future

According to our estimation of the implementation work and in accordance to the SAC resolutions, we plan to release the first version of PH2, including all the mandatory forms, at the beginning of August 2000. In September 2000 we plan to install the French mirror (depending of contacts with CDS and shipping of the Sybase Rep Server)

Poopsy data along with TAC ranking should be integrated into the Ph2 database after the TAC evaluation for semester 2001A (mid-October 2000) and PH2 should be open to the P.I.s one month before semester 2001A starts, i.e. at the beginning of January 2001.

7 Glossary CF ColdFusion (C.f. part 4) CFML ColdFusion Markup Language, a superset of HTML allowing embedding server-side scripting into HTML pages. Phase 2 2nd phase of the program/observation submission process where P.I.s must describe their observations with enough details to allow the work to be carried out in QS/SO mode PH2 Name of the Phase 2 Web tool, which allows the P.I.s to complete the Phase 2. P.I. Principal Investigator, the astronomer responsible for a program. QS Queue Scheduling: A queue of observations to be carried out in SO mode are optimized using a scheduler and scheduling rules. QSO Name of a the project consisting of implementing QS and SO at CFHT SO Service Observing: observations is done by the local staff for the P.I.s SP Stored Procedure, a procedure written in T-SQL and residing on the Sybase server. T-SQL Transact-SQL, the internal language of the Sybase RDBMS. WDDX Allaire’s Web Dynamic Data Exchange, format designed to exchange data between the components of a web application written in different languages, using XML

8 Bibliography

[BF1] Ben Forta - The Coldfusion web application construction kit, 3rd edition, QUE Ed. [BF2] Ben Forta – Advanced coldfusion development, QUE Ed.