Station Software for Jodrell Bank Observatory VLBI and eMerlin 2014. P. Burgess, Jodrell Bank Observatory 1 March 2015
Updates: 28 March 2014; 1 March 2015
References:
1. The Goddard VLBI Field System (Manuals) (Vandenberg/Himwich) 2. Hsl_messages.pdf (Noble, Roger, JBO) 3. xml_messages.pdf (Noble, Roger, JBO) 4. Mark5A, B and C manuals (Haystack Observatory) 5. Linux-Based GPS Servers for VLBI (Burgess, P. JBO) 6. Jodrell Station Software Reference Manuals 1986-2014 (Burgess, P. JBO) 7. Agilent Programming Manuals (E8444 synthesisers; 9320B Analyzer)
Contents:
1. Introduction. 2. Framework 3. eMerlin Upgrade Requirements 4. New 'High Speed Link' – network basis. 5. Software development approaches. 6. 'antcn' – antenna control program 7. 'ancom' antenna control scheduler 8. 'stqkr' station command program 9. Additional station programs and functions 10. Libraries 11. Operational Notes and Functions 12. JBO ST Commands 13. Scripts 14. Diagrams 15. Notes and Updates
1.Introduction -
This document describes station software written at Jodrell Bank to allow scheduled control of radio telescopes during VLBI observations. These observations are supported in Europe and globally, by a variety of hardware and software systems both in operational and post-processing modes. Operational software is based almost exclusively on the 'Field System' developed at Goddard Spaceflight Center in Maryland, U.S.A., for over 30 years.
1.2 The GSFC Field System ('FS')
The FS was originally developed on HP minicomputers running the RTE (Real-Time Executive) operating system, which was closely integrated with hardware to provide a precise interrupt-driven machine. Rapid advances in computer technology resulted in these being replaced with standard PC hardware in the early 1990's. The current Unix-based 'RTE-A' FS was the ported to run under Linux,which remains the case.
There are two areas of code and data: the 'fs' area for the Field System itself and the 'st' (Station) area. In general the fs area is supported by GSFC and maintained to handle the large variety of hardware in the field. The 'st' area is provided for operational VLBI stations to write and install their own software. This document primarily describes Jodrell-specific soafwatre in the 'st' area (the FS itself, and its 'SNAP' control language, is well described elsewhere).
2. Framework
The FS on startup loads a series of memory-resident programs into memory which are controlled by a supervisor program. The RTE real-time system calls are emulated in this set of programs. The resulting code is preemptive wwell within period of 1 second.
Inter-process communication is handled in several ways: firstly a modelling of the HP 'Class IO' – a postbox system where programs post data to each other using a 'class' or postbox. Secondly there is a shared memory structure for both the FS and ST areas, which is highly effective for control functions. Program scheduling also involves a common integer array 'IP[5]' which passes control values to/from a scheduled program. This typically includes the 'Class IO' postbox and number of records, and errors. Finally there is conventional disk and network I/O as handled by the Linux OS. (Also we may include time-scheduling of other programs by the FS).
Note: It was convenient to duplicate some FS shared memory variables in the st area, notably the source and offset co-ordinates, to maintain isolation of variables, and to facilitate using more than one telescope. 'antcn' initialises a number of variable in routine hs_jn_init.c. It also contains some commented-out lines where variables have been left unitinitiated, so they do not get changed between FS restarts.
2.1 Station Programs.
The primary focus is on 'antcn', which is the station's interface to its own telescope control system. This program allows control of multiple devices such as receivers, wether sensors, and similar. A stub version is provided with the FS in the st 'default' area. The intention is that any dedicated I/O device (such as a serial or network interface) should be controlled by one proram, usually antcn, which is then time-scheduled by the FS.
It is frequently necessary (as at JBO) to provide a series of 'station commands' which provide I/o to the rest of the FS via the 'oprin' user interface. These commands will normaly read and may modify shared memory locations. These commands are handled by a program 'stqkr' which resides in the st area and is provided, again in stub form, in the FS default area.
There is provision for a time-schedule checking program: the Station version is called 'cheks' and again resides in the st area. This is also up to the stations to implement. There is also a 'Station Error' handling program 'sterr' which may be used, but not normally by JBO at the moment.
2.2 Shared Memory Areas
The two shared memory areas are set up in the OS on bootup, and are reserved. Any changes to their stucture requires a PC reboot. They may be read and written directly (in 'c') or by an I/O library (Fortran). (Note. Fortran is now regarded as legacy and is being replaced by 'c' code.)
The FS shared memory would normally only be accessed for functions needed by 'st', eg source co- ordinates or log file names. The form is 'fsm_addr->ffff' where 'ffff' represent a variable defined in the FS common in /usr2/fs/include/fscom.h and associated files. They may be any allowed 'c' variables. The station equivalent is of form 'stm_addr->sssss' where 'sssss' is a station variable of our choice, defined in /usr2/st/stcom.h and associated files.
The shared memory page sizes are define in /usr2/fs/include/fsparam.h and /usr2.st.stparams.h: they may be expanded if necessary, but currently have enough space for the JBO code.
2.3. Jodrell-specific requirements.
The JBO situation is slightly more complex than many stations as we are required to operate two telescopes locally, and in general, another two Merlin outstations, with disparate hardware, and integrated as far as possible with Merlin operations. We maintain two VLBI systems each with its own PC, so we allow for two telescopes to be controlled from each PC. The system has always operated on a 'readback and check' basis, rather than assuming control commands will be obeyed without checking. For this reason the 'antcn' function at JBO is split into two physical programs: 'antcn' which handles antenna commands, and 'ancom' which is time-scheduled to check for changes requested by the operator, and runs as a 'demon'. It isn't possible to interact directly with ancom.
ancom has its own window with minimal information for the primary telescope (and a secondary if it is in use). Further, an additional monitor program 'monitjb' runs in a thin window located between the 'oprin' and Jodrell FS main login windows. These two programs are additional to the FS standard setup. They are required due to the complex nature of the eMerlin system.
3. EMerlin Upgrade Requirements.
The old Merlin system operated outstations via a series of synchronous modems. This principle has been logically extended for the new eMerlin system by the replacement of modems and local serial connections by COTS network switches and cables 1t 100mbps or 1Gbps. In many cases GPIB and JB-proprietary interfaces have also been replaced by net devices, although there remains a number of legacy devices at time of writing. The connections to outstations were originally to be replaced with private networking structure, but this was regarded as prohibitively expensive. A system based on commercial broadband has been built, including reducndancy, by having two providers per station. This system is functional but occasionally causes concern with issues of latency. Also the broadband connections, which are asymmetric, are efffectively being operated 'backwards' compared to home users, which limits the bandwidth in one direction.
3.1 JBO 'st' Upgrade
The existing software had been stable for some time, other than necessary modifications due to replacement equipment. It was expected that the eMerlin project would result in significant changes both in RF and in control systems, which wold take a period of years to fully complete. At the same time it was required to maintain VLBI observing operations as actively as possible. Objectives are described more fully in an Appendix later, but in short we worked round changes in hardware whilst implementing the control upgrade.
4. New 'High Speed Link' – Network Basis.
Additional local network hardware (NICs and network switches) were added at JBO to form the 'HSL2' network along with the otcx (outstation control computer extension) PC. This formed a local network with addressess 192.168.xxx.yyy (xxx being subnetted for outstations and special functions). FS and Merlin control computers typically have two network interfaces: HSL2 and 'house' (Jodrell public) although it is intended to reduce the public network addresses over time.
Once the old Merlin analog links had been decommissioned it was no longer possible to observe VLBI with outstations by directly recording, although eMerlin recording was possible. The restoration of eMerlin outstation recording for VLBI is still under development.
The old 'antcn' software was converted to 'c' and tested under the old system, then new code written to interface with the network-based HSL2. The form of HSL2 data is described in References X and Y. The code was tested over a period of time on the second ('VLBA') field system hardware and ported step at a time to the primary ('Mk4') system. The local.MK4 system was switched permanently to HSL2 in late 2013.
5. Software Development.
It was decided to extend the shared memory environment and introduce extra functionality, also to remove some historical anomalies. Also the remaining Fortran software would be converted to 'c' in stages. Attention would be given to robustness and debugging would be inbuilt.
The form of the HSL2 messages is similar to the old one but is extended in several ways, also the separate met messages have been incorporated in the individual telescope messages. These latter have to be 'picked off' from the multicast otcx data stream. In order to access incoming data quickly and without limitations, binary-level decoding was used for the received data. Transmitted data could be handled similarly but in practice that required an extensive database-drive environment which is complex and would probably not run on our FS PCs. Accordingly an xml-based interface ('Simple Telescope Interface') was developed by Roger Noble. This is actually bidirectional but at this time it is proving effective to send with it, whilst retaining the faster binary receive. This does result in a need for extra drivers.
5.1 'ntcom' Network Driver and devices.
Unsurprisingly network-based devices have been increasing so it was necessary to interface them to the 'st' software. Although 'antcn' would normally do this, we took the approach of producing an additional memory-resident program 'ntcom' which handles non-HSL2 network devices locally. It is primarily intended for local synthesisers such as the Agilent units but flexibility has been built in which has proven useful for, for example, a remote spectrum analyzer. The structure of ntcom is integrated with the FS, and uses a device control table similar to that which the FS uses for GPIB devices.
Note: eMerlin control remains entirely with the antcn/ancom combination and is piped through the HSL2/otcx route, such that otcx and the individual outstation computers handles all device interfacing. At the time of writing, some critical devices are not yet integrated into the otcx path, so diagnostic commands written for the FS are used in the highly flexible FS procedures. These commands will be phased-out as the otcx implementation is completed. (March 2015: specifically the eMerlin attenuator devices are in this class: the 'ntatn' command works round it).
Important: Once the local telescope control computer (typically an 'Outstation VME') has taken over an instrument or device, that device is polled and updated at 1s. Intervals. This will prevent the diagnostics working in 'send' mode', which of course should not matter.
5.2 Structure and Building Station Files
The 'st' subtree is located at /usr2/st. Typically 'st' is a soft link to the current station software tree, which will be named as the current version, eg st-1.1.6/. The 'st' area has a top-level Makefile which typically accesses those directories we want to rebuild (and may omit any that are not required, eg backups). To rebuild the station software, as root, copy to /usr2/st/s-x.y.z (or untar an archive copy):
#> scp -pr prog@kelvin:/usr2/st-1.x.y /usr2 cd /usr2/st/control
– At this point check that oper:rtx owns files in /usr2/st/control. If not:
chown oper:rtx /usr2/st/control/* ; chmod og+rw /usr2/st/control/* make a link ln -sfn /usr2/st-1.x.y /usr2/st ; exit 'root' and relogin as 'prog', then rebuild the station software: prog> make rmdoto rmexe all
this will remove any existing object and binary files, and rebuild the full set, It is usually only necessary to do this when making changes to 'stcom' shared memory, and occasionally when the FS updates call for it. Important: reboot the PC after changing any shared memory.
– then edit /usr2/st/control/antdv.ctl so device 'l1' is our host.
When developing single programs, eg stqkr, it is enough to run a 'make' on that directory; suitable Makefiles are maintained. If extra code functions are added then they need to be added to that directory's Makefile. (nb: Gnu 'make' has some quirks regarding end of line separators and the placement of libraries, it's best to copy existing ones as far as possible).
To rebuild stqkr: prog> cd /usr2/st/stqkr ; make all (or rm -f *.o first).
6. Jodrell 'antcn' Station Program
Location of program files: /usr2/st/antcn Associated file: usr2/st/control/setup.jb ; /usr2/st/control/antdv.ctl
This program is now a 'c' version of the original Fortran Jodrell 'antcn' written in 1986+. It still follows the structure laid out then, and now, in the station default version, with some updates. This structure is, simplified:
Initialize variables and memory access Initialise comunication devices
Begin a loop 'forever':
Always get the incoming required station's data and store in 'stcom'
Wait to be scheduled in one of N modes by 'ancom'
Carry out that mode by sending new parameters
(end of loop and repeat).
Thus the program divides conveniently into the main program; an 'init' function, and a seriesof functions, one for each mode. There are approximately 7 'FS' modes which are required, and a number of 'Station' modes (starting from Mode 100). There are some 'legacy' modes which have been left as comments. In addition to those two layers, there is a third layer of 'utility' functions, many of which are located in Jodrell station libraries. This allows a structured, modular approach, which is generally easier to maintain and upgrade; for example, the HSL2 'transmit' functions don't need much work at the Main level, we just develop new functions.
Where possible library routines are made general so they may be used by more than one program, asl allowed by the powerful Gnu 'Make' utility. There is some duplication still present, which may be removed in future. In general we have avoided any non-standard methods in favour of simple, lowest-common-denominator approaches. Further, there is minimal use of complex 'c' forms which in any case will be resolved by the compiler. This generally improves redability and allows more comments, for a small amount of extra type.
JBO 'antcn' is therefore primarily concerned with two meta-functions: getting the data messages in from a station every second (synchronous) and sending out any request for changes in parameters from the operator (via 'ancom': see below) – asynchronous.
Init routines: the program variables init is in hs_jn_init.c; the routine hs_mrx_init.c intialises the multicast group and IP; hs_nrx_init initialises the unicast (mostly transmit) socket.
Control Files; 'antcn' has an associated control file /usr2/st/control/setup.jb. This contains default values for the Telescope, receiver, LOs, Cal diode factors, etc. and is required. Many of the variables it contains are, or may be, overridden by st commands and procedures but it should be retained to set defaults. We keep copies set up for different configurations and use scripts to switch them.
A second control file /usr2/control/antdv.ctl must be edited to contain the correct devices to enable network (HSL2) communication with the otcx eMerlin control computer. It currently requires devices for (a) the multicast message group (m1); (b) the otcx IP address (h1) ; (c) the FS host PC address (l1) and the unicats IP for transmitting requests (h2). A typical file for 'kelvin' looks like this:
*/usr2/st/control/antdv - jb antcn network devices 2010; 2013-10-14 * line 1: host(IP address or name) port(2620) time-out(centiseconds) * using an IP address avoids name server and potential network problems * * example: local host uses a short time-out * faraday (test server) kelvin (otcx) * 192.168.101.31 3000 60 192.168.101.33 7021 80 * * nx = gen. net device; h1 = hsl2 xmit; m1 = multicast; l1 = this host * h2 is hsl2 Rx; h1 with p3000 is for local testing: 'server2 3000' * n1 192.168.101.122 2061 100 h1 192.168.101.3 7024 60 * h1 192.168.101.33 3000 60 h2 192.168.101.3 7022 80 * h1 130.88.9.210 7021 60 m1 239.0.0.254 7022 60 l1 192.168.101.33 7021 60 * * end
Note: the receive and transmit IP's are currently different ports of the same host (otcx). The IP's which are commented-out have diagnostic uses, for example, by running the test program 'server2
6.1 HSL2 Message Reading Routines: 'ancom' reads back a the required telescope message once a second from the multicast group/port, usually about 8.5K of data per telescope (we can read back two stations in principle, still in testing stage). The multicast reader is in file 'mc_com.c'. The returned data is parsed in a series of smaller functions which are called from file 'hs_com.c'. Not all the variables are retained. To add an extra variable, we would add a store to stcom.h, then referring to the 'hsl_messages' pdf document (Ref 2) add the 'stm_addr->=xxx' statemnt in one of the hs_com functions.
HSL2 Transmit Commands (XML) The XML language is in increasing use for web based instrument control. There is an xml 'simple telescope interface' process which runs on otcx, and is bi-directional in that it can receive control commands and also return status 9a subset of the full Merlin set). In practice VLBI has found it most efficient to use the control-transmit mode only with received data being in binary form) but the status commands remain available for any major upgrades in the future.
(Note: XML is based on 'schemas' and libraries, which are public domain for the most part. The receiver code is mostly defined by these schemas, which are essentially templates. XML libraries and schemas have not been used for the transmit code as there is relatievly little of it, and it is desired not to install software on th eFS machines which is newer than the (fairly old) LInux OS.)
The XML commands are text-based, but some caution is required with syntax, and one should not assume anything withou checking the specifications in Ref.2 – it does however make it easier to test the transmit software, as the Test Server (see Appendix T) may be used to view the exact format.
The XML server is based on a 'schema' which returns quite alrge error messages, hence a simple decoder 'hs_xml_err.c' was developed to extract the immediatley relevant part of the message.
The other transmit commands have names of form 'hn_xml_xxx' where xxx is replaced by a function such as ource ('src'). See code for details.
Test Server: A simple tcp/udp socket server has been developed using snippets of code available on the Web: see /usr2/st/hsli/server2. To test a transmit function: change the otcx IP address in /usr2/st/control/antdv.ctl to the local host (it will be commented-out) then start the FS, which will show typically 'Error4'. Then start 'server2 3000' in an xterm and the sent messages will appear in it. This is extemely helpful when testing and debugging.
'jdebug' Command and Functions: The complex nature of the JBO HSL required debugging of some sort to be inbuilt some years ago. The HSL2 upgrade has expanded that, and it is controlled bya n stqkr 'jdebug=
Enter a set of jdebug parameters to enable the 1s onscreen printout, eg jdebug=hsl,sta,2 turns on the 2nd level of received message status. Turn it off with jdebug=off,off [0].
Important Note: The incoming multicast data is generated by a database-style object-oriented program on otcx, which suggests the actual positions and relevant locations of variable within the returned message could vary, if aditional functions are added. Since we don't use the same system to decode it (due to software compatibility isssues) it is necessary to be aware of any changes, which should be fairly rare. The 'hs_com' calls are laid out in series to allow the mods to be made coherently. (See Diagrams)
MJD find: The above process relies on a sequence of events. The met. Message is at the end of a longer run od data which is mostly not used by VLBI (site and DINT status etc ) so it is located by searching for 'n' Modified Julian Date stamps in the return buffer. This is basically a trick, but it may be useful for other purposes in the future: a function 'mjd_find.c' does this.
The returned data is illustrated in a block diagram (Figure XX). The antcn/ancom loop is illustrated in another diagram, Figure yy.
'B' Telescope Control: This is extra code to allow a second telescope to be 'slaved' to the primary dish, when making multiple-station VLBI observations. It was developed to allow Cambridge and toher telscopes to be used otside the Merlin system around 1990, and has some 'legacy' aspects in that it accepted some compromises to make it work within the limited data rate of the old modem control.Bringing it up to the same specification as the later software essentially would rewuire rewrite of the wholed JB 'antcn' system, so this is not proposed at the moment.
'B' control follows a 'master/slave' approach in that it allows a second telescope to be enabled in the /usr2.st.control/setup.jb file, which then follows the master's source requests. It does of course have to allow for offsets and some other reciever parameters to be different so there are a set of 'B' variables callable from the 'hs_com' function. Also there are a set of secondary 'B' commands in stqkr, which allow these variables to be read and set. The 'ancom' window ahs a minimal one-line indication if a secondary/slave station isenabled.
In a more modern approach, data structures would be used to duplicate functions across 'N' Merlin stations, however it has rarely proven necessary to control more than two per PC. If so, then the eMerlin system can control the network and the FS operates in 'follow along' mode by time synchronisation (no speecuial communication is required), so we leave this as-is for the pressent.
(At the time of writing the B functions have yet to be fully tested as no outstations are available).
7. 'ancom' Station Program.
As stated above this program handles the despatching of commands from 'antcn' and is required to allow the check-and-readback function, as well as the multi-telescope and instrument control we require. The function is:
Initialise the ancom window and variables on FS startup
Read the time and repeat a loop every second
Call antcn to read back the required telescope Compare the received data with requested data in 'stcom'
If new data is requested, schedule antcn to transmit
And perform some control and error checks
(end loop and repeat 'forever')
'ancom's structure is borrowed from the 'cheks' station program, but rather than issuing messages it specifically activates 'antcn' control output routines (Modes 100+ ). 'ancom' routines should not employ the nsem_take and nsem_put priority semaphores.
8. 'stqkr' Station Commands
This program also divides conveniently into levels and sections. The Main level is a command despatcher: it interprets a command typed inot the 'oprin' window and defined as an 'stqkr' function or subroutine. (Commands also need to be entered in the control file /usr2/control/stcmd.ctl).
The Command level is defined by two nested 'case' staments defined as 'isub' (command number) and 'itask' (subset of command) – these numbers are defined in stcmd.ctl so have to match the correct commands in stqkr.c.
Again there is a third 'utility' level, some of which is in shared libraries. This is convenient when building up a command for a new instrument. Many of these utilities are similar, but often not identical, as for example one may be a network device and another a GPIB device. The approach used is to duplicate an existing command and modify it. This does result in extra code residing on disk, i.e. near-duplication of code, but disk space is effectively free, and this method allows rapid developments.
The structure of a command is approximately as follows:
Get the command line buffer
Check if it's 'report only' – if so jump to 'report' and end
Else Extract the parameters and test against allowed limits
Signal any errors
Insert any defaults if requested
Store the new parameters in stcom
Perform the command :
Build a data buffer output to a device if necessary
Return data and status with a 'Class' call.
8.1 'stqkr' Station Command Notes
Note 1: For some years, most FS/ST commands reported the variables as stored in fs/stcom, so they would be '0' if no values had been sent. As instruments have become more capable, some commands will now also read back stored values from the instrument. For JBO 'st' commands an example is the 'ntlos' network synthesiser command.
Note 2: The returned data is passed back from stqkr via a 'Class IO' call, also the IP integer array is used to input the command function and task, an return any errors and the 'postbox' number.
Note 3: Some station commands which rely on HPIB communications, such as 'calon' and 'caloff' are not primitive ST or FS commands, but are made from procedures which then call the general- purpose FS command 'hpib'. This is a very convenient way to build new commands for GPIB instruments, and does not change under ST upgrades as it is based on FS-area primitives.
Writing a new command: Identify which functions are to be performed, and which I/O devices may be involved (so as to schedule the appropriate control program). It will usually be possible to find an existing command which is similar so that may be copied and edited. Note there are FS functions which may beused to conveniently parse floats and integers. Decide if the command values need regular checking: if so add the 'stchk' flagging bytes and write a module for the 'cheks' program: see below.
Debugging Aids: 'dump.c' This is a simple hex/ascii dump routine which allows binary-level debugging of buffered data (its Fortran predecssor is 'hexdump.f'). The ST software has a number of calls to this function in startegic places, usually commented-out. The can conveniently turned on when testing new commands or instruments.
Void dumpb(buf,start,nbytes,label-field); label-field is to allow a specific test to be identified.
Example: Dump an Agilent data request buffer: dumpb (ibuf,0,nch,”agdata”) gives
Dumpb agdata Start = 0 Nchar = 14 [ 0]=>02006131 46524551 3f3b504f 573f<=[ 13] ..a1FREQ?;POW?
Note that this function is in addition to the inbuilt 'jdebug' switchable functions, see Appendix.
HP/Agilent Network Instrument Commands These employ a language usually described as SCPI, which is similar but not quite identical for HPIB and Network based instruments (and also for serial-based instruments: JBO uses some of those connected via serial to netwrok converters). There are two things to note.
(a) The SCPI language vaguely resembles a stack-based language in that it has a colon-separated command tree, which requires caution to retian ones position within. Further the syntax is quite 'picky': it does not accept minimum-match, but requires the apporved shortened or full command.
(b) There are diffferences in the error return mechanism between models and between serial and network-connected version. Apply caution, and see References.
Diagnostic 'st'commands – cautionary note
Usually we expect to set all parameters from otcx via the HSL2 network. At present some functions are not completed.
The station commands ntlos,ntatn, and emlos are direct-to-hardware rather than mediated by the otcx eMerlin control software. They are primarily intended for diagnosti c use and for when otcx is unavailable (as is still the case at the time of writing). It is safe to use them in'readback' mode as required, (when they may be able to cross-confirm a setting) but should only be used to set parameters when the otcx system is not driving an instrument.
Note that when otcx does have control, it will reset an insytrument at 1s intervals, hence there is a possibility of a 'clash of interests': users need to know what the current status of otcx is.
Instruments local to the VLBI Room,such as the VLBI Agilent, and the Marconi units, are not controlled by otcx, hence direct-to-hardware is always the case.
9. Additional Station Programs.
Monitjb – a monitor-only window that shows status of source changes and basic LO settings. Located in /usr2/st/monitjb.
Cheks – time-schedule program compares actual parameters with requested ones and raises an error message if there is a mismatch. It can make use of 'sterr' to actually fix things but in practice we have not needed to do that. 'cheks' also detects when control is not available and flags up a message then turns itself off. There is an automatic 'follow-along' mode when eMerlin controls the telescopes but a concurrent VLBI schedule still runs on the FS PC, for joint observations. The FS equivalent of 'cheks' is 'chekr', which sould not be modified.
Cheks Mechanism: there is an array of byes 'stchk' which needs to be set on or off to enable or disabe checking, Typically the behaviour is not to check a function until it is set with data, at that point the appropriate 'stcmd' command should set and retain the the stchk flag. If a device is selected for checking the 'cheks' program will test its variables as defined in a suitable module which we write as part of 'checks'. Typically it will return an error message as defined in /usr2/control/sterr.ctl. The cheks flags are slightly tricky to handle and an existing example should be copied or studied to seee where to insert the code. (see for example 'mrlos'). Finally note that when cheks accesses devices used by other parts of the FS, the 'nsem_take/_put' flags needs to be employed to prevent a clash of interests. (This should not be needed in stqkr commands)
Readan – a recent program to read the Agilent 9320B spectrum analyzer via a network interface. The data is used to display a plot of the spectrum on the FS screen, useful for checking Rx band remotely. It has two modes: display a spectrum, or save it in /usr2/oper as a gnuplot data file. Gnuplot needs to be installed on the FS PC for the first mode. An earlier version of this command exists for the older Anritsu analyzer, but the later Agilent 9320B sends too much data to handle within the Class IO system, so the newer version is a separate program run from FS procedures.
Commands: anspec – Read and siplay the current 9320B spectrum on the FS PC. anspecsv: Read and store the spectrum as a file
This has recently been expanded using procedures to read and save a spectrum to the 'newton' web server (' anspecww')
Ntcom – general memory-resident network communications. This program is based on the 'ibcon' Field System GPIB comms program, but has some differences in syntax due to differing responses from network devices. It can handle binary and ascii-string data, with or without expecting a reply.
Ntcom currently has its own I/O driver or 'client' routine nclient.c. It is slightly different from the antcn equivalent version, a situation which may eventually get resolved so only one version is needed. Location of program: /usr2/st/ntcom Associated file: /usr2/st/control/netdv.ctl (example shown below)
*netdv.ctl example file * line 1: host(IP address or name) port(2620) time-out(centiseconds) * using an IP address avoids name server and potential network problems * * example: local host uses a short time-out * kelvin * 192.168.101.33 3000 100 * * dv ip pt to n1 192.168.101.222 5025 100 n2 192.168.101.122 2071 100 * n3 192.168.101.120 100 100 * u1 192.168.101.31 10001 100 u1 192.168.101.220 10001 100 * u2 192.168.101.221 10001 100 an 192.168.101.135 5025 100 t1 192.168.101.212 2051 * t1 192.168.101.33 3000 t2 192.168.101.215 2011 * * end
Programs within the FS HSL2 environment usually refer to a selected default telescope number (1- 10) in setup.jb to know where to send commands. However ntcom is able to directly address network hardware using a two-letter mnemonic defined in /usr2/st/control/netdv.ctl. This allows a potential conflict when using these commands for eMerlin devices, so they should not be used for devices which are permanently committed to eMerlin (those devices will in any case be updated every second from otcx). The ntcom devices are primarily intended for VLBI local instruments, and occasional diagnostic use elsewhere. Ntcom is required for observations using (some of ) the Agilent local synthesisers and the spectrum analyser.
Rdlog and noddy: calibration using the JBO radiometer – the radiometer box is now quite old and limited to a single 2MHz. Band, but still of use for gain curev plotting under some circumstances. The 'noddy' program does this by moving the telescope on and off source (The FS fucnction which replaces this is 'aquir' and gnplt).rdlog was used for background noise-adding radiometery and logging: the current EVN practice using a system demon (tpicd) has replaced it. hp
10. JBO Station Libraries.
They are located in /usr2/st/ and are as follows: jbclb – 'c' to Fortran interface routines (deprecated) jblib – Fortran antenna and debug routines (deprecated) jclib – 'c' functions for 'monitjb' and 'checks' jnclb – 'c' functions for antcn/ancom jstlib – standard station functions (reserved) ntlib – 'c' ntcom and network functions rdlib [ Fortran routines for cal programs rdlog and noddy stlib – 'c' Station parameters as supplied by the FS .
11. Operational Notes and Functions
11.1 Timekeeping. Telescopes need to know the time to (approximately) 10mS or btter, so those functions are typically performed locally by the 'outstation VME' computers via a timer/counter card. VME system time is usually maintained using the ntp/ntpd from servers maintained at JBO. These servers are Stratum 0 GPS units based on the CNS Systems 'TAC' clocks, potentially better than 8nS of UT. The Field System computers are now also locked to these clocks, so it is possible to set the VLBI Formatters and Mark5B clocks from it using the FS 'setcl' utility. (FS/ST utilities are available from the centre mouse button).
The primary JBO GPS server is a 1U pc (escher.ast.man.ac.uk) running Debian Wheezy (kernel 3.2) with some additional support software (Se Ref. Gg for more detail). Linux kernels from 3.2 onward have the so-called 'nanokernel' precise timing functions inbuilt.
11.2 Starting the FS and ST programs – The FS area programs are started accoridng to the list in the control fle /usr2/control/fspgm.ctl. The equivalent ST area file is located in /usr2/control/stpgm.ctl.Note that antcn is actually started from the fs (fspgm.ctl) file as it must be present in some form, evne if not used. Users programs may be ades to stpgm.ctl as required.
11.3 Other control files – Station-defined error messages are defined in /usr2/control/sterr.ctl. The files location.ctl, antenna.ctl, are required to define the active antenna (and at Jodrell are copied or linked from individual station versions). These and the other device files (ibad, dev etc) may be found described in Reference 1.
11.4 The X Window Manager and FS Setup – The FS has run under the 'fvwm' window manager since it was transported to Linux. JBO uses a 'theme' that resembles Windows (95+) but of course is not: it is done by installing 'fvwm2' and the 'fvwm95' theme files. It has been fairly extensivel customised to cover the JBO operational requirements. The relevant files are located in the 'oper' directory un 'fvwm/'. The 'menus' and 'modules' files a re the ones most likely to be changed.A set of custom icons and 'fvwmbuttons' is located in /usr2/local/icons.
11.5 'kelvin ' 'remote' X-windows; The VLBA DAR FS PC is located in a rack next to the VLBA terminal, making it inconvenient to operate directly. Hence it is usually run from the 'newton' server over a series of X windows which effectively looks as if one is directly connected. This requires ssh keys from 'oper' to be set up between 'newton and 'kelvin', and some changes to the .Xresources files on 'newton'. Remote ('exec' xterm commands are also set up in .fvwm2/modules. Hence it is important to ensure theses files are retained and copied when upgrading 'newton' (which is not subject to the FS I/O limitations discussed below).
Note: The 'pfmed' editor should be used to edit procedures. On the 'kelvin' X system it has proven tricky to set environment variables for the default editor, so pfmed picks up 'vim' instead of Emacs. The current (nasty) firx for this is to edit the file /usr2/fs/pfmed/ftnedit.c and change the default editor name. A better fix would be to set up $EDITOR on 'newton' and make it work.
11.6 The FS Hardware and dual-scsii FS Backup Approach – The computing power required is not especially large, and our system runs well enough on a dual-1.2GHz Pentium hardware, with 512MB of DRAM. More recent updates to the offline and calibration functions may benefit from more power, as would recompiling, but it is not strictly necessary.
Typically FS machines had two matched SCSI disks: the first was then backed-up to the second offline using the 'dd' utility from a debug disk such as 'Tomsrtbt' (it needs to be one with a scsii driver included in it , of course). This system has proven very robust over the years, although we do recommend replacing disks approximately every 4-5 years. Existing disks are about 40GB, and are now UW64 – ultrawide scsii. (Later FS machines from manufactureres have used triple-sata disks, which should be cheaper of course, however sata disks rely on motherboard inbuilt controllers).
JBO FS machines have complete backups made when we complete an observing session, when updated FS software is installed (before and after); and when extensive ST area development has been commisioned. There is a log file on 'newton': oper:/gnotes/Backup.log which is useful promarily for the last backup date, and some historical interest. It is now intended that the operational logs in htdocs/gm_logs be backed up entirely in the newton /data area regularly.
Development: There are images of the st area maintained on both machines: /usr2/jodrell/st-x.y.z on kelvin and /usr3/jodrell/st-x.y.z on faraday. These are typically used whne small running modifications are made. Once tested the mods. Are copied back to the backup tree. Further, the entire st tree is backed-up on PB's machine vpntst: (Projects/vlbi14/ st-.y.z) and of course when a disk backup is made.
There is no specific 'Version Control' in use. Usually we develop on a separate machine, test as well as possible, and then test operationally on (usually) 'faraday'. At that point the st version is frozen on 'faraday' and moved up to the next version for development elsewhere.
11.8 Serial Multiplexer: Older VLBI Terminals are mostly serially-controlled via the MAT (RS232) and MCB(RS422) interfaces: the standard four PC 'COM' ports are not enough so the FS and OS suppport several different serial multiplexer units. These are 'passive' units (i.e. no inbuilt buffering) which are normally set up by the Linux OS itself. The Mk4 system uses a BOCA 16-port system; on the later VLBA PC this has been replaced (by us) with a VSCOM 8-Port unit. The BOCA units require an old-style (obsolete) PC-AT interface. The VSCOMs are PCI, and the Mk4 unit will also switch to one of those units soon. The devices are set upin /usr2/control/matad.ctl and dev.ctl.
11.9 GPIB (HPIB, or IEEE488) Instrument Bus interface: Both JB machines have GPIB interface cards either from National Instruments or lower-cost clones thereof. It is still possible to run into occasional driver-rekated problems with both of these but in general they work well enough with either the proprietary driver, or the open source 'Linux Labs' drivers. These drivers were the primary reason the FS machines ran under 2.4 Linux kernels but this is now resolved with 3.x+.
11.10 JBO Serial Switch – A 1U unit in the timing rack has a number of TTL compatible 1-bit outputs (BNC connectors) controllable from the FS with the FS 'srlsw' command (eg srlsw=a,on turns on output 'a'). This usually controls the Mk2 Cal enable switch and the phasecal enable. The unused outputs are available for other uses.
Linux Distributions: FS setups are supposed to be based on Debian Linux, for its long-term stability. We now run systems based on Debian 'woody' with the 3.x kernel. (The 2,6 kernel option exists for 'sarge' but it employs the 'udev' device subsystem which was nota fully hardware- compatible).
Proposed Upgrade: It is now the case that newer GPIB drivers will operate under later Linux kernels. We have built a new FS achine running Debian 'Woody', with Kernel 3.2, which seems to find and install both the VScom and the GPIB cards. It has been tested and operational in Autumn 2014. The PC is based on an Intel/Asus quad-core CPPU and provides more power for running th e cal software etc.
GPIB Replacement: The GPIB has been around for a long time and is still in use, but has been replaced in many areas with standard Ethernet. Where possible we are replacing GPIB units with Ethernet as it is generally cheaper and easier. GPIB remains in place for some of the older units which are hard to replace.
Mark5 Recorders: These are fully described by Haystack and in their manuals (Ref.nn) but we note here that they are interfaced over the JBO 'House' network at the present moment. The eVLBI data interfaces are 1GBPS 'lightpaths' directly to JIVE and are connected separately. EVLBI operation presently requires JIVE to be able to set up Mark5 modes, so we can't put them on the internal network. For convenience, we access the Mark 5's from terminals on 'newton', although its recording functions are controlled from the Field System, we need to check the software is running, and perform offline operations such as disk pack formatting.
When operating the Mark5 recorders for observations, either recorded or eVLBI, the Mk5A should be running the program 'jive5ab' (started with ./j5) which now handles both recorded and network (e) VLBI. If this proggram is not running typically we see one or more 'mk5cn' errors returned on the FS window. On the Mark5B jive5ab should also be running, and the 5B also contains the Formatter clock, which needs to be set from the FS using the 'fmset' command.
The Mark5 is used to perform the 'autoftp' automatic fringe tests. Small data recordings are made from a running schedule, then 'ncftp'd' to a Jive server. The script /usr2/fs/autoftp does this and needs to be correctly modified on installing. (We usually have it linked to our own copy so it stays in place during FS upgrades). If this script fails it is usually because (a) the Jive server has not been started or (b) the 1G lightpath or the Mark5 1G connection is down. Our Mark5A has data output split between wo NIC's ('Interface Bonding') to allow the >1GBPS needed for 1G data transfer.
Note (Mar 2015) – To allow simultaneous recording and ftp fringe tests, 'jive5ab' now has a 'buffered' mode ( option -b) which should normally be used. The script 'Start5' does that, invoked by our ./j5 script. The Jive version needs modifying to leave data in the 'oper/' directory as it's otherwise incompatible with autoftp, which would fail.
Network Interface Cards (NICs): These are standard, usually including one on-board PCI NIC and one additional card, on the 192 and 'House' (130.) networks. They are now nearly all 1GBPS cards; we favour the Intel Pro/1000. It would be preferable if the 'House' connection were eventually removed to reduce the risk of malicious hacking: the JBO 'Bridge' machines 'goedel' and 'tycho' run proxy servers which allow external network connection from the 192 networl. For VLBI we need primarily http access for the 'apt-get' software installer and ftp for downloading schedules and sending logs. This can be setup in for example, oper '.bashrc'.
Regular Network Transfers: as stated above the primary need is to download the schedule files, usually from the EVN sever located in Medecina, Italy ('vlbeer').Due to the increasing load from eVLBI observations we automated the downloads some time ago; the script 'vxget
Station.prc: The FS can access a procedure library provided with the experiment, and a default 'station' libray /usr2/proc/station.prc. The station library occasionally requires changes during FS upgrades. Ours has accumulated a lot of old procedures which are no longer relevant and a cleanup is due soon.
Logon accounts: the accounts 'oper' (Operator) and 'prog' (Programmer) are FS standards which we also adopt locally for tother VLBI-related PCs. Typically oper and prog have group access to most areas, however it is important that FS and ST builds are done as 'prog' (after installing the files as root). See the FS upgrade notes.
General functions on the 'newton' server PC: 'newton' originally handled print services for the VLBI laser printer. This functionis retained, and uses the older 'lprng' print system with /etc/printcap definnition files on all 3 PCs. (With newer distributions the CUPS system needs to be removed and lprng installed; the /etc/printcap file should be retained. The 'magicfilter' scripts should be installed and note that lprng needs permissions enabled in /etc/lprng to allow network print access. In files /etc/lprng/lpd-perms) – It is strongly suggested that a backup copy of the entire newton /etc/directory be maintained in, sayy /data/bakcups, using dated directories.
'newton' also now hosts a VLBI webserver (http://newton.ast.man.ac.uk) which hosts our generally-available files and also an 'intranet', accessible from 130.88.9 and 130.88.24 networks only. A web logging system based on the (fairly old) 'Greymatter' GPL free software is now used to host the VLBI Operational Log System. This is being augmented slowly.
The software setup for 'newton' is described in Fig.zz, a 'Freemind' mindmap-style diagram. In general 'newton' gets upgraded every two years or so with the current Debian ('whhezy' at the time of writing). In early 2014 the hardware was upgraded to a quad-core Intel/Asus combination. It is also in use for running the 'gnplt' and 'antabfs' calibration software, which is easier to maintain on the newer system.
11.10 General Principles of VLBI Operations: Always automate new functions; always maintain the disk backups; alsways backup individual directories and files when they change. Making operations into FS procedures means they are written down precisely, and once debugged remain so.
Backup Strategy: The operational FS machines 'faraday' and 'kelvin' should have afull 'dd' disk- to-disk backup after every session completes. Usually we also do this before an FS or ST upgrade, then fairly soon after testing. Using a 'Tomsrtbt' CD reboot the machine, pasword 'xxxx' for Tomsrtbt. When it has started, the backup command is
dd if=/dev/sda of=/dev/sdb bs=1024k
The current 40GB UW SCSI disks take about an hour to complete. Note that 'dd' does not show any progress indicators: wait for the root '#' prompt to come back.
'newton' is normally backed-up less frequently but the Greymatter logs should be copied :
cd ~oper/htdocs/htintra/; cp -ar gm_log/ /data/backup14/ i.e. just copy the gm_logs directory to the newton 'data' partition. Note the use of the cp 'a' option which copies the attributes, retaining timestamps.
11.11 eVLBI and 'lightpath' Mark5 Operation – A script 'vxget' exists on the FS machines which will find and download a schedule from the 'vlbeer' repoitory (provided it's in an approved location). The 'skproc' script will process a schedule for eVLBI leaving copies of intermediate files and placing the sched and proc files ready to go. The appropriate setup procedure need to be edited in to the *.prc files. This can be done be hand but the scripts make it faster and more repeatable.
Note re fibre bandwidths: Primarily this is a Mark5, rather than FS, matter, but we note that there are three 1_Gbps fibre connections available. Currently Mark5-554 is configured with two NICs in 'bonded' configuration giving a throughput of >1200Gbps, which is required to allow for formatting headroom in a 1Gbps VLBI-data connection. Mark5-606 is limited to ~ mbps but will be configure to use the spare 'half-a-lightpath' when it is again used for routine eVLBI.
11.12 DBBC Operations (Mar 2015) – An EVN digital baseband converter (DBBC) made in Italy bu HAT labs) has been running running in parallel with the Mk4 system since late 2014, and will eventually replace it. It is a network peripheral, controlled during observations by a Field System PC running a valid schedule. The DBBC needs its network address setting up in a control files /etc/usr2/control/dbbc.ctl. The DBBC runs Windows XP and may be directly accessed for startup and any special manual configuration by VNC: 'vncviewer 192.168.101.134:1'. The DBBC will also require firmware updates as capacity is increased. It records via a VSI interface connected to Mark5B or greater machine: currently mark5-606. There is a capacity for a 10G interface ('Fila10g') which we don't have at this time.
DBBC Procedures: The DBBC has inbuilt calibration functions for 'snapshot' or 'continuous noise- adding' calibration. (The latter is usually preferred). There are differing procedure libraries needed for these, and they are again different from non-DBBC procedures. Currently the 'kelvin' FS PC has all three sets available and switching scripts are provided. We are still testing non-continuous or 'snapshot' cals.
11.13 FS Operational 'exceptions' and troubleshooting.
(where 'exceptions' are anomalies which are not necessarily errors.) a. No Multicast Data – typically the 'ancom' window shows just one (top) line and otherwise stays blank. This is the case when otcx is not transmitting multicast data on the 192.168.101. ntework. Typically there will be an otcx software update in progress, or occasionally there could have been a problem requiring a restart. Check with Controllers. b. Repeating message 'Dummy Data present'. The otcx system allows for a duplicate set of 'dummy' telescope data to be returned, used for testing. The actual telescope will be unavailable. The FS does not currently attempt to interpret dummy data. Wait for conclusion/check with Controllers.
c. The FS gives a series of errors relating to 'mcbcn' and unavailable sockets. This usually means the 'jive5ab' program is not running on the Mark5's. d. The FS shows 'unable to set clock' messages referring to 'Formatter': this is usually on the Mark5B units and simply means the Mark5B clock should be set using 'fmset' on the middle mouse button. First 'sync', answer yes; then use '.' to set the Formatter time from the FS. (nb: The Mark5B contains its own Formatter. The DBBC may also require occasional resynching from 'fmset'.). e. Fringe Test if 'autoftp' cannot locate JIVE server: two possibilities, either the JIVE server is not enabled, or possibly the Mk5 lightpaths are not connecting. Test the former by manually ftp'ing the current JIVE server; the latter with './pingj' on the Mark5.
Timesetting and clocks: Although ultra-precise timing is not required by the FS, we use network time protocol ('ntp') to set all the VLBI PC's, keeping them well within 1sec. of UTC. The master clock is escher.ast.man.ac.uk: a GPS Stratum 0 server using a CNS Systems GPS clock located in the VLBI Room.
The PC's take typically a few seeconds to a minute to lock up after a restart, assuming the clock has not been allowed to drift for a long time such that it can't lock. In that case, as root, stop ntpd, run 'ntpdate escher' on the PC, then restart ntpd. Use 'ntpq -p' to check the list of servers which is set up in /etc/ntp.conf. Once it's locked, one server will be starred: '*'. Now the FS time may be used to synchronise both Mk4 and Mark5B formatters. Once running this system is normally very stable. 12. JBO Field System Station Commands Mar 2014 aglos – get /read Agilent Local Synthesiser (VLBI Final LO) aglos=
See: Appendix xx azlfb – set/read az/el offsets for the 'B' telescope azlfb=
diode =
emlos=
dv: device is u1,u2 or u3 - note: devices 'u-' are UDP comms. f1 is between 200 and 500 MHz f2 is 0 (off) or 128 (defaul t on) f3 is 38-39.9 (no longer relevant)
associated setup file: /usr2/st/control/netdv.ctl getsr – return the current eMerlin osurce and set the FS source from it
getsr mrlos g et /read Marconi Local Synthesiser (VLBI Final LO) (GPIB) mrlos = dv,
net =
'dv' must be set up in /usr2/st/control/netdv.ctl ntatn - Set/read the eMerlin attenuator box directly
ntatn=
ntlos =
rates =
jde bug =
jbtel = jblos - Set the eMerlin(otcx) LO values (1st,2nd and 3rd)
jblos =
- eMerlin/otcs operational command jblosb - Set the eMerlin (otcx) values for 'B' telescope jblosb =
- eMerlin/otcs operational command atten - Set the eMerlin attenuator values
atten=
eMerlin operational command) attenb - Set the eMerlin attenuator values for 'B' telescope
atten=
eMerlin operational command) jbrx – Select an eMerlin receiver
jbrx = n where n is in 1..5 (at present) jbrx n,fs name,eM name eMerlin operational command jbrxb - Select an eMerlin Reciver for 'B' telescope jbrxb = n where n is in 1..5 (at present) jbrxb n,
------
Procedure-based Commands: anspec – Read back a spectrum from Agilent 9320B analyzer
Returns: ascii list of data points in temporary file oper/spectrumspc. Displays: 'gnuplot' of above data points Requires; 'c' program 'readan' ; gnuplot control file spec.plt Note: .plt file is overwritten by new spectra. Note: Dvice 'an' mustbe set up in /usr2/st/control/netdv.ctl
Alternative command: anspecsv – as above but data is stored in a local file named from the current schedule (which may be 'none') and the current date/time. This is intended for archiving plots which may be needed occasionally. anset - Set the Agilent 9320B to default VLBI mode (150MHz., 10dB span) antpi – Set the Agilent 9320B to 'total power' mode (full bw, zero span) (Show total power increase when cal diode switches) 13. 'jdebug' Debugging Command.
Form of command: jdebug=
Functions: '*' - Default – leave unchanged. , - Default value 'off ' off - Turn off hsl - High-Speed Link: net comms data I/O src - 'source' data transmit or received rat - 'rates' data, source or offset (Tx/Rx) rcv - 'Rx' config and select data. Tx/Rx los - LO tables and config Tx/Rx alc - Allocation status, jobs etc., Tx/Rx. Met - Met. Message for telescope
Modes: '*' - leave unchanged , - Default value 'off' off - Turn Off txd - outgoing data/parameters rxd - incoming data/parameters both - txd and rxd sta - hsl status and job status.
Levels; '*' - leave unchanged , - default 0 0 to 9 – sets a level of debug nb: currently only 4 levels are available but others could be added if needed. This may however result in excessively complex displays.
Data vs. parameters: in general the higher the level number the more hex formatted debug data is shown; lower levels are string printouts of values.
Internal Format: the 'jdebg' long integer (4 bytes) is available as stm_addr->jdbg. The three least significant bytes (bytes 3,2,1 or 2,1,0 in 'c' parlance) are used for func,mode, level. The most significant is currently unused. See /usr2/st/include/jdbg.h for examples of usage.
There is currently no specific 'print' function, one way round this is to run a c ommand for a second or two then cut-and-paste the screen data into a text editor in an xterm.
Levels 3 and 4 usually turn on a large ascii/hex dump: often one has to terminate the FS to 'freeze' a display. Needs a better fix.
Antcn Program Modes and Command Functions 2014-03-06 PB v1.1
Antcn Mode Purpose Routine Access Mode 0 Init. LU and stcom variables hs_jn_init fs/stcom Mode 1 FS req.source variables jn_src, jh_src fscom Mode 2 FS req. Offsets jh_off fscom Mode 3 FS report onsource status jh_sor fscom Mode 4 Send message directly to antenna (class I/O) deprecated Mode 5 FS Report Onsource, with checking jh_sor, jh_off, stcom jh_src stcom Mode 6 FS reserved for focus ctrl Mode 7 FS Report Onsource quiet mode jn_trk stcom
Mode 100 Allocate telescope control/jobno/name New mode v6.1+ Stcom/alc Mode 101 Obsolete/unused Inactive jn_com Tx Mode 102 ST Met. Data (replaced) Inactive jn_com Tx Mode 103 ST return HSL buffer (rdlog/noddy) Under review jn_com Tx
Mode 104-108 ST Data return routines (serial) Obsolete (Tx/Rx)
Mode 109 ST New Source Request hn_xml_txs hs_com Tx Mode 110 ST New Offsets Request hn_xml_txf hs_com Tx Mode 111 ST New Rx params Request hn_xml_crx hs_com Tx ST Fetch Telescope Data from HSL mc_com/hs_get mc_com Mode 112 Rx mcast Mode 113 ST send azel. Rates jn_trt (review) hs_com Tx Mode 114 ST New 'B' source Request hn_xml_txs hs_com Tx ST Fetch 'B' Telescope Data from HSL mc_com/hs_getb mc_com Mode 115 Rx mcast Mode 116 ST request new 'B' Rx. Params. hn_xml_crx hs_com Tx Mode 117 ST request new 'B' Offsets hn_xml_txf hs_com Tx
Mode 118 ST Request a Receiver Setup (w. Mode 111) hn_xml_srx hs_com Tx Mode 119 ST Request a 'B' Receiver Setup hn_xml_srx hs_com Tx Mode 120 ST Set Noise Diode Function/parms. hn_xml_tnd hs_com Tx Mode 121 ST Set 'B' Noise Diode Function/parms. hn_xml_tnd hs_com Tx
Global Send change warning to console jwarn none Antcn Mode Purpose Routine Access
Table 1 (above) describes the functions used in the foreground 'antcn' program for Jodrell telescope control. A Master/Slave model is used with a Primary or 'A' telescope fully controlled and a Secondary or 'B' telescope following it. Offsets and Rx parameters may differ.
The antcn functions are divided into two sets: passive functions which only access shared memory, and active functions which transmit demand messages via the serial port to the High-Speed Link system. The latter are scheduled by a background program 'ancom' which firstly reads back the necessary buffer (Modes 112/115) then tests for parameter changes, and schedules 'antcn' in active modes. This method ensures the telescope data display is updated once per second, and that the serial calls are scheduled precisely. See the 'ancom' diagram.
Functions which respond directly to 'oprin' input have now been removed and all changes of pointing parameters are entered into fs/stcom via FS foreground 'stqkr' and 'quikr' commands; i.e. requests are made asynchronously from 'oprin', tested by 'ancom', then (if changed) transmitted synchronously by 'antcn' calls.
The current (and operational) version of 'antcn' is the 'c' code version 6.x (based on 5.x which was converted from Fortran in Dec 2010/Jan 2011), and in use on both systems since then. It is located (initially) in /usr2/st/antcc then antcn_60, etc., with a link to 'antcn'.
Serial 'antcn' has now been replaced with a network-based (HSL2) version using a background multicast receiver, and unicast transmit. This network-based version (generically 'antcn6') is functionally equivalent to the antcn5 system and reuses a lot of the code. The structural changes in the network messages required some recoding, trabsmit routineds use text-based xml, and libraries are being improved.
Notes for Table 1:
1. Routines 'stcom' and 'fscom' only access shared memory. 2. Routines 'Tx' actively send data in XML via a unicast socket to otcx. 3. For FS, only one program should access a given I/O port at one time.
Antcn Function Protoypes (under revision 2014) int hs_jn_init(); /* Init. station parameter space. */ int hs_com(unsigned char *ibuf, char *dev); /* low-level comms */ int hs_get(unsigned char *msg,int nc); /* background data loader*/ int hs_gtb(unsigned char *msg); /* background data loader 'B' int read_dv(struct devtab dtab[]); /* read in the device control file */ int find_dv(struct devtab dtb[], char *dev); /* look up host parms for a net device */ int hs_nrx_init(char dv); /* Init. A unicast socket */ int hs_mrx_init(char *dv);/* 131013 Init. Multicast */ int hn_xml_tnd(char *dev, int itl, int mode); int nc_com(int ss, int it, int jdbg, unsigned char* msg); /* unicast comms */ int mc_com(int ss, int it, int jdbg, unsigned char* msg); /* 131013 Multicast comms */ int jn_src(int it,int im,double ra,double rd,float ep); int jh_src(int it,float f); void jwarn(int it,int im); int jn_trk(int itls); int jh_off(int it,float f); int jh_sor(int it,float t); int hn_tst(unsigned char *dev,int itls,double ra,double dec, int epcb); int hn_xml_txs(unsigned char *dev,int itls,double ra,double dec, int epcb, unsigned char int hn_xml_txf(unsigned char *dev,int itls,float azf,float elf); int hn_xml_srx(unsigned char *dev,int itls,int irxn); int hn_xml_crx(unsigned char *dev,int itls,int irxn); int hn_xml_alc(unsigned char *dev,int itls); int hn_trt(char *dev,int itls, float azf, float elf); int hn_off(char *dev,int itls, float azf, float elf); int hn_txl(char *dev,int itl); int hn_tlb(char *dev,int itl); int hn_trx(char *dev, int itl);
File naming Conventions: some attempt has been made to use similar filenames for similar functions: 'hsxxx' for HSL 'get' or receive; 'hnxxx' for HSL 'put', or transmit; jhxxx for general jodrell functions; hs_xml_xxx for xml-based receiver; hn_sml_xxx for transmit. General data manipulating routines are arbitrary but try to indicate their function.
13. Jodrell FS Operational Scripts
Clearly the Linux-based control system benefits from the sue oif script to augment or replace some functions. There are a s,mall number in use, they are located in /usr2/st/jbscr, and are re-installed by the Station Makefile.
Jprl and jprp: Printing 'drudg' Output and schedule information: The control file /usr2/control/skedf is the configuration for printing and other factors, of 'drudg'-based output ('drudg' is the FS utility which converts an incoming .vex file into schedule language). There ae two simple scripts required to pass files to the system printer, which in our case is on the serfver'newton'. These two scripts are required.
Print2dymo – This script is associated with the Dymo label printer on 'newton', and has an equivalent driver defined in newton:/etc/printcap. It is primarily of value when retrospectively printing a Field System disk pack label, and has mostly been replaced by the Windows 'Dymo' program and printer (nb: The Windows version requires a specific version of the 'Dymo software;not the latest, otherwise the barcodes are not correct).
Lgput – Send a log file automatically to a pre-defined destination at the end of a VLBI schedule. It is called by station procedure 'sendlog'. lgput is now a soft link to one or more other scripts which invoke different destination; a further station procedure logsw_xx select which destination, either manually or under schedule control.
Vxget – Automatically locate and trabsfer a schdule
Skproc – Process a schedule file in /usr2/sched for either 9a0 recorded VLBI – eg. 'skproc rp023a jb' or eVLBI – 'skproc rp023a jb e'. The latter form removes the recorder commands, and reuires a script 'mk5_del.pl' which is also present in the jbscr directory. Perl is also required (installed as part of the VLBI FS). Intermediate versions of the files are left in the directories for reference. This script doe not install a setup procedure, this should be done by hand (eg adding 'set4700-mk2').
Two other, less-used script are 'rdput' which sends one of our old-style radiometer files, and ebbc.sed and exsrc, which were used with skproc when processing multiple Merlin stations for the VLBA system.
FS Installation (March 2015) – This is described in more detail on the Haystack Web site, and in the FS Documentation. A short overview follows.
Given suitable hardware, usually with a serial multiplexer, a GPIB card, and one or more network interfaces, a default Debian system Currently the recent 'wheezy' Linux system is installed as a base system. Then a set of modified tag files allows the 'apt-get' function to install suitable sets o packages: this ithen beecomes 'FS Linux'. JBO keeps Install CD's and some notes in the green A4 'FS Linux' folder. See notes on the hardware earlier in this document.
In general updating is a matter of linking to the new FS then running 'make', however sometimes additional features of programs will require other changes, hence the 'fsupxyz' document from GSFC should be carefully followed. The 'st' area may occasionally need remaking. In both cases it's necessary to reboot the PC to reset the shared memory area.
Current 'st' Status (March 2015) Station software is at st-1.1.9 and is regarded as stable for single local telescope control. Version st-1.1.10 contains additional code for second telescope control but this is currently not enabled. The intention is to develop this further on 'kelvin' and test extensively. The 'ancom' window is likely to need some attention due to the additional 'allocation' messages, and the semi-duplicate data get/put routines need testing with debug printouts. This will become st-1.2.0 when it is issued.
14. Diagrams
(i) Operator Input flow (ii) antcn/ancom Control Flow and sequence (iii) Multicast Received Data Block Diagram (iv) Multicast Data diagram for Receiver Data (v) Screenshot of JBO FS with additional windows
15. Notes, Changes and Upgrades.
Note (a) – 'antcn' non-XML transmit. A series of functions and routines exists in 'antcn' and ntlib which perform most of the functions needed to implement the transmit blocks in 'c' without XML. This method has proved too complex and time-consuming as the Linked List structure has to be implemented by hand. The 'simple telescope' XML should now makes this redundant.
Note (b) – The eMerlin system has a series of code libraries used for eMerlin control machines. These are fairly large in size and are not likely to install and run correctly on our older OS-based machines, especially as we are obliged to follow the Field System compatibility path. It now appears this approach won't be needed unless there are major enlargements in scope.
Note (c) – The 'faraday' FS PC has been upgraded to a dual 2.8GHz AMD64 spec., i.e. not the most recent, but more powerful than the previous 1.2G units. The Linus OS is now 'wheezy'/current FS Linux spec., kernel 3.x.
Status and To-Dos: (March 2015) – The 'c' translated versions of the stqkr commands are mostly complete, and testing under operational loads is stable. Currently both the Fortran (stqkf) and 'c' (stqkc) versions are available in the st-1.1.x code area and are available for reference purposes.