Testing SDI on Ciabatta

Faults with Aleph SDI mechanism Outline of document Previously reported problems This summarises the problems that have been reported before, and is really for our reference: you can skip this bit, though it does indicate that the SDI mechanism has not been working satisfactorily for a long time.

Remaining faults with SDI This section explains the remaining faults, and suggests how they could be fixed by altering the Z324 mechanism.

Testing Some scenarios that can be tried to see the problems, and what should happen in a correctly working system. Previously reported problems May 2006 Support Incident 8192-24473 of 8th May 2006 reported problems with the SDI function. Another incident, 8192-48720 seems to have been created by Ex Libris and relates to the same issue. If an item was unique on a BIB record and no Z324 existed for that BIB, then any change to the item (for example, call number) triggered the creation of Z324 records at ALL, ADM and sub-library levels: dun01@ALEPH0> select Z324_REC_KEY, Z324_LOCATION, Z324_DATE, Z324_HOUR from z324 where Z324_REC_KEY like '000000303%';

Z324_REC_KEY Z324_LOCATION Z324_DATE Z324_HOUR ------000000303000000001 DOJ 20060508 1711 000000303000000002 DUN50 20060508 1711 000000303000000003 ALL 20060508 1711

3 rows selected.

If an item was unique at a particular site, but not unique on the BIB, only the sub-library level Z324 was created.

I did not investigate the Z105 message mechanism that is behind the creation of the Z324s in May 2006, so I do not know whether a message was sent for every item update that occurred, or whether it was more selective. There must have been some selection on the receiving end along the following lines to determine whether to create or update the Z324 records.

 A Z324 was only created if the item had no process status  A Z324 was not created if there was already a Z324 at the same level. I did not do exhaustive tests regarding enumeration fields etc.  A Z324 was not created if there were other items at the same sub-library with no process status, but I am not sure whether the sub-library matching was sufficient to inhibit this as I did not test whether the collection or item status had to match. Nor did I check to see whether receiving two ordered items simultaneously created a Z324 or whether because they each matched another item with no process status the Z324 creation would be inhibited.

April 2007 Incident 8192-58315 was opened on 6th April 2007 after lengthy testing of the SDI system. A Word document attached to that incident laid out how I thought the Z324 creation should be implemented, and covered issues such as the question of retrospective conversion and of receiving two or more copies of a new book simultaneously. Some of the situations posited were speculative because the system was not working adequately enough to test them.

The latest changes, in v. 18 change no. 1243, improve the creation of the Z105 message from the ADM library to the BIB. This corresponds to section B of my specification:

A Z324 should not be created unless  the item is newly created, or  the item’s process status was previously non-null and is now null, or  the item’s enumeration fields have changed, or  the item’s sub-library field has changed. These tests are most easily done by inhibiting the creation of the Z105.

The Z105 message had not been created on a change of process status or on a change of sub-library, which meant any books newly transferred to a different site were not flagged up, and books created as On Order and subsequently arrived were not notified either! Change 1243 fixes these issues, and the SDI system is now much more usable. Some issues remain, and these are explained below.

Remaining faults with SDI

If a brand new book is ordered in multiple copies, the system fails to report them as new books if the first two copies have their process statuses cleared very swiftly one after the other. This problem was predicted in test 3 of my previous document, and covered in section C of my proposed specification.

Demonstration: On our test server, Ciabatta, in acquisitions I catalogued a new order in DUN01, title “SDI test 1”, BIB 581241, ADM 581272.

I added an order for two copies, creating item records in sublibrary ML. Their barcodes were B55642 and B55643, and they were in status AO (on order). The order was sent.

I moved to the arrival tab and arrived both copies. The items move to status AR (processing).

At this point, the following SQL demonstrates that we have no Z324 records present as yet: usr00@ALEPH1> column z324_enumeration_a format a4 truncate usr00@ALEPH1> column z324_enumeration_b format a4 truncate usr00@ALEPH1> column z324_enumeration_c format a4 truncate usr00@ALEPH1> column z324_enumeration_d format a4 truncate usr00@ALEPH1> column Z324_LOCATION format a5 truncate usr00@ALEPH1> select * from dun01.z324 where substr(z324_rec_key,1,9) in ('000581241'); no rows selected

After the books have been processed, which would usually involve putting in new barcodes, labelling, etc., they would have their process status cleared. Generally the member of staff would be doing several books in one go, and would scan the items with a scanner. We will use the keyboard, and type fairly fast!

Open the Item Process Status tool from the Utilities menu in Acquisitions. Enter ZZ in the process status box, and untick the Check Requests box. Then enter the barcodes in turn, as fast as you can type (pretending you’re a very quick barcode scanner):

B55642 B55643

Close the tool, and re-run the SQL. You may need to wait up to half a minute for the messages to be processed. usr00@ALEPH1> select * from dun01.z324 where substr(z324_rec_key,1,9) in ('000581241'); **** Hit return to continue ****

Z324_REC_KEY Z324 Z324 Z324 Z324 Z324_ Z324_DATE Z324_HOUR ------000581241000000001 ML 19000101 1041 000581241000000002 DUN50 19000101 1041 000581241000000003 ALL 19000101 1041

3 rows selected.

As you can see the Z324s have been created with Z324_DATE equal to 19000101, which means that the books will not be notified to any users. If you do not get a date of 19000101, but instead get today’s date, you will need to try again, faster! I’ve found on Ciabatta that it is quite easy to enter the barcodes fast enough to cause the problem, and it is easily within the speed expected of acquisitions staff going about their work normally.

Why is this happening? The reason this occurs, is that if you are quick enough the first Z105 message is not processed until after the second item has had its process status cleared. The process which deals with the Z105 messages has to cope with the fact that Z324s do not already exist for all records on the system. It adopts a procedure along these lines:

For the ALL level: 1. If there is an ALL Z324 with matching enumeration, leave it as it is. 2. If there is another available item, apart from that in the Z105, attached to this BIB record in any ADM, create a Z324 for ALL with date of 19000101. 3. Otherwise create a Z324 for ALL with today’s date.

For the ADM level, e.g. DUN50: 1. If there is a DUN50 Z324 with matching enumeration, leave it as it is. 2. If there is another available item, apart from that in the Z105, attached to this BIB record in DUN50, create a Z324 for DUN50 with date of 19000101. 3. Otherwise create a Z324 for DUN50 with today’s date.

For the sub-library level, e.g. ML: 1. If there is a ML Z324 with matching enumeration, leave it as it is. 2. If there is another available item, apart from that in the Z105, attached to this BIB record in sub-library ML, create a Z324 for DUN50 with date of 19000101. 3. Otherwise create a Z324 for sub-library ML with today’s date. At the sub-library level, at least, the process also looks around at items belonging to other sub- libraries, and if there is no Z324 for that sub-library, it will create one with date of 19000101 if the item is available.

This generally works, except if the other item is also newly arrived. In the above demonstration, when processing the Z105 for item B55642, the process will have observed that there was another item, B55643, already available, and so creates a Z324 with date 19000101 to prevent users being notified of what it thinks is merely an extra copy.

How to solve this problem The problem as it stands is an example of a race condition. If the acquisitions staff are very slow at clearing the process statuses of the books, we will be all right, but we cannot rely on that.

I can think of two ways to alter Aleph to avoid the problem.

 The procedure could be adapted to consider the arrival dates of the items in addition to their availability.  The procedure could consider any other Z105 messages waiting in the queue.

Here’s how the first method would work. The processing of the Z105 would be altered as follows (taking the sub-library level as an example):

For the sub-library level, e.g. ML: 1. If there is a ML Z324 with matching enumeration, leave it as it is. 2. If there is another available item attached to thie BIB in sub-library ML with arrival date of zero or arrival date not equal to the arrival date of this item, create a Z324 for DUN50 with date of 19000101. 3. Otherwise create a Z324 for sub-library ML with today’s date.

If there is another available item with arrival date of zero, it is likely to have been in the library a long while, so we create a Z324 with date 19000101.

If the only other available items have the same non-zero arrival date as the item in question, however, they are likely to be part one batch of copies arrived together, and we will want a Z324 with today’s date.

The second method would examine the other pending Z105s with Z105_TYPE = 8. Any items with a Z105 pending would not be included when looking for already-available items.

I cannot judge which of these methods would work better, as it depends on how the transaction locking works in Aleph.

D: Notification should not be made for items whose arrival date is a long time ago The checks in C will not be sufficient to prevent the system notifying users of books which are not new, as thinking about the case of an item which is unique on a BIB record will show. Suppose there is only one item on a BIB record, and no Z324 has been created. The item might be put into a “Mending” process status, and then later have this process status clear. At this point, according to the proposed rules above, it would trigger a Z105 message. Because there is no Z324 and there are no other already existing items, the item would be notified to users as new. To guard against this, I suggest that a Z324 should not be created at any level if the item’s arrival date is zero or is longer ago than one month.

It would also prevent items created as part of retrospective conversion projects appearing as “new” because these will usually have no arrival date set. Most new items come through the acquisitions process and would gain an arrival date. Any donated items could have a non-zero arrival date applied manually by staff.

The criterion of “one month” maybe needs to be higher, as some sites might be slow at processing new books.

It might even be possible to dispense with the complicated code implemented for (C) if check (D) were introduced.

There is a slight problem in that if a book which arrived in 2005 at one sub-library was sent to another instead, it would not get notified as new at its new location. This would probably not matter a great deal. Staff could always update the arrival date manually before changing the sublibrary value in order to trigger notification.

Cannot test this at the moment as changing process status does not trigger the message, but it is unlikely to work like this at present! It would be best to implement this requirement at the message generation stage, that is, not to generate the Z105 at all in this case.

E: Comparison of enumeration fields (only reporting on later issues) This is a condition which appears to have been implemented (see the Oracle Table documentation for Z324 where it is referred to as the enumeration check). However, I think that this condition should probably be removed, for reasons given below.

It might be argued that a Z324 should not be updated if the enumeration fields are greater than any existing Z324 for that level. That is, if A1, B1, C1, D1 are the enumerations for the new or updated item, and A2, B2, etc. are the enumerations for the existing Z324, the Z324 should only be updated if A1>A2 or (A1=A2 and (B1>B2 or (B1=B2 and (C1>C2 or (C1=C2 and D1>D2))))) This is logically equivalent to the slightly more obvious condition A1>A2 or A1=A2 and B1>B2 or A1=A2 and B1=B2 and C1>C2 or A1=A2 and B1=B2 and C1=C2 and D1>D2

This would prevent users being notified about older issues of journals.

On balance, I think this condition would be undesirable, especially if issues of periodicals arrived out of order. A user wishing to know that issue 4 had arrived would probably still want to hear about issue 3 arriving two days later. Also enumeration is applied to multi-volume texts, which can often be published out of order, or if published in order might be ordered out of order. Just because volume 2 of The Oxford history of the British empire was published after volume 3 does not mean that the user would not want to know it had arrived. Of course, if two issues of a journal arrived within a few days and the user’s profile did not trigger a notification in between the two, then because of the way the Z324 is structured, the user will only be told about one of the issues next time they are notified. I think we’ll just have to live with that!

I have not done any tests with items with values in enumeration fields so I do not know whether the system is actually using this rule at present. If it is, it should probably be removed.

F: General requirement The Z105 messages should be created no matter how the item is updated, i.e. in Change Item Information and Change Item Process Status in Circulation, in the Group or Items nodes in Acquisitions and Serials, or automated changes in acquisitions or by services, as well as editing in the items form. Items can also have their sub-library changed en masse if they are attached to a HOL record which is edited.

The Change Item Process Status and Change Item Information boxes in Circulation have now been tested and are OK. Editing in the Serials Group node and the Serials Item List node appears to work. See tests 1 and 2 below.

I have not tested editing a HOL record.

Testing The following tests were conducted on ciabatta.dundee.ac.uk with version 18 service pack 1280 (October 2007) installed.

Test 1: what item changes will cause a Z105 message to be created? These tests are designed to check that the Z105 message is created in the correct circumstances.

Switched to the messaging library, USR00, and used Util E 12 to turn off the messaging daemon temporarily. This allows us to view the messages by doing select * from z105;

Opened catalogue record 504390 which had one item attached, belonging to sub-library DOJ.

In Cataloguing GUI, Items tab, altered following fields: Field Z105 created? Is this OK? Sub-library Yes Yes Collection No Yes Material type No Yes Call number No Yes Item status No Yes Item process status Yes Yes Enumeration A and B Yes Yes

In Cataloguing GUI, from the menu Items  Item Process Status. Set and cleared process statues. Messages created each time, which is correct.

In Cataloguing GUI, tried Global Changes in Items view. Changed following fields individually:

Field Z105 created? Is this OK? Sub-library Yes Yes Collection No Yes Material type No Yes Call number No Yes Item status No Yes Item process status Yes Yes

Did less exhaustive tests in items section of Acquisitions Orders. All OK. Serials Group node also all right.

The “Change Item Information” box in Circulation Items menu produces Z105 messages when the sub-library or process status is changed.

I then restarted the messaging daemon with Util E 11 to allow the Z105s to be processed. Afterwards, the Z324s were revealed as follows: usr00@ALEPH1> column z324_enumeration_a format a4 truncate usr00@ALEPH1> column z324_enumeration_b format a4 truncate usr00@ALEPH1> column z324_enumeration_c format a4 truncate usr00@ALEPH1> column z324_enumeration_d format a4 truncate usr00@ALEPH1> column Z324_LOCATION format a5 truncate usr00@ALEPH1> select * from dun01.z324 where substr(z324_rec_key,1,9) in ('000271239'); **** Hit return to continue ****

Z324_REC_KEY Z324 Z324 Z324 Z324 Z324_ Z324_DATE Z324_HOUR ------000271239000000001 2 DOJ 19000101 1150 000271239000000002 2 DUN50 19000101 1150 000271239000000003 2 ALL 19000101 1150 000271239000000004 NWMED 20070315 1150

4 rows selected.

Thus a new book notification has been created for NWMED. The system has, at the point of creating the NWMED Z324, observed that another item already exists at DOJ and has created a Z324 at DOJ, DUN50 and ALL levels with the date set to 19000101. This means users will not be told of the new copy at NWMED unless they have set their SDI filter to report only on NWMED items. Testing on service pack 1280: Moved DOJ item to ML and created new item at LAW. Correct Z324 created for each.

Users who want new items at DOJ reported will not be notified, despite the copy moving from GR to DOJ. This is because the Z105 for the DOJ copy was processed second, and by the time it was processed the Z324 with date 19000101 had been created at the DOJ level. If the messages had not been held up artificially, at the point the NWMED copy was adjusted the DOJ copy would still have been at sublibrary GR, and a Z324 for GR would have been created. Then after moving the GR copy to DOJ and adjusting its enumeration, another Z105 would have triggered the creation of a Z324 for DOJ with today’s date.

It might be safer not to create the Z324s with date 19000101 to avoid problems if Z105 messages are not processed swiftly. The code which determines whether to create a Z324 with date 19000101 could instead simply refrain from creating a Z324. Test 2: are Z105 messages created when automatic changes occur in acquisitions? Switched to the messaging library, USR00, and used Util E 12 to turn off the messaging daemon temporarily. This allows us to view the messages by doing select * from z105;

On our test server, Ciabatta, in acquisitions I catalogued a new order in DUN01, title “SDI test 3”, BIB 581243, ADM 581274.

I added an order for two copies, creating item records in sublibrary ML. Vendor code 10, changed order delivery type to LE. Budget AD-GEN-2007.

Their barcodes were B55646 and B55647, and they were in status AO (on order). At point of creating the item records, a Z105 was generated for each.

The order was sent. The items were then arrived, changing to process status AR (processing). Again, two Z105 messages were generated.

So it looks as though the messages are generated correctly when changes are made to order statuses.

Matthew Phillips University of Dundee 1st November 2007