Irods and Spectrum Archive

Integration of iRODS with IBM Spectrum Archive™ Enterprise Edition A flexible tiered storage archiving solution

Version 1.1 (02/17/2020)

Nils Haustein IBM European Storage Competence Center Mauro Tridici Euro-Mediterranean Center on Climate Change (CMCC)

Contents Introduction ...... 3 Executive summary ...... 4 Tiered storage file systems ...... 4 IBM Spectrum Scale with IBM Spectrum Archive ...... 5 Challenges with tiered storage file systems ...... 6 iRODS ...... 6 Integration of iRODS with IBM Spectrum Archive ...... 7 Typical workflows ...... 8 Preventing transparent recalls ...... 9 Solution in action ...... 12 Further work ...... 14 Quota ...... 14 Automated file tagging ...... 16 Display file state ...... 18 References ...... 21 Disclaimer...... 22

Introduction Arching means storing data over long time periods while providing search capabilities to find and access the data. One of the key requirements for archiving is cost efficiency, especially when the volume of archive data that must be kept for long period of time grows into Petabyte ranges.

Depending on the type of the archived data and its value the requirements for performance and cost of storage change over time. Different storage requirements can be addressed with different types of storage media, such as Flash and SSD, disk and tapes. As Figure 1 shows, Flash and SSD media provide fast access and is most expensive. Accessing data on hard disk is slower while disks are less expensive than Flash and SSD (considering enterprise grade Flash and disk). While tape storage is suitable for storing large volumes of data over long periods of time at lowest cost, access time to data on tape is significantly higher than to data on disk.

Figure 1: Storage media characteristic

Different storage media like Flash and SSD, disk and tapes can be combined in a tiered storage system. The tiered storage systems have the capability to automatically move archived data between the different storage tiers based on policies. This assures that archived data is stored on the most appropriate storage media over the lifecycle.

Another key component for archiving is a data management system that allows to index archived data and allow the user to search, share and access the data he is authorized for. Data management capabilities can be provided by the open source data management software iRODS [1].

In this paper we describe the challenges of tiered storage systems using tape as a storage tier and how these challenges can be addressed when combining iRODS and IBM Spectrum Archive™ Enterprise Edition. We demonstrate how to prevent transparent recalls when migrated files are accessed through iRODS and instead perform tape optimized recalls with IBM Spectrum Archive. Furthermore, we demonstrate configuring quota in iRODS that applies to the disk and tape storage tiers. And we demonstrate harvesting metadata from the file content and using this for subsequent searches.

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 3 Executive summary The combination with iRODS and IBM Spectrum Archive Enterprise Edition allows to establish a tiered storage environment where transparent recalls triggered by the user accessing a migrated file are prevented. Instead migrated files are recalled in a tape optimized manner which is much faster than transparent recalls and more resource gentle. The user also has the possibility to check the state of a file in order to determine if the file is on disk or on tape of the tiered storage file system.

Furthermore, this combination allows to enforce quota that applies for all user data stored in the tiered storage environment, regardless if the user data is stored on the disk or the tape tier. This prevents users from storing unlimited amount of data in a tiered storage environment.

The automatic data harvesting and indexing capabilities of iRODS allow to automatically extract information from files after they have been stored in the iRODS system and before these files are migrated to tape. The harvested information can be used to discover and search for data based on metadata information.

Finally, iRODS’ flexible plugin architecture, its data virtualization capabilities and its indexing and search capabilities paired with the cost efficiency of tiered storage file systems provided by IBM Spectrum Scale™ with IBM Spectrum Archive™ Enterprise Edition make this combination a perfect archiving solution for any types of data. There are many other use cases that can be addressed with the combination of iRODS and IBM Spectrum Archive Enterprise Edition.

Tiered storage file systems A tiered storage file system (Figure 2) provides flash, disk and tape storage within a global file system namespace where a space management component transparently moves files between the storage tiers. The file system name space is accessible by users and applications through standard file system protocols such as NFS, SMB and POSIX.

Figure 1: Tiered storage file system

Tiered storage file systems with tape are primarily used for archiving large volumes of data for long period of time. An example of a tiered storage file system is IBM Spectrum Scale with IBM Spectrum Archive Enterprise Edition that is explained in the next section.

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 4 IBM Spectrum Scale with IBM Spectrum Archive IBM Spectrum Scale™ [2] is a software-defined scalable parallel file system providing tiered storage capabilities. The IBM Spectrum Scale file systems are provided by servers that are configured in a cluster. The storage used for the file system can be within the servers or externally connected to the servers. IBM Spectrum Scale also offers exporting the file systems via standardized protocols such as NFS, SMB, HDFS and Object.

IBM Spectrum Archive Enterprise Edition [3] provides the tape tier within an IBM Spectrum Scale file system. IBM Spectrum Archive manages the tape library operations and stores the data on the tape tier in the Linear Tape File System (LTFS) format [4].

As shown in Figure 3, the combination of IBM Spectrum Scale with IBM Spectrum Archive provides a tiered storage file system with different storage media including Flash and SSD, disk and tape. While Flash and disk storage are managed by IBM Spectrum Scale directly, the tape storage is managed by Spectrum Archive. IBM Spectrum Scale provides a policy engine that allows to place the files on a storage tier upon file create and migrate the files to other storage tiers over the data lifecycle. Policies are defined and tested once and can then be configured to run automatically in the background. For example, a policy can places all new files on the disk storage tier of the IBM Spectrum Scale file system and if files have not been accessed for 30 days then migrate these files to tape storage.

Figure 3: Combination of IBM Spectrum Scale with IBM Spectrum Archive tape tier

After the migration of files to tape these files are still visible in the IBM Spectrum Scale file system and can be readily accessed. Upon access the files are automatically copied (or recalled) from tape back to disk storage. For this purpose, the tape-ID number is stored in the file metadata.

There are two kind of recalls: transparent recall and tape optimized recall. A transparent recall is triggered transparently when a migrated file is accessed in the file system. IBM Spectrum Archive loads the tape-ID referenced in the file stub, positions to the file on tape and reads the file back to disk. When multiple migrated files are accessed at the same time, then multiple independent transparent recalls are triggered, each requiring a tape to be loaded, positioned and a file to be read. If multiple files are on the same tape, then these files are not recalled in the order they are stored on tape. This causes tape start – stop operations which are usually time consuming. In addition,

Tape optimized recalls are much faster and resource gentle. The tape optimized recall is triggered by a command that includes a list of file names to be recalled. This command is passed to IBM Spectrum Archive that sorts these file names by the tape-ID (located in the file stub) and the position on tape. Afterwards it mounts the required tapes and copies the files back to disk in parallel. Hence, all files that are on one tape are copied back in the order they are stored on tape. In addition, it prevents chaotic tape loads because each tape that is required is loaded and processed only once. The downside is that tape optimized recalls in a tiered storage file system are not transparent to the user and require an administrative command to be issued.

Challenges with tiered storage file systems Tiered storage file systems with tape storage are a blessing and a curse. The blessing is that the user can see all files regardless if these are stored on disk or tape. Cursing starts when the when the user opens a file that is stored on tape because the recall takes one minute or more. Unfortunately, the user is not aware that the file is on tape because standard file systems do not indicate whether the file is on disk or on tape. The user must wait and does not know why.

It gets even worse if the user simultaneously opens several files that are on tapes. This causes even longer waiting times because transparent recalls are not tape optimized. Standard file systems cannot leverage tape optimized recalls where files are sorted by their tape-ID and location on tape before the recall is initiated in this order.

To address these challenges, transparent recalls should be avoided. Instead files that are requested for access must be collected and recalled in a tape optimized fashion. iRODS as data management layer can be used to transform transparent recalls into tape optimized recalls.

iRODS The iRODS Consortium brings together businesses, research organizations, universities, and government agencies to ensure the sustainability of iRODS by:

• Guiding further development of the software; • Growing the user and developer communities; and • Facilitating iRODS support, education, and collaboration opportunities.

The Consortium maintains and supports a commercial-grade distribution of iRODS software [10].

iRODS software is a data management layer that sits above the storage that contain data, and below domain-specific applications [1]. Because iRODS software has a plugin framework and is technology- agnostic, it provides insulation from vendor lock-in. The data virtualization capabilities of iRODS make it a one-stop shop for all data regardless of the heterogeneity of storage devices. Whether data is stored on a local hard drive, on remote file systems or object storage, iRODS' virtualization layer presents data resources in the classic files and folders format, within a single namespace. iRODS is open-source, data management middleware that enables users to [6]:

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 6 • access, manage, and share data across any type or number of storage systems located anywhere, while maintaining redundancy and security • search and find data through descriptive metadata and query tools • automate workflows through powerful rules and microservices • exercise precise control over their data with extensible rules that ensure the data is archived, described, and replicated in accordance with their needs.

iRODS rules are executed based on conditions or, in iRODS parlance, Policy Enforcement Points (PEPs). Consider, for example, a rule to prevent accessing a file that is migrated in the underlying file system. Such rule would be triggered when a file is accessed through iRODS. The rule checks if the file is migrated and if this is the case then it informs the user that it takes 1 or 2 hours before this file is being recalled. In addition, it adds the requested file name to a queue for a deferred tape optimized recall.

Users can access and manage their data and the metadata in iRODS through iRODS clients. Clients communicate with iRODS through an API. Some examples for available APIs are:

• Web-based graphical interface such as MetaLnx • Graphical desktop client such as Cyberduck • Command line interface using iCommands • Network file system interface like WebDAV and Fuse • Programming API in Python, Java, REST and C++

iRODS can be integrated with different kind of storage system providing storage space for the archived data. In the next section we describe a solution that integrates iRODS with a tiered storage file system based on IBM Spectrum Scale and IBM Spectrum Archive.

Integration of iRODS with IBM Spectrum Archive This solution integration iRODS and IBM Spectrum Archive is shown in Figure 4 and is comprised of three servers that are interconnected. One server represents the IBM Spectrum Scale cluster containing a tiered storage file system which is placed on disk and tape. The tape-part of the file system is managed by IBM Spectrum Archive. This file system is exported via NFS to the iRODS server.

The iRODS server hosts the iRODS Metadata Catalog (iCAT) database. The iCAT is a relational database that holds all the information about data, users, and zone that the iRODS servers need to facilitate the management and sharing of data. The iCAT contains the information about:

• the zone for the purposes of sharing across zones, • data and their metadata, • the virtual File system, • resource configuration including storage resources, and • user information.

The file system provided by IBM Spectrum Scale is mounted in the iRODS server via NFS (e.g. under the mount point /archive) and configured as storage resource with the type unixfilesystem. The iRODS administrative commands to configure this storage resource is the following whereby buffer is the name of the storage resource provided by IBM Spectrum Scale.

The iRODS client can host an application that interacts with the iRODS server through the available API. In this example the iRODS client command line (iCommand) is used to archive, describe, search and retrieve data.

iRODS server, client and the NFS mounted tiered storage file system represent an iRODS zone.

Figure 4: Solution architecture of iRODS with IBM Spectrum Archive

Typical workflows From the iRODS client an iRODS user – once defined and authenticated with the iRODS server –can simply archive data using the following iCommand:

$ iput file1 This command will copy the file “file1” from the local hard drive to the storage resource configured in the iRODS zone. In our example the storage resource is the NFS mounted IBM Spectrum Scale file system that consists of a disk and tape pool.

To display the files the user has archived he can do the following iCommand:

$ ils

Furthermore, the user can add metadata to the file. In this example the user adds two metadata tags: author=user1, type=text

$ imeta add -d file1 author user1 $ imeta add -d file1 type=text

To show the metadata the user can run:

$ imeta ls -d file1 AVUs defined for dataObj file1: attribute: author value: user1 units: ----

To retrieve a file the user can perform this iCommand:

$ iget [-f] file1 The file will be retrieved from the IBM Spectrum Scale file system through the NFS mount point in the iRODS server and stored on the local disk of the iRODS client in the current directory of the user.

The files archived by the iRODS client using the iput command are stored in the IBM Spectrum Scale file system that is mounted via NFS on the iRODS server. This file system is comprised of two storage tiers: the disk tier and the tape tier. Archived files are placed in the disk tier and can later be migrated to the tape tier by leveraging IBM Spectrum Archive. For example, assume files older than 10 days should be migrated to the IBM Spectrum Archive tape pool pool1@lib1. The following migration policy can be used for this:

/* define external pool */ RULE 'extpool' EXTERNAL POOL 'ltfs' EXEC '/opt/ibm/ltfsee/bin/eeadm' OPTS '-p pool1@lib1

/* define macro for access_age */ define( access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) )

/* Migration rule */ RULE 'ageMig' MIGRATE FROM POOL 'system' TO POOL 'ltfs' WHERE (access_age >= 10) AND (KB_ALLOCATED > 0)

To execute this policy, write it into a file (e.g. mig_policy.txt) and run the following IBM Spectrum Scale command on an IBM Spectrum Archive server:

# mmapplypolicy /path -P mig_policy.txt The parameter /path refers to the mount point of the IBM Spectrum Scale file system that is exported via NFS to the iRODS server. To periodically run this policy the command can be registered with a scheduler or the IBM Spectrum Scale automation framework can be used [5].

After files have been migrated, they are visible in the file system and the NFS mount and can be transparently accessed. However, accessing a migrated file using the iget command causes a transparent recall which should be avoided and replaced by tape optimized recall. This can be done by leveraging the iRODS rule engine and some custom programs as shown in the next section.

Preventing transparent recalls To prevent transparent recalls and queue the requested file for later tape optimized recall three programs are required [7]. Figure 5 shows where these programs are position within the solution architecture:

Figure 5: Solution programs

The rule-program (checkmig.re) is an iRODS rule, runs on the iRODS server, intercepts an open request for a file and invokes a check-program along with the path and file name of the file to be opened. It uses the system defined rule acPreprocForDataObjOpen to intercept the file open processing and invokes the check-program. Depending on the output returned by the check- program, the rule permits or fails the open requires using the iRODS microservice msiExit, as shown in the pseudo-code example below: acPreprocForDataObjOpen { ON($writeFlag == "0") { *fname=str($objPath); *resc=str($rescName); writeLine("serverLog", "DEBUG: Entered PreprocForDataObjOpen for file: "++*fname++" on resource: "++*resc++".");

if ( *resc == "buffer" ) then { # calling checkmig, 0=migrated, 1=not migrated, 2=stat error # if the file is migrated then checkmig will send the file name to Spectrum Archive msiExecCmd("checkmig.sh", "\""++*fname++"\"", "irods", "", "", *cmdRes); msiGetStdoutInExecCmdOut(*cmdRes,*fstat); writeLine("serverLog", "DEBUG: fstat="++str(*fstat));

# if the file is migrated (fstat=0) then fail the open processing if ( int(*fstat) == 0 ) then { msiExit("-1", "file "++str($objPath)++" is still on tape, but queued to be staged."); } else { msiExit("1", "file "++str($objPath)++" is NOT migrated."); } } } } Add this rule to the iRODS rule repository by placing the rule-file in directory /etc/irods and adding the name of the rule-file to the configuration file /etc/irods/server_config.json in the iRODS server that mounts the IBM Spectrum Scale file system. In the example below the rule file is named checkmig.re, the according entry in the configuration file is checkmig:

… "rule_engines": [ { "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance", "plugin_name": "irods_rule_engine_plugin-irods_rule_language", "plugin_specific_configuration": { "re_data_variable_mapping_set": [ "core"

The check-program (checkmig.sh) is invoked by the rule-program along with a path and file name relative to the iRODS zone encoded in $1. The check-program transforms the iRODS path and filename into name of the local mount point and determines if the file is migrated using the Unix command ls -ls. If the file is not migrated, then the check-program returns “1” to the rule- program. Otherwise, if the file is migrated, then the program returns “0”. It also transforms the iRODS path and filename into the name of the IBM Spectrum Scale file system export and adds this name to a file list located in the IBM Spectrum Archive server. For this the check-program uses an SSH-communication that requires to share the public key of the iRODS administrator with appropriate user in the IBM Spectrum Archive server. Find below a pseudo-code example of the check-program:

… # assigning variables # $1 is the objPath we get from the rule. It is composed of /$ZONE/path/name whereby path is the directory level next to the #map the file name to the local path (from iRODS Server point of view) FNAME=$(echo "$1" | sed -e "s/$ZONE/$LOCALMP/1")

#map the file name to the remote path (from EENODE point of view) GPFSFNAME=$(echo "$1" | sed -e "s/$ZONE/$REMOTEMP/1")

#stat of file passed as input argument blocks=$(/usr/bin/stat --format=%b "$FNAME" 2>/dev/null) size=$(/usr/bin/stat --format=%s "$FNAME" 2>/dev/null)

#if blocks=0 then result=0 (migrate file) #>>> here we check blocks and size to make a decision if (( "$blocks" == 0 && "$size" > 0 )); then # add the file name to the file list in the Spectrum Archive server ssh -l root $EENODE bash -c "' if [ ! -d "$FLISTPATH" ] then mkdir -p $FLISTPATH fi echo "$GPFSFNAME" >> $FLISTPATH/$FLISTNAME '" echo "0" else echo "1" fi … The check-program must be place in the microservices directory (/var/lib/irods/msiExecCmd_bin) on the iRODS server that mounts the IBM Spectrum Scale file system.

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 11 The recall-program (bulkrecall.sh) runs on the IBM Spectrum Archive server and periodically checks the file list containing migrated files to be recalled. The file list is created and updated by the check-program and contains the path and file names to be recalled. If there are files to be recalled, then this program recalls these files using the tape optimized recall function as shown in the pseudo- code example below:

… num=$(wc -l $oList | awk '{print $1}') if (( $num >= $minEntries )); then echo "INFO: Recalling $num files in $oList." /opt/ibm/ltfsee/bin/eeadm recall $oList rc=$? if (( rc > 0 )); then echo "$(date) ERROR: Recall of output file list $oList ended with error (rc=$rc)" exit 3 else echo "INFO: recall finished successfully." fi else echo "INFO: Deferring recall Candidates in $oList: $num < $minEntries." fi exit 0 … The recall-program is placed on the IBM Spectrum Archive server and scheduled to run periodically. Depending on the workload the program can run every 1-2 hours. This scheduled time period for the recall-program is the time the iRODS user has wait at maximum before he can access a migrated file. Note, when scheduling the recall-program ensure that it does not overlap with the periodic migration of files with IBM Spectrum Archive.

Once the file is recalled and the user does an iget command for this file then this command will retrieve the file immediately.

Solution in action Let us look how this solution works once the programs above have been installed and tested.

First let us put some files into the IBM Spectrum Scale file system using the iRODS client. Remember the files written by the iRODS client are stored in the NFS mount point of the iRODS server.

$ ils /archive/home/mia/col1:

$ iput file0 $ iput file1 $ iput file2 $ iput "file 1" Note the iRODS path of the file shown with the ils command. Let us have a look into the IBM Spectrum Scale file system which exports the path /gpfs/fs1/irods:

# ls /gpfs/fs1/irods/home/mia/col1/ file0 file1 file 1 file2 file3 file4 The path name of the file stored in IBM Spectrum Scale is composed of the export path plus the iRODS path followed by the zone name (archive).

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 12 Now let us migrate these files in the IBM Spectrum Archive server. For this purpose, we store the file names to be migrated in a file list and use the command eeadm migrate to migrate these files to the IBM Spectrum Archive tape pool pool1 in library lib1:

# find /gpfs/fs1/irods/home/mia/col1/ -type f > /tmp/mig.list

# eeadm migrate /tmp/mig.list -p pool1@lib1 2020-02-05 09:42:30 GLESA116E: Invalid file type /gpfs/fs1/irods/home/mia/col1/file0. 2020-02-05 09:42:31 GLESL700I: Task migrate was created successfully, task ID is 1158. 2020-02-05 09:42:31 GLESM896I: Starting the stage 1 of 3 for migration task 1158 (qualifying the state of migration candidate files). 2020-02-05 09:42:31 GLESM897I: Starting the stage 2 of 3 for migration task 1158 (copying the files to 1 pools). 2020-02-05 09:42:32 GLESM898I: Starting the stage 3 of 3 for migration task 1158 (changing the state of files on disk). 2020-02-05 09:42:32 GLESL159E: Not all migration has been successful. 2020-02-05 09:42:32 GLESL038I: Migration result for task 1158: 5 succeeded, 1 failed, 0 duplicate, 0 duplicate wrong pool, 0 not found, 0 too small to qualify for migration, 0 too early for migration. Note the error message of the migrate command for file /gpfs/fs1/irods/home/mia/col1/file0. This file has a size of 0 bytes and cannot be migrated:

$ ils file0 -L mia 0 buffer 0 2020-02-05.19:24 & file0 generic /archive/home/mia/col1/file0

Let us copy files back from the iRODS client using the iget command with the flag -f to force overwriting the file locally:

$ iget -f file1 remote addresses: 192.168.56.103 ERROR: getUtil: get error for ./file1 status = -1 Unknown iRODS error, Operation not permitted Level 0: file /archive/home/mia/col1/file1 is still on tape, but queued to be staged. As shown above our rule-program in collaboration with the check-program determined that the file is migrated and does not allow access to prevent a transparent recall.

Meanwhile we tried to copy more files from iRODS to the local disk and all ended with the error above, except file0.

$ iget -f file0 Level 0: file /archive/home/mia/col1/file0 is NOT migrated. Remember, file0 has a size of 0 bytes, so it was not migrated.

Let us investigate the recall list on the IBM Spectrum Archive server:

# cat recall_input_list.irods /gpfs/fs1/irods/home/mia/col1/file1 /gpfs/fs1/irods/home/mia/col1/file2 /gpfs/fs1/irods/home/mia/col1/file3 /gpfs/fs1/irods/home/mia/col1/file4

Since we have multiple files to be recalled, let us recall them using the recall-program on the IBM Spectrum Archive server:

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 13 # ./bulkRecall.sh ======Wed Feb 5 09:54:17 EST 2020 ./bulkRecall.sh version 0.9 started on spectrumscale. INFO: Processing file list /gpfs/fs1/.recall/recall_input_list.irods INFO: Checking if /gpfs/fs1/.recall/recall_input_list.irods exists. INFO: sorting /gpfs/fs1/.recall/recall_input_list.irods and copying to /gpfs/fs1/.recall/recall_temp_list.irods. INFO: Recalling 4 files in /gpfs/fs1/.recall/recall_output_list.irods. 2020-02-05 09:54:18 GLESL268I: 4 file name(s) have been provided to recall. 2020-02-05 09:54:19 GLESL700I: Task selective_recall was created successfully, task ID is 1160. 2020-02-05 09:54:19 GLESL263I: Recall result: 4 succeeded, 0 failed, 0 duplicate, 0 not migrated, 0 not found, 0 unknown. INFO: recall finished successfully. Wed Feb 5 09:54:21 EST 2020 Program finished successfully. As shown in the output above, 4 files have been recall.

Let us now access these files from the iRODS client:

$ iget -f file1 Level 0: file /archive/home/mia/col1/file1 is NOT migrated.

$ iget -f file2 Level 0: file /archive/home/mia/col1/file2 is NOT migrated.

$ iget -f file3 Level 0: file /archive/home/mia/col1/file3 is NOT migrated.

$ iget -f file4 Level 0: file /archive/home/mia/col1/file4 is NOT migrated.

As shown in the demonstration above, transparent recalls can be avoided while automating the tape optimized recall for all files that have been accessed while they were migrated.

Further work In this section we elaborate on further use cases and show how these can be addressed with the combination of iRODS and IBM Spectrum Archive. Quota One typical requirement for archiving systems in multi-user environments is the capability to set quota for users, groups or entire storage spaces. While IBM Spectrum Scale allows setting quota for the files stored on disk, there is no quota for files stored on tape. This means that a user with 10 TB quota can easily store 100 TB if the data is migrated to tape.

Fortunately, the combination of iRODS with IBM Spectrum Archive allows setting quota which applies for all data the user has stored in the iRODS storage resource, regardless if the data is on disk or on tape.

To set and enable quota for a given user (e.g. user1) using a given iRODS storage resource (e.g. buffer), perform the following steps:

Enable quota by editing the file /etc/core.re and adding the following line:

acRescQuotaPolicy {msiSetRescQuotaPolicy("on"); }

Set quota limit of 2 GB for user1 on the iRODS storage resource that represents the tiered storage file system provided by IBM Spectrum Scale. In this example we have one iRODS storage resource in the zone that is named “buffer”. Because we only have one storage resource the total quota limit is identical to the quota limit of the storage resource buffer:

$ iadmin suq user1 buffer 2147483648 $ iadmin suq user1 total 2147483648

Create a delayed rule that periodically checks quota by the msiQuota microservice every 60s. Store the rule in /etc/irods/quota.r:

quotacheck { delay("1s60s") { msiQuota; writeLine("serverLog","INFO: Quota updated"); } }

INPUT null OUTPUT ruleExecOut

Load the delayed quota rule that runs every 60 seconds:

$ irule -F /etc/irods/quota.r -r irods_rule_engine_plugin-irods_rule_language-instance

Now check quota for user1:

$ iquota -u user1 Resource quotas for user user1: Resource: buffer User: user1 Zone: archive Quota: 2,147,483,648 (2 billion) bytes Over: -2,147,483,648 (-2 billion) bytes (Nearing quota)

Global (total) quotas for user user1: Resource: All User: user1 Zone: archive Quota: 2,147,483,648 (2 billion) bytes Over: -2,147,483,648 (-2 billion) bytes (Nearing quota)

Group quotas on resources: None

Group global (total) quotas: None

User user1 is a member of groups: user1, public

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 15 After the user stored a 1 GB file then the quota limit changes (it may take up to 60 seconds before the quota limit is calculated, because the quota calculation runs every 60 seconds.

$ iquota Resource quotas for user user1: Resource: buffer User: user1 Zone: archive Quota: 2,147,483,648 (2 billion) bytes Over: -1,064,293,825 (-1 billion) bytes (Nearing quota)

Global (total) quotas for user user1: Resource: All User: user1 Zone: archive Quota: 2,147,483,648 (2 billion) bytes Over: -1,064,293,825 (-1 billion) bytes (Nearing quota) Group quotas on resources: None

Group global (total) quotas: None

User user1 is a member of groups: user1, public

When the user tries to store another 1 GB file then the following message appears:

$ iput bigfile2 remote addresses: 192.168.56.103 ERROR: putUtil: put error for /archive/home/user1/col1/bigfile2, status = -110000 status = -110000 SYS_RESC_QUOTA_EXCEEDED

To check a delayed rule, use the command:

$ iqstat

To delete a delayed rule, use the following command. The ID is given with the iqstat command:

$ iqdel ID

Automated file tagging When files are ingested into iRODS, for example by using the iput command, then iRODS creates some file system specific metadata and stores it in iCAT. The typical metadata includes the file name, timestamps and the owning user. It might be valuable to add more metadata for the files automatically after files have been stored. The advantage of metadata is that users can easily search for files without knowing the file name.

Consider files stored in the NETCDF-format [8]. The NETCDF-format is a self-describing, machine- independent data format that support the data exchange in science. Files stored in this format contain header information that represents valuable metadata. Let us consider an example where

© IBM Corporation 2020 and © CMCC Foundation 2020 Page 16 the header information from NETCDF-files are extracted and automatically added to the file metadata in iCAT. This is an existing iRODS project provided by Daniel Moore [9].

The github project [9] provides a new microservice (msiextract_netcdf_header) that extracts header information from NETCDF-files. To manually install the microservice, perform the following steps:

# git clone https://github.com/d-w-moore/extract_netcdf_header_msvc.git

# cp /root/extract_netcdf_header_msvc/docker_demo/extractor_Msvc /root/extractor_Msvc

# chmod +x /root/extract_netcdf_header_msvc/docker_demo/jqeditconfig.sh

# /root/extract_netcdf_header_msvc/docker_demo/jqeditconfig.sh

# export CC=/opt/irods-externals/clang3.8-0/bin/

# export CXX=/opt/irods-externals/clang3.8-0/bin/

# cd /root/extractor_Msvc && mkdir obj && cd obj

# /opt/irods-externals/cmake3.5.2-0/bin/cmake -DCMAKE_BUILD_TYPE=Debug ..

# make install At the end of the installation steps, there is a new microservice named msiextract_netcdf_header. Now let us create an iRODS rule that is invoked after a file has been stored, checks if the file is a NETCDF-file, extracts the NETCDF header information and inserts this information as file metadata in the iRODS catalog (iCAT). The rule looks like this: acPostProcForPut { writeLine("serverLog", "INFO *** data object [$objPath] has been PUT ; --> [$filePath] is physical path") if ($objPath like "*.nc") { if (0==msiextract_netcdf_header( *kv_pairs , "$filePath")) { writeLine("serverLog", "NETCDF header written:\n\t FROM physical [$filePath]\n\t TO logical [$objPath]")

foreach ( *key in *kv_pairs ) { *value = *kv_pairs.*key *pair."irods::netcdf" = "*key=*value" msiAssociateKeyValuePairsToObj( *pair, "$objPath", "-d") } } } } As shown above the predefined iRODS rule acPostProcForPut is used to intercept the put request after the file has stored.

Copy this rule to the directory /etc/irods and add the rule name to the /etc/irods/server_config.json file Let’s see how it works. First store a NETCDF-file from the iRODS client:

$ iput 20170727_restart_ice_0948.nc

After this operation completed the new iRODS rule should have extracted the header information and added it to the file. Let us look at the file metadata.

$ imeta ls -d 20170727_restart_ice_0948.nc AVUs defined for dataObj 20170727_restart_ice_0948.nc: attribute: irods::netcdf value: 27_VAR;type=NC_DOUBLE units: ---- attribute: irods::netcdf value: 18_VAR;_dimLengths=[1,102,182] units: ---- attribute: irods::netcdf value: 11_VAR;0_DIMS;id=3 units: ---- attribute: irods::netcdf value: 51_VAR;_dimNames=["t","y","x"]

etc. etc.

As shown in the output of the imeta command above, the file obtained a new attribute irods::netcdf with different NETCDF-format specific values. These attributes can now be searched using the following command:

$ imeta qu -d irods::netcdf like %27_VAR% The result of this search is that all file names that contain the attribute irods::netcdf with a value containing the pattern 27_VAR will be displayed

Display file state Another user requirement in an environment with tape as a storage tier is to check if a file is on tape or not. The user can use this information to be prepared that an access to a migrated file takes a longer time because files are collected and recalled in a bulk in an asynchronous manner.

To accommodate this, we implemented three components.

The first component is a new command script ifilestate for the user, to determine the state of a file. Install the new command in /usr/local/bin of the iRODS client. Here is an example of the ifilestate script:

rname="/etc/irods/filestate.r"

ZONE="archive" IRODSMP="archive"

#physical path to file fname=$(echo "$1" | sed -e "s/$ZONE/$IRODSMP/1")

if [[ ! -z $fname && -a $rname ]]; then irule -s -F $rname "*f=$fname" "*lf=$lfname" else echo "Syntax: ifilestate irods-path-and-filename" fi

The ifilestate script invokes a new rule named filestate.r using the irule command. This new rule gets two parameters as input from the ifilestate script: the path and file name relative to the iRODS zone and the path and file name relative to the mount point of the IBM Spectrum Scale file system (physical path). This rule invokes a new microservice named filestate.sh with the physical path, that checks the file state and returns the appropriate result. Depending on the result string returned by the microservice the filestate.r rule prints the appropriate message to the user utilizing the msiExit microservice. This new rule needs to be installed on the iRODS client in /etc/irods. If this directory does not exist, then it must be created. It can also be installed on the iRODS server. Find an example of the new rule (filestate.r) rule below:

filestate { writeLine("serverLog", "DEBUG: Rule filestate for file: "++*f++"."); msiExecCmd("filestate.sh", "\""++*f++"\"", "irods", "", "", *cmdRes); msiGetStdoutInExecCmdOut(*cmdRes,*fstat); writeLine("serverLog", "DEBUG: fstat="++str(*fstat));

# if the file is migrated (fstat=0) then return file is migrated if ( int(*fstat) == 0 ) then { msiExit("0", "file "++*lf++" is MIGRATED"); } else { if ( int(*fstat) == 1 ) then { msiExit("1", "file "++*lf++" is NOT migrated."); } else { msiExit("2", "file "++*lf++" NOT FOUND."); } } }

# assign input parameters obtained from the ifilestate command INPUT *f=$"1", *lf=$"2" OUTPUT ruleExecOut

The new microservice filestate.sh checks if the file is migrated by performing a stat-command and comparing the block size and the file size. If the block size is 0 and the file size is greater than 0 then the file is migrated. The filestate.sh return a string “0” if the file is migrated and “1” if the file is not migrated. This string is evaluated by the rule explained above. The filestate.sh script must be installed in the iRODS server directory for the microservices (/var/lib/irods/msiExecCmd_bin). Find an example of the new microservice below:

# $1 is the physical path

# do one stat call to get blocks, size and extract it out="" out=$(/usr/bin/stat --format=%b,%s "$FNAME" 2>/dev/null) if [[ ! -z $out ]]; then blocks=$(echo $out | cut -d',' -f1) size=$(echo $out | cut -d',' -f2) else # error on stat echo "2" exit 0 fi

# check blocks and size and return 0 if the file is migrated if (( "$blocks" == 0 && "$size" > 0 )); then echo "0" else echo "1" fi

# always exit with 0 exit 0

Once these three components are installed on the iRODS client and server the iRODS user can simply invoke the new command to determine the file state as shown in the example below:

$ ifilestate /archive/home/mia/col1/file1 Level 0: file /archive/home/mia/col1/file1 is MIGRATED

$ ifilestate /archive/home/mia/col1/file0 Level 0: file /archive/home/mia/col1/file0 is NOT migrated

[2] Spectrum Scale https://en.wikipedia.org/wiki/IBM_Spectrum_Scale

[3] Spectrum Archive https://developer.ibm.com/storage/products/ibm-spectrum-archive/

[4] LTFS SNIA https://www.snia.org/ltfs

[5] IBM Spectrum Scale automation framework https://github.ibm.com/ESCC/Spectrum-Scale-Automation

[6] iRODS training for beginners https://github.com/irods/irods_training/blob/master/beginner/irods_beginner_training_2019.pdf

[7] Github repository for this project: https://github.com/nhaustein/irods-tieredStorage-tape

[8] NETCDF data format https://en.wikipedia.org/wiki/NetCDF

[9] Project for extracting NETCDF metadata by Daniel Moore: https://github.com/d-w-moore/extract_netcdf_header_msvc

[10] iRODS consortium: https://irods.org/about/

The Foundation Euro-Mediterranean Centre on Climate Change has its registered office and administration in Lecce and other units in Bologna, Venice, Capua, Sassari, Viterbo and Milan. The CMCC Foundation doesn’t pursue profitable ends and aims to realize and manage the Centre, its promotion, and research coordination and different scientific and applied activities in the field of climate change study.

The information contained in this documentation is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information provided, it is provided “as is” without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this documentation or any other documentation. Nothing contained in this documentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM (or its suppliers or licensors), or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM Spectrum Scale, IBM Spectrum Archive

LINUX is a registered trademark of Linus Torvalds.