Loom REST API V1

Loom REST API Document Revision 0.10 - 01 Aug 2014 Loom REST API API Version ‘V1’

This documents the ‘v1’ version of the Loom API. This version is applicable for Loom 2.2 and beyond. API Overview

Operations in Loom are all managed through a set of HTTP-based APIs. While all operations can be performed through the Loom application, users may also access the APIs directly.

The Loom API is designed to be versioned. The initial version is ‘v1’, accessible through the Loom server URL from the root:

http://:/api/v1/

Basic Organization of API

The Loom API is organized into two parts: the Resource API centers around resources managed in the Loom Registry: sources, datasets, etc; and the Activity API focuses on activities that users can perform with Loom: executing transforms, accessing data, etc.

Resources and Resource API

The Resource API is focused on entities, which are exposed as resources according to their types. The routes shown in the following tables are relative to the Loom root, e.g. the ‘datasets’ resources are accessible from http://:/api/v1/datasets.

Resource Description Route sources Sources of data, not directly controlled by Loom. /sources datasets Sets of data whose lifecycles are controlled by Loom. /datasets processes Processing performed on datasets. /processes jobs Tracking of asynchronous processes executed through /jobs Loom. users User accounts. /users glossaries Glossaries of business terms. /glossaries relationship Dynamic relationships between entities. / s relationships

-1- Loom REST API Document Revision 0.10 - 01 Aug 2014

The ‘generic’ form of the Resource API is accessed through the following endpoints. Note that these parts of the API are not as well-developed as the type-specific instance methods shown above.

Resource Description Route entities Generic entities (irrespective of type) /entities types Type information. /types

Activity API

The Activity API is focused on activities that users can perform against the Loom system.

Activity Description Route connection User login, logout, ping /connect search Related to search - fulltext search, filters for search, etc /search data Data access, reading files and getting data from datasets /data execution Executing transformations against datasets /execute environment Environment interactions - browse the file system, etc /environ system Loom system information /system

API Standard Response

Every method returns the following standard response:

{ results: { ...the actual result(s) ... }, related: { ...any entities that are referred to by id from within 'results'... }, count: ...the size of 'results' ... , errors: [ ...any errors that occurred in processing the request... ] }

Note that the results will be returned as an array for ‘many’ requests, and as a scalar for ‘individual’ requests (such as when is in URL). When the return value is the unique identifier of an entity, the results will contain a map with a single key, 'entity/id'.

In the documentation of each method, generally only the ‘results’ part of the response structure are described.

Related Section

The ‘related’ section is a map of entity ID’s to properties. The entity ID’s will match property values returned in the ‘results’ section. In that way, results containing properties whose values are entity ID’s can be resolved into a more human-processable form. For example, the ‘entity/createdBy’ value is the unique entity identifier of a user in the system; for display purposes though, usually the user’s actual name is preferred. This can be obtained from the related section using the createdBy value from the results section to look up the key, from which the user’s name can be obtained.

Example:

{ “results”: { "entity/name": "SomeEntity", "entity/description": "The entity description.", "entity/tags": "tag1, tag2, tag3" "entity/folder": "test1/test2", "entity/createdBy": "52e28419-2c48-436d-8e7c-643cf331e071", "entity/modifiedBy": "52e28419-1725-47bd-9884-6149e7b9b446", } "related": { "52e28419-2c48-436d-8e7c-643cf331e071": { "user/username" : "smusial" } "52e28419-1725-47bd-9884-6149e7b9b446": { “user/username” : “bgibson” } } }

Resources and Resource API

This API exposes the Loom registry entities as resources, using standard REST conventions. The Resource API is organized by type (e.g., Source, Dataset, etc), with two generic parts for entity instances and entity types.

For each section, the following information is provided: ● Attributes - orange indicates a domain entity, magenta a struct (subordinate to an entity) ● Requests - summary of the methods available for the resource ● Request Details - calling details for each method

Note that every Entity in Loom automatically has all the core Entity attributes (entity/id, entity/name, etc). See the Model Overview below for details on the core Entity attributes.

Sources

Sources represent sources of data whose lifecycles are not controlled by Loom. Sources are containers of data units that conform to some structural form (tables are currently supported by Loom). Sources are similar in structure to Datasets, but are semantically different, as Datasets are managed by Loom.

Source Entity Attributes

The Source entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description data/structuralForm The structural form of the data contained within the data string container. Currently, the only supported form is “table”. data/structure Default structure for the data units contained in this source, array of embedded struct defined by a schema (or possibly, multiple schemas), of type ‘data/Schema’. Applies to all data units in the source, unless specifically overridden by a data unit. source/expandable Whether the source is an expandable collection or not. boolean source/metadataAccessible Whether the source's metadata can be accessed by Loom. boolean source/dataAccessible Whether the source's data can be accessed by Loom. boolean source/entityState Indicator of the state the entity is in. One of ‘potential’, ‘active’, string or ‘deleted. persist/storage Reference (pointer via entity ID) to a persist/Storage. reference data/dataUnit References (pointers via entity ID) to one or more

array of references data/DataUnits.

DataUnit Attributes

A Source is a data container, represented by the type DataContainer. All data containers own a set of DataUnits. DataUnits represent the actual data (although in most cases, they are merely proxies for the data, and do not physically contain it). DataUnits are first-class entities, with unique identifiers, so they may be referenced from outside the context of the containing data container.

Name [Type] Description data/structuralForm The structural form of the data contained within the data unit. string Currently, the only supported form is “table”. data/structure Structure for the data, defined by a schema (or possibly, array of embedded struct multiple schemas), of type ‘data/Schema’. Overrides the default structure for the containing source, to set the structure on this data unit. persist/storage Reference (pointer via entity ID) to a persist/StorageUnit. reference

Schema Attributes

Data units contain one or more schemas. A Schema is a structure; it is fully-owned by a DataUnit and does not (currently) have a unique identifier

Name [Type] Description data/structuralForm The structural form that the schema represents. string Currently, the only supported form is “table”. data/isDefault If true (or nil if one schema), the schema is the default boolean one for the data unit

The type of the schema depends on the structural form of the data unit. For table data units, with structural form of ‘table’, the schema type is TableSchema. Storage Attributes

A Source is physically persisted to some system (HDFS, database, etc). The Storage entity represents the persistence information. For example, a source may hold its information in a directory of files, or in a Hive database. Storage is a container; it owns a set of StorageUnits. For example, if the Storage is a directory, the storage units are individual files within the directory.

Name [Type] Description persist/storageType The type of storage. E.g., ‘file/text’, ‘file/binary’, ‘rdb/hive’, string ‘rdb/generic’. persist/location Location of storage; used to connect to or otherwise access the string storage. persist/application Application that can process this type of storage. string persist/storageUnit References (pointers via entity IDs) to the storage units for this

array of references storage. persist/format Default format for the storage; applies to all storage units embedded struct unless overridden by a storage unit.

There are additional properties that apply for extensions to the base Storage. For example, a FileSet. StorageUnit Attributes

StorageUnits are proxies for the individual units that hold the data that is exposed from a Source as a DataUnit. For example, a Source represent the actual data (although in most cases, they are merely proxies for the data, and do not physically contain it). DataUnits are first-class entities, with unique identifiers, so they may be referenced from outside the context of the containing data container.

Name [Type] Description persist/location Absolute location of storage unit, if applicable. string persist/relativeLocation Relative location of storage unit in storage. string persist/containsData True if storage unit contains data (will get exposed from source boolean as a DataUnit). persist/format Storage format for storage unit; overrides storage-level format. string persist/formatType Type of storage format. string

There are additional properties that apply for extensions to the base StorageUnit. For example, a FileSetFile. Format Attributes

Formats define how the bits in persistent storage are to be read and parsed. There is a default format nested under a Storage, which applies to all Storage Units in that Storage unless overridden. Each StorageUnit can explicitly define a Format, which takes precedence over the default stored in its Storage container.

Name [Type] Description persist/formatType Type of storage format. E.g., ‘textdelim’ or ‘text/pattern’ for string storage types of ‘file/text’; ‘binary/avro’ for storage types of ‘file/binary’. There is no format for storage types of ‘rdb/*’.

The interesting properties are associated with specific subclasses of Format, e.g., in DelimitedFormat and PatternFormat. Source Summary Attributes

The SourceSummary structure captures a ‘view’ of a Source entity, pulling in information from related entities (such as scan measurements). Instances of these structs are returned from the ‘summary’ API methods.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityDescription The description of the entity that the summary is for. string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string data/structuralForm The structural form of the data contained within the data string container. E.g., “table”. persist/storageType Storage form: file/text, file/binary, rdb/hive, rdb/generic, etc. string persist/location The location of the source; duplicate of Storage location, for string convenience. data/expandable Whether the source is an expandable collection or not, boolean data/autoUpdate Whether the source will be auto-updated as new files, etc are boolean created. source/metadataAccessible Whether the source's metadata can be accessed by Loom. boolean source/dataAccessible Whether the source's data can be accessed by Loom. boolean source/entityState Indicator of the lifecycle state of the source entity. One of string ‘active’ or ‘potential’. summary.source/nDataset Number of datasets derived from the source. long summary.source/nDataUnit Number of data units exposed by the source. long summary.data/dataUnit Relationship to summary items for each data unit array of embedded structs (DataUnitSummary structs).

For a particular source instance, a SourceSummary structure contains one DataUnitSummary for each data unit in the source.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityDescription The description of the entity that the summary is for.

string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string data/structuralForm The structural form of the data contained within the data string container. E.g., “table”. summary.data/nRow Number of rows in the (2-dimensional) data item long summary.data/nCol Number of columns or fields in the data item long summary.data/sizeBytes Size of the data item, in bytes; null if unknown long summary.source/nameInSourc The name of the data entity in its native source (e.g. file path e for files) string

Requests

Request Description GET sources Get all sources matching the provided filters. POST sources Create a new source, given entity metadata and storage information. GET sources/default Get a default source instance, for use when creating a new source. POST sources/default Create a new source given a location, using all default settings. GET sources/summary Get summaries of all sources matching the provided filters. GET sources/ Get the source with the specified entity ID. PATCH sources/ Modify the source with the specified ID with updated attributes and storage information DELETE sources/ Delete the source with the specified entity ID. GET sources//summary Get the summary of the specified source. POST /sources//data_units Add data unit to an existing source.

The ‘default’ methods are convenience functions, to help in defining a source. The ‘GET’ version provides default settings, which can be edited by users, and then saved to Loom using ‘POST /sources’. The ‘post’ version simply creates a source using all defaults, with no user interaction.

Note on setting Data Unit schemas:

There are several possible ways in which Loom will determine the structure (i.e. schema) with which to access data in a data unit contained in a given source. ● The schema may be read directly from the physical storage for that source, e.g. a file header or the Hive metastore.

● The schema may be explicitly set by the user or by the API client. ● The schema may be omitted, and inherited from the default schema set on the source container.

The API methods which result in creation or modification of data unit entities (POST sources, PATCH sources/, POST sources//data_units) all use the following rules to determine the schema that ultimately gets used for a given data unit entity, listed in priority order: 1. If the 'data/structure' property is set on a data unit entity passed through the API and is a non- empty array, then this property is stored unchanged on the resulting data unit entity in the registry. a. No validation is done of such a manually-supplied schema; if it is inconsistent with the underlying data (e.g. missing fields, unknown fields) then this may result in missing data or errors when accessing the underlying data through Loom. 2. If the 'data/structure' property is explicitly set to null on the data unit entity passed through the API, or is an empty array, then no structure is set on the resulting data unit entity stored in the registry; the structure is inherited from the parent source. a. If no default data/structure is set on the source, then an error code is returned and the operation will fail. b. No validation is done of the source default schema with respect to the data unit's data; if it is inconsistent with the underlying data (e.g. missing fields, unknown fields) then this may result in missing data or errors when accessing the underlying data through Loom. 3. If no data unit entity is supplied for a table to be created, or a data unit entity is passed through the API without a 'data/structure' property, and the 'source/metadataAccessible' property is true for the source, then Loom will infer the table structure from the underlying storage. This is the recommended mode of operation when working with Loom's supported storage formats. a. If Loom can directly access metadata about the structure of the data unit through underlying storage format (e.g. an embedded schema in a file header, or a schema managed by another metastore) then this structure is translated into a schema entity and stored as part of the tata unit entity in Loom, and used whenever the data is accessed through Loom. b. If the structure for the underlying data cannot be directly determined by examining the underlying storage, as might be the case for unstructured or semi-structured text formats such as CSV or log files, then a structure is generated for the data unit based on examining a sample of the underlying data, using auto-generated field names. b.i. If a default structure is present on the parent source entity, and matches the generated structure in number and type of fields, then no structure is set on the resulting data unit entity in the registry; the source default schema is used whenever accessing this data unit. b.ii. If the parent Source entity has no default structure, or it has one that does not match the generated structure in number or type of fields, then the generated structure is stored in the data/structure property of the resulting DataUnit entity in the registry, and will override the Source default structure. 4. If no DataUnit entity is supplied for a table to be created, or a DataUnit entity is passed through the API without a 'data/structure' property, and the 'source/metadataAccessible' property is false for the Source, then Loom will not include a structure on the DataUnit entity that is stored in the registry.

Request Details

Paths are all relative to the root of the API, including version number.

Get Registered Sources

Get all sources matching the provided filters.

Path: sources Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_source_storagetype: filter on the source’s storage type; values ‘all’ (default) or one of the valid values of persist/storageType filter_source_entitystate: filter on the source’s entity state; values ‘all’ (default) or one of the valid values of source/entityState filter_ndataunit: filter on the number of data units in the source filter_use_frequency: filter on how often the source has been used to create datasets Returns: array of Source entities See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Register (Create) a Source

Create a new source, given entity metadata and storage information. The values returned from ‘GET /sources/default’ can be edited and passed into this method to create a new source.

Path: sources Method: POST Parameters: entity: a set of core entity properties (e.g., entity/name, entity/description, etc); optional, if name can be derived from the storage location or storage name data_units: an array of DataUnit’s, which define metadata and schemas of data units (tables) corresponding to storage units specified via the storage_units parameter. The correspondence between data units and storage units is through their entity names. Optional - if not specified, all storage units with their source/containsData property set to true will be exposed as tables named the same as the storage units, and the schemas will be derived from the underlying physical persistence if the source/metadataAccessible property is true and they are not explicitly set in the corresponding data unit entity. storage: a Storage instance, which defines the container persistence; required storage_units: an array of StorageUnit’s, which define the unit-level persistence; required - there must be at least 1 storage unit. All storage units with their containsData property set to true will be exposed as tables in the source. Returns: id: ID of created source Notes: The storage units drive the source container contents. Any storage unit with its source/containsData property set to true will be exposed as a table (data unit) in the source. If the data_units section is not specified, or if there is not a data unit in that

section corresponding to a storage unit, then the name of the data unit will be derived from that of the storage unit, and the schema will be inferred from the physical persistence (e.g., a file on disk) if source/metadataAccessible is true. If there is a data unit associated with a storage unit, then that data unit can define the descriptive properties (such as description, tags), and the schema to be used. The data unit is tied to its corresponding storage unit through their names, which must exactly match.

Get a Source Instance with Default Settings

Get a default source instance, for use when creating a new source. The values returned from this method can be edited, and used with ‘POST /sources’ to create a new source in Loom.

Path: sources/default Method: GET Parameters: location: the location of the source (semantics based on storageType); required storage_type: the type of storage; optional, if server can derive it format_type: the type of format to use; optional (not needed for all storage types) Returns: entity: source object (no entity/id) storage: Storage, without embedded StorageUnits storage_units: StorageUnits, separate from Storage

Register (Create) a Source using Default Settings Create a new source given a location, using all default settings. This is a convenience to combine the calls to ‘GET /sources/default’ and ‘POST /sources’ when no modification of the default source is required.

Path: sources/default Method: POST Parameters: location: the location of the source (semantics based on storage_type); required storage_type: the type of storage; optional, if server can derive it format_type: the type of format used to read and parse the source; optional (not needed for all storage types); defaults to ‘text/delim’ if storage type==’file/text’, else null entity: a bundle of core properties (e.g., entity/name, entity/description, etc) to be set on the created Source entity; optional (name can be derived from location) Returns: id: ID of created source

Get Source Summaries

Get summaries of all sources matching the provided filters.

Path: sources/summary Method: GET

Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_source_storagetype: filter on the source’s storage type; values ‘all’ (default) or one of the valid values of persist/storageType filter_source_entitystate: filter on the source’s entity state; values ‘all’ (default) or one of the valid values of source/entityState filter_ndataunit: filter on the number of data units in the source filter_use_frequency: filter on how often the source has been used to create datasets Returns: array of SourceSummary structs See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Get a Source Instance

Get the source with the specified entity ID.

Path: sources/ Method: GET Parameters: none, except for entity/id that is built into the URL Returns: entity: source entity; persist/storage matches Storage’s entity/id storage: Storage; persist/storage references StorageUnits’ entity/id’s storage_units: StorageUnits, separate from Storage

Replace a Source Instance

Replace the source with the specified ID with updated attributes and storage information.

Path: sources/ Method: PATCH Parameters: entity: a set of core entity properties (e.g., entity/name, entity/description, etc); optional, if name can be derived from the storage location or storage name; if entity/id is included (e.g., if using results from GET /sources/), it must match the ID in the URL, or it is considered an error data_units: an array of DataUnit’s, which define metadata and schemas of data units (tables) corresponding to storage units specified via the storage_units parameter. The correspondence between data units and storage units is through their entity names. Optional - if not specified, all storage units with their source/containsData property set to true will be exposed as tables named the same as the storage units, and the schemas will be derived from the underlying physical persistence if the source/metadataAccessible property is true and they are not explicitly set in the corresponding data unit entity. storage: the Storage to replace the existing Storage; optional, if not changing Storage

storage_units: the StorageUnits to replace the existing StorageUnits, en masse; optional, if not changing StorageUnits Returns: id: ID of source Notes: The properties specified under the entity parameter are merged in with the existing properties.

Delete a Source Instance

Delete the source with the specified ID.

Path: sources/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing

Get a Source Instance’s Summary

Get the summary of the specified source.

Path: sources//summary Method: GET Parameters: none, except for entity/id that is built into the URL Returns: SourceSummary struct

Create a DataUnit in an Existing Source

Creates a new data unit (table) in an existing source. This is useful to represent a table that was added to the source through an external system (as opposed to through Loom).

Path: sources//data_units Method: POST Parameters: location: is a relative path to the item in the source (if absolute path provided, will attempt to reconcile with source location, and extract relative path if compatible) format: is a Format object, for interpreting the persisted data pointed to by location. Optional; will use source’s Storage format (the ‘default’ format) if not specified; if specified, must be of the same type as the source’s Storage format. entity: is a set of properties to assign to the created table. Optional, will derive table name from location if not specified. schema: an optional schema to associate with the data unit as the ‘default’ schema for that data unit. This is only allowed if the source that the data unit is being added to has the source/metadataAccessible property set to false. If this parameter is set, then the format parameter is optional, and if specified, will not be used to read metadata from the persistent store. Returns: id: The ID of the data unit

Notes: 1. If the source’s source/metadataAccessible property is set to true, then the location and format will be be used to read the metadata and infer a schema for the associated data unit. If the source’s source/dataAccessible property is set to true, then the new table’s associated storage unit will have its persist/containsData property set to true; otherwise that property will be set to false. 2. A schema can only be specified if the source’s source/metadataAccessible property is false. Otherwise, as noted above, the schema will be inferred from the location and format information. 3. If a schema is specified, the format property is for informational purposes only; it is not used to infer the table metadata. See Also: POST /sources: to create a new source

______

Datasets

Datasets represent sets of data whose lifecycles are controlled by Loom. Datasets originate from Sources, and from other Datasets through transformations. Datasets are containers of data units that conform to some structural form (such as tables). Datasets are similar in this way to Sources, but are semantically different, as Datasets are managed by Loom whereas Sources are controlled by some external entity (and so may be changed without direct involvement of Loom). Dataset Entity Attributes

The Dataset entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description data/structuralForm The structural form of the data contained within the data string container. E.g., “table”. dataset/entityState Indicates the ‘state’ the entity is currently in. One of ‘pending’, string ‘active’, or ‘deleted’. A dataset is ‘pending’ if it is in the process of being created from a long-running transformation. dataset/sourcedFrom The Source identifier, if the dataset was created directly from a reference source Null if the dataset was created through a transformation. persist/storage Reference (pointer via entity ID) to a persist/Storage. reference data/dataUnit References (pointers via entity ID) to data/DataUnits. array of references

DataUnit Attributes

A Dataset is a data container, represented by the type DataContainer. All data containers own a set of DataUnits. DataUnits represent the actual data (although in most cases, they are merely

-14- Loom REST API Document Revision 0.10 - 01 Aug 2014 proxies for the data, and do not physically contain it). DataUnits are first-class entities, with unique identifiers, so they may be referenced from outside the context of the containing data container.

Name [Type] Description data/structuralForm The structural form of the data contained within the data unit. string Currently, the only supported form is “table”. data/structure Structure for the data, defined by a schema (or possibly, array of embedded struct multiple schemas), of type ‘data/Schema’. persist/storage Reference (pointer via entity ID) to a persist/StorageUnit. reference

Schema Attributes

Data units contain one or more schemas. A Schema is a structure; it is fully-owned by a DataUnit and does not (currently) have a unique identifier

Name [Type] Description data/structuralForm The structural form that the schema represents. string Currently, the only supported form is “table”. data/isDefault If true (or nil if one schema), the schema is the default boolean one for the data unit

The type of the schema depends on the structural form of the data unit. For table data units, with structural form of ‘table’, the schema type is TableSchema. Storage Attributes

A Dataset is physically persisted to some system. The Storage entity represents the persistence information. For the most part, storage information for datasets is hidden from Loom users (unlike for Sources).

See the persistence information under Sources.

Dataset Summary Attributes

The DatasetSummary structure captures a ‘view’ of a Dataset entity, pulling in information from related entities (such as scan measurements). Instances of these structs are returned from the ‘summary’ API methods.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityDescription The description of the entity that the summary is for. string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant

summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string data/structuralForm The structural form of the data contained within the data string container. E.g., “table”. data/expandable Whether the dataset is an expandable collection or not, boolean data/autoUpdate Whether the dataset will be auto-updated as its associated boolean source (sourcedFrom) is updated. dataset/entityState Indicator of the lifecycle state of the source entity. One of string ‘active’ or ‘potential’. persist/storageType Storage form: file/text, rdb/hive, etc. string summary.dataset/nDataUnit Number of ‘data units’ (e.g., tables) in the dataset. long summary.dataset/sourcedFrom Name of Source, if dataset directly derived from source. string summary.dataset/nUses Number of times Dataset has been transformed. long summary.dataset/lastProcessedBy Who last used the dataset in a transformation (username). string summary.dataset/lastProcessedAt When the dataset was last used in a transformation. instant summary.data/dataUnit Relationship to summary items for each data unit array of embedded structs (DataUnitSummary structs).

For a particular dataset instance, a DatasetSummary structure contains one DataUnitSummary for each data unit in the dataset.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityDescription The description of the entity that the summary is for. string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string data/structuralForm The structural form of the data contained within the data string container. E.g., “table”. summary.data/nRow Number of rows in the (2-dimensional) data item long

summary.data/nCol Number of columns or fields in the data item long summary.data/sizeBytes Size of the data item, in bytes; null if unknown long summary.data/transformedFro Name of transformation that emitted the table (if from a m Transformation, not ‘from source’). string

Note on setting Data Unit schemas:

Unlike the Source API, where data units can be saved in the registry without a schema and inherit the source default schema at runtime, the Dataset API explicitly attaches a schema to each data unit in datasets saved in the registry. The dataset default schema, if present, will be copied onto each data unit passed to the API that does not have a schema provided by the client. The copying of the default schema onto the data unit prevents subsequent modifications to the default schema from causing problems with processing jobs that were written against a previous version of the schema. If a request is made to store a data unit without a schema, and there is no dataset default schema, then an error will be raised. This applies to all API methods that result in creation or modification of a data unit (POST and PUT). Requests

Request Description GET datasets Get all datasets matching the provided filters. POST datasets Create a new dataset, given a local instance. GET datasets/default Get a default dataset tied to an existing source, for use when creating a new dataset. POST datasets/default Create a new dataset tied to an existing source, using all default settings. GET datasets/summary Get summaries of all datasets matching the provided filters. GET datasets/ Get the dataset with the specified entity ID. PUT datasets/ Replace the dataset with the specified ID with a local instance. DELETE datasets/ Delete the dataset with the specified entity ID. GET datasets//summary Get the summary of the specified dataset. POST datasets//data_units Add a data unit to an existing dataset.

Request Details

Paths are all relative to the root of the API, including version number. Get Registered Datasets

Get all datasets matching the provided filters.

Path: datasets Method: GET

Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_dataset_obtained_from: filter on where the dataset was directly obtained from; values of ‘all’ (default), ‘source’, or ‘dataset’. filter_ndataunit: filter on the number of data units (tables) in datasets; values ‘all’ (default) or one of several discretized facets filter_use_frequency: filter on how often the dataset has been used by processes (in transforms, etc) Returns: array of Dataset entities See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Register (Create) a Dataset

Create a new dataset from a local dataset object. The values returned from ‘GET /datasets/default’ can be edited and passed into this method to create a new dataset.

Path: datasets Method: POST Parameters: local Dataset object Returns: id: ID of created dataset

Get a Dataset Instance with Default Settings

Get a default dataset instance, for use when creating a new dataset. The values returned from this method can be edited, and used with ‘POST /datasets’ to create a new dataset in Loom.

Path: datasets/default Method: GET Parameters: source_id: the ID of a source registered in Loom; required Returns: dataset object (no entity/id)

Register (Create) a Dataset using Default Settings

Create a new dataset tied to an existing source, using all default settings. This is a convenience to combine the calls to ‘GET /datasets/default’ and ‘POST /datasets’ when no modification of the default dataset is required.

Path: datasets/default Method: POST Parameters: source_id: the ID of a source registered in Loom; required

entity: a bundle of core properties (e.g., entity/name, entity/description, etc) to be set on the created Dataset entity; optional (will be obtained from source fields if not present) Returns: id: ID of created dataset

Get Dataset Summaries

Get summaries of all datasets matching the provided filters.

Path: datasets/summary Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_dataset_obtained_from: filter on where the dataset was directly obtained from; values of ‘all’ (default), ‘source’, or ‘dataset’. filter_ndataunit: filter on the number of data units (tables) in datasets; values ‘all’ (default) or one of several discretized facets filter_use_frequency: filter on how often the dataset has been used by processes (in transforms, etc) Returns: array of DatasetSummary structs See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Get a Dataset Instance

Get the dataset with the specified entity ID.

Path: datasets/ Method: GET Parameters: none, except for entity/id that is built into the URL Returns: entity: dataset, with embedded DataUnits

Replace a Dataset Instance

Replace the dataset with the specified ID with the new dataset object.

Path: datasets/ Method: PUT Parameters: the dataset to replace the existing dataset Returns: id: ID of updated dataset

Delete a Dataset Instance

Delete the dataset with the specified ID.

Path: datasets/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing

Get a Dataset Instance’s Summary

Get the summary of the specified dataset.

Path: datasets//summary Method: GET Parameters: none, except for entity/id that is built into the URL Returns: DatasetSummary struct

Create a DataUnit in an Existing Dataset

Creates a new data unit (table) in an existing dataset. There are two overloaded options available, depending on which of the first parameters is passed in. The first creates a data unit in the dataset corresponding to a new data unit in the associated (‘sourcedFrom’) source, if the dataset was created directly from a source. The second is for when a table was added to the dataset through an external process (as opposed to through a Loom process). The latter option should be used with great care as incorrect use can result in datasets which are not processable by Loom.

Path: datasets//data_units Method: POST

These are the parameters for the first overloaded option:

Parameters: source_data_unit_name is the name of a table in the source (pointed to by the dataset’s ‘sourcedFrom’ property) to add to the dataset; optional; if unspecified, then all ‘new’ tables in the source. If this is specified and the dataset does not have a sourcedFrom field set, then an error will be raised. schema: is a Schema struct, to define the structure of the new data unit (table) columns. Required. entity: is a set of properties to assign to the created table. Optional, will derive table name from location if not specified. Returns: id: The ID of the data unit

These are the parameters for the second overloaded option:

Parameters: location: is a relative (or absolute) path to the persisted item, either on disk (hdfs) or in Hive. If a relative path is passed in, it will be used in conjunction with location in the dataset’s Storage to determine the location of the persisted item on disk or in

Hive. If an absolute path is passed in, it will be checked for consistency with the location in the dataset’s Storage, and then used as the location of the persisted item. Required. format: is a Format object, for interpreting the persisted data pointed to by location. Optional; will use dataset’s Storage format (the ‘default’ format) if not specified; if specified, must be of the same type as the dataset’s Storage format. schema: is a Schema struct, to define the structure of the new data unit (table) columns. Required. entity: is a set of properties to assign to the created table. Optional, will derive table name from location if not specified. Returns: id: The ID of the data unit

______

Processes

Processes represent processing performed on Datasets (and, to a lesser degree, on Sources). Process Entity Attributes

The Process entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description process/processType Type of process, corresponding to the ProcessDefintiion. string One of 'sql-query', 'hiveql’, ‘dataset-import', or user-defined type. process/processClass General class of this process. One of ‘transform’, ‘import- string export’, ‘descriptive’, or user-defined class. process/processScope The scope of the process in terms of its data inputs and string outputs. One of ‘container’ or ‘dataunit’. process/isExecutable True if processes conforming to this definition are executable boolean process/entityState Indicator of the state the entity is in. One of ‘active’ or string ‘deleted. process/argument Configuration arguments defining the process configuration reference process/context Optional default data context for input; can be overridden for reference ProcessUse

ProcessUse Attributes

A ProcessUse is a snapshot of a Process at the point in time that process is used. A process that is executable (isExecutable = TRUE) is ‘used’ by virtue of its being executed, through POST /execute/transform. A process that is not executable (isExecutable = FALSE) can still be used,

-21- Loom REST API Document Revision 0.10 - 01 Aug 2014 establishing a link between two or more data containers or data units; that is done via the POST /processes//uses method.

The ProcessUse entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description process/process Reference (pointer via UUID) to the Process that this is a use reference of. process/processClass General class of this process. One of ‘transform’, ‘import- string export’, ‘descriptive’, or user-defined class. process/isExecuted True if processes conforming to this definition was executed. boolean (Corresponds to isExecutable field of corresponding Process). process/jobIdentifer Stringified identifier of Job that was spawned, for informational string purposes. Not a formal UUID-reference, to avoid bi-directional dependency. process/argument Configuration arguments defining the process configuration . reference process/context The data contexts for input and output. reference

Argument/ConfigArgument Attributes

Processes and ProcessUses contain one or more arguments. An Argument is a structure; it is fully-owned by the Process or ProcessUse it is attached to, and does not have a unique identifier. A ConfigArgument is an Argument with a value.

Name [Type] Description entity/name The name of the argument. string process.param/index The index of the argument (1, 2, etc). long process.arg/value The value of the argument, corresponding to the string parameter's valueType

Context Attributes

Processes and ProcessUses contain one or more contexts. A Context is a structure; it is fully- owned by the Process or ProcessUse it is attached to, and does not have a unique identifier.

Name [Type] Description entity/name The name of the context. string process.context/inout The structural form that the schema represents. string Currently, the only supported form is “table”. process.context/container Reference (pointer via UUID) of the data container that reference the context is for. process.context/dataUnitName Name of the data unit in the data container. May be null string (missing) for container-level contexts used with

Processes with processScope = ‘container’.

Contexts associated with Processes (or their ProcessUses) which are container-level (processScope = ‘container’) will not have a dataUnitName set.

Process Summary Attributes

The ProcessSummary structure captures a ‘view’ of a Process entity, pulling in information from related entities (such as scan measurements). Instances of these structs are returned from the ‘summary’ API methods.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string process/processType Type of process, corresponding to the ProcessDefinition. One string of 'sql-query', 'hiveql’, ‘'dataset-import', or user-defined type. process/processClass General class of this process. One of ‘transform’, ‘import- string export’, ‘descriptive’, or user-defined class. process/processScope The scope of the process in terms of its data inputs and string outputs. One of ‘container’ or ‘dataunit’. process/isExecutable True if processes conforming to this definition are executable. boolean summary.process/nUses Number of times transform has been used/executed. long summary.process/lastUse The entity ID of the last process use. reference (uuid) summary.process/lastUsedBy Who last used the process (username). string summary.process/lastUsedAt When the process was last used (e.g., executed). instant

Requests

Request Description GET processes Get all processes matching the provided filters. POST processes Create a new process, given a local instance. GET process/default Get a default process, for use when creating a new process.

GET processes/summary Get summaries of all processes matching the provided filters. GET processes/ Get the process with the specified entity ID. PUT processes/ Replace the process with the specified ID with local instance. DELETE processes/ Delete the process with the specified entity ID. GET processes//summary Get the summary of the specified process. GET processes/uses Get all the ‘uses’ of the specified process POST processes//uses Create a new ‘use’ of the specified process. GET processes//uses/ Get a specific process use for a specific process.

Request Details

Paths are all relative to the root of the API, including version number. Get Registered Processes

Get all processes matching the provided filters.

Path: processes Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_process_class: filter on the process class; values ‘all’ (default) or one of the valid values of process/processClass filter_executable: filter on whether executable or not; values ‘all’ (default), true, false filter_use_frequency: filter on how often the process has been used Returns: array of Process entities See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Register (Create) a Process

Create a new process, given entity metadata and storage information. The values returned from ‘GET /processes/default’ can be edited and then passed into this method to create a new process.

Path: processes Method: POST Parameters: local instance of process Returns: id: ID of created process

Get a Process Instance with Default Settings

Get a default processinstance, for use when creating a new process. The values returned from this method can be edited, and used with ‘POST /processes’ to create a new process in Loom.

Path: processes/default Method: GET Parameters: container_id: the ID of a dataset or source registered in Loom data_unit_name: the name of a data unit in the container Returns: process object (no entity/id)

Get Process Summaries

Get summaries of all processes matching the provided filters.

Path: processes/summary Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_process_class: filter on the process class; values ‘all’ (default) or one of the valid values of process/processClass filter_executable: filter on whether executable or not; values ‘all’ (default), true, false filter_use_frequency: filter on how often the process has been used Returns: array of ProcessSummary structs See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Get a Process Instance

Get the process with the specified entity ID.

Path: processes/ Method: GET Parameters: none, except for entity/id that is built into the URL Returns: entity: process entity

Replace a Process Instance

Replace the process with the specified ID with the new process object.

Path: processes/ Method: PUT Parameters: the process to replace the existing one Returns: id: ID of process (same as on input)

Delete a Process Instance

Delete the process with the specified ID.

Path: processes/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing

Get a Process Instance’s Summary

Get the summary of the specified process.

Path: processes//summary Method: GET Parameters: none, except for entity/id that is built into the URL Returns: ProcessSummary struct

Create a Process Use

Creates a unique ‘use’ of the specified process. A ProcessUse is a snapshot of a Process at the time it was used; this ensures the viability of lineage calculations, as ProcessUses are immutable (whereas Processes are mutable).

Path: processes//uses Method: POST Parameters: contexts: The input and output contexts for this particular use. Returns: id: The ID of the ProcessUse

Get a Process’s Uses

Get the ‘uses’ of the specified process. This returns a set of ProcessUse instances. A ProcessUse is a snapshot of a Process at the time it was used; this ensures the viability of lineage calculations, as ProcessUses are immutable (whereas Processes are mutable).

Path: processes//uses Method: GET Parameters: none, except for entity/id that is built into the URL Returns: array of ProcessUse’s

Get a Process Use

Get a specific process use for a specific process.

Path: processes//uses/ Method: GET Parameters: none, except for entity/id that is built into the URL Returns: ProcessUse

______

Jobs represent asynchronous activity performed in system, and tracked by Loom. Jobs are linked to executable Processes. Job Entity Attributes

Name [Type] Description job/processUse Pointer to the ProcessUse the job is tracking execution of. uuid job/process Pointer to the Process that the ProcessUse was created from. uuid job/status Current status of the job execution. One of created, started, string in-progress, completed, failed, cancelled, timed-out job/executedAt When the job was initiated. Should be equal to or greater than instant the value of entity/createdAt. job/errorMessage Error message, if an error occurs (status == failed). string job/jobTrackerID Hadoop job tracker ID, if available. string job/jobLog Log associated with the job, if available. string job/progress Job progress information. Contains a single JobProgress if embedded struct available.

JobProgress Attributes

A JobProgress is a structure that captures progress information related to a job. It can be embedded in a Job instance, if the Job is completed, or obtained independently, if a Job is still in progress.

Name [Type] Description

job.progress/state Internal state of the processing engine. Corresponds roughly string to the Job’s status field.

job.progress/startedAt When processing began by the processing engine. Should be instant equal to or greater than the Job’s entity/createdAt. job.progress/finishedAt When processing was completed by the processing engine. instant Should be equal to or greater than the value of startedAt. job.progress/duration Duration of the job so far, in milliseconds. duration = long (finishedAt - startedAt). job.progress/cpuTime Total CPU time used so far for the job, in milliseconds. long job.progress/stepMetrics Job progress metrics related to job steps. JobStepMetrics embedded struct struct. job.progress/dataMetrics Job progress metrics related to data read and written. embedded struct JobDataMetrics struct.

JobStepMetrics Attributes

A JobStepMetrics is a structure that captures low-level details about a Job’s execution and progress. It is always embedded in a JobProgress.

Name [Type] Description job.progress.metrics/stepCount Number of steps. long job.progress.metrics/stepsFailed Number of steps that failed. long job.progress.metrics/stepsPending Number of steps that have not run yet. long job.progress.metrics/stepsRunning Number of steps currently running. long job.progress.metrics/stepsSkipped Number of steps skipped. long job.progress.metrics/stepsStarted Number of steps that have been started. long job.progress.metrics/stepsStopped Number of steps that have been stopped. long job.progress.metrics/stepsSubmitted Number of steps submitted. long job.progress.metrics/stepsSuccessful Number of steps completed successfully. long

JobDataMetrics Attributes

A JobDataMetrics is a structure that captures low-level details about a Job’s execution and progress. It is always embedded in a JobProgress.

Name [Type] Description job.progress.metrics/tuplesRead The number of tuples read so far. long job.progress.metrics/tuplesWritten The number of tuples written so far.

Job Summary Attributes

The JobSummary structure captures a ‘view’ of a Job entity, pulling in information from related entities (such as the Process that was executed to spawn the job). Instances of these structs are returned from the ‘summary’ API methods.

Name [Type] Description summary/entityID The unique identifier of the entity that the summary is for. string summary/entityName The name of the entity that the summary is for. string summary/entityCreatedAt The creation timestamp of the entity that the summary is for. instant summary/entityCreatedBy The username of the person who created the entity. string summary/entityModifiedAt The timestamp when the entity was last modified. instant summary/entityModifiedBy The username of the person who last modified the entity. string job/status Current status of the job execution. string job.progress/startedAt When the execution engine started processing the job. instant job.progress/finishedAt When the execution engine finished processing the job. instant job.progress/duration The duration of the job so far; milliseconds. long (Equals finishedAt - startedAt when job completed). process/processType Type of process, corresponding to the ProcessDefinition. One string of 'sql-query', 'hiveql’, ‘'dataset-import', or user-defined type. process/processClass General class of this process. One of ‘transform’, ‘import- string export’, ‘descriptive’, or user-defined class. job/jobTrackerID Hadoop Job Tracker ID, if applicable string job/errorMessage Error message, if an error occurs (status = FAILED) string job/processUse Pointer to the ProcessUse the job is tracking progress of. reference (uuid) summary.job/processID Pointer to the Process that the ProcessUse is a snapshot of. reference (uuid) summary.job/processName Name of the Process (from Process entity/name). string

Requests

Request Description GET jobs Get all jobs matching the provided filters. GET jobs/summary Get summaries of all jobs matching the provided filters. GET jobs/ Get the job with the specified entity ID. PUT jobs/ Replace the job with the specified ID with the new job object. GET jobs//summary Get the summary of the specified job.

Request Details

Paths are all relative to the root of the API, including version number. Get Registered Jobs

Get all jobs matching the provided filters.

Path: jobs Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_job_status: filter on the job status; values ‘all’ (default) or one of the valid values of job/status filter_job_duration: filter on job duration Returns: array of Job entities See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Get Job Summaries

Get summaries of all jobs matching the provided filters.

Path: jobs/summary Method: GET Parameters: filter_recency: filter on when the entity was last modified; values ‘all’ or ‘recent’ (default) filter_job_status: filter on the job status; values ‘all’ (default) or one of the valid values of job/status filter_job_duration: filter on job duration Returns: array of JobSummary structs See Also: GET /search/filters/values: to get the allowed values for a specific filter GET /types/attributes/values: to get the values of enumerated attributes which are used in filters.

Get a Job Instance

Get the job with the specified entity ID.

Path: jobs/ Method: GET Parameters: none, except for entity/id that is built into the URL Returns: entity: job entity

Replace a Job Instance

Replace the job with the specified ID with the new job object. This is used to update a job’s core metadata (name, description, folder, tags); it cannot be used to modify job progress or metrics information.

Path: jobs/ Method: PUT Parameters: the job to replace the existing one Returns: id: ID of job (same as on input)

Get a Job Instance’s Summary

Get the summary of the specified job.

Path: jobs//summary Method: GET Parameters: none, except for entity/id that is built into the URL Returns: JobSummary struct

______

Users represent users (and in the future, groups) who use Loom.

See also the Connection methods for logging into Loom.

User Entity Attributes

The User entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description

user/username User’s unique system user name. string user/password User’s password, in hashed form. string user/email User’s email address. string

Requests

Request Description POST users Create a new user, given a local instance. Returns entity ID. PATCH users/ Modify user’s password and/or email address.

Request Details

New User

Creates a user account, and logs in to that account.

Path: users Method: POST Parameters: body: User object to be saved Returns: id: the entity ID for the created user entity.

Update User

Modifies the specified user account (e.g., password, email, etc).

Path: users/ Method: PATCH Parameters: password: current password, if changing body: attributes from User entity, excluding username Returns: id: the entity ID for the modified user entity.

______

Glossaries

Glossary Entity Attributes

A glossary is a container that holds terms. In addition, terms in a glossary can be organized by ‘subject areas’. So, a glossary is also a container of subject areas.

The Glossary entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description glossary/businessUnit The business unit that owns or manages the glossary. string glossary/steward The name of the steward who maintains and manages the string glossary. glossary/namespace The set of namespaces the glossary imports. Used for RDF namespace/Namespace interoperability. glossary/import The set of namespaces the glossary imports. Used for RDF array of interoperability. namespace/Namespace glossary/subjectArea The subject areas in the glossary. Each subject area is owned array of references by the Glossary. Each term is associated with exactly one of these. glossary/term The terms in a glossary. Each term is ownded by the Glossary, array of references but is considered to be 'contained' within a single subject area.

SubjectArea Attributes

A glossary is a container that holds terms. In addition, terms in a glossary can be organized by ‘subject areas’. So, a glossary is also a container of subject areas. Subject areas are owned by their containing glossary; if the glossary is deleted, the subject areas within it are deleted also.

Note that terms are not ‘contained’ by subject areas; they reference them through a property.

The SubjectArea entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description glossary/steward The name of the steward who maintains and manages the string subject area in the glossary. glossary/namespace The set of namespaces the subject area imports. Used for namespace/Namespace RDF interoperability. glossary/import The set of namespaces the subject area imports. Used for array of RDF interoperability. namespace/Namespace

Term Attributes

A glossary is a container that holds terms. Terms are owned by their containing glossary; if the glossary is deleted, the terms within it are deleted also.

Terms can reference a subject area in the glossary. However, terms are not ‘contained’ by subject areas; they reference them through a property.

The Term entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description glossary/acronym An acronym by which the term may be known. May be the string same as entity name or label. glossary/alternateName Alternate names by which the term may be known. array of string glossary/longDescription A verbose description of the term. (entity/description is treated string as a short description). glossary/subjectAreaRef Reference to a SubjectArea within the glossary, by name. string glossary/termStatusRef Reference to a TermStatus within the glossary, by name. string glossary/term The terms in a glossary. Each term is ownded by the Glossary, array of references but is considered to be 'contained' within a single subject area.

There are sub-types of terms -- SimpleTerms and (in the future) CompositeTerms.

TermStatus Attributes

A glossary term can have a status, indicating some level of ‘governance’ of the term. Typically, term governance will be managed by a glossary steward.

The TermStatus struct has the following attributes, in addition to the core entity attributes such as entity/name, entity/label, and entity/description.

Name [Type] Description glossary/statusIndex Index of the status, for ordering. long

Namespace Attributes

Glossaries and subject areas can be organized by namespaces. This is for the purposes of RDF interoperability.

The Namespace struct has the following attributes, in addition to the core entity attributes such as entity/name, entity/label, and entity/description.

Name [Type] Description namespace/namespaceUri The URI of the namespace for RDF interoperability. string namespace/namespacePrefix The prefix of the namespace for RDF interoperability. boolean

Requests

Request Description GET glossaries Get all glossaries matching the provided filters. POST glossaries Create a new glossary, given glossary metadata. GET glossaries/ Get the glossary with the specified entity ID. PATCH glossaries/ Update the glossary with the specified entity ID. DELETE glossaries/ Delete the glossary with the specified entity ID. GET glossaries//areas Get all glossary subject areas matching the filters. POST glossaries//areas Create a new subject area in a glossary. GET glossaries//areas/ Get the subject area with the specified entity ID. PATCH glossaries//areas/ Update the subject area with the specified entity ID. DELETE glossaries//areas/ Delete the subject area with the specified entity ID. GET glossaries//terms Get all glossary terms matching the filters. POST glossaries//terms Create a new term in a glossary. GET glossaries//terms/ Get the term with the specified entity ID. PATCH glossaries//terms/ Update the term with the specified entity ID. DELETE glossaries//terms/ Delete the term with the specified entity ID.

Request Details

Note this API is ‘flat’ with respect to the containment hierarchy of a glossary. I.e., since glossary terms have unique entity IDs, they can be referenced directly, without referencing the containing subject area. Similarly, subject areas also have unique IDs, and can be referenced directly. (The containing glossary is still included in API methods, to be consistent with other parts of the Loom API, and to allow for retrieval by name in addition to by unique ID).

Also note that while Terms are ‘contained’ within SubjectAreas, in terms of the formal data model, they are serialized as siblings, with the ‘whole-part’ relationship defined indirectly, using the subjectAreaRef property of a Term to point to their associated SubjectArea using the name (or namespace prefix) of the Subject Area.

Register (Create) a Glossary

Create a new glossary with a set of descriptive properties.

Path: glossaries Method: POST Parameters: entity: properties to assign to the new glossary; must include glossary/namespace if the namespace parameter is not specified, and may contain glossary/import if the import parameter is not specified. namespace: the namespace to assign to the glossary entity; optional, if there is a glossary/namespace property defined within the entity that is passed in. imports: the namespaces of other glossaries (or subject areas) to import, so that terms within the glossary can reference terms in other glossaries; optional, if a glossary/import property is defined within the entity that is passed in. Returns: ID of the new glossary entity. Errors: 1. If the glossary name (entity/name) matches that of another glossary in Loom. 2. If the glossary’s display name (entity/label) matches that of another glossary in Loom. (Would be confusing in user interfaces). 3. If the glossary’s namespace prefix or URI match those of another glossary in Loom. (In reality, the prefix does not have to be unique; this is a convenience, or sloppiness, initially.) Notes: 1. The entity parameter is required, and must have at least the entity/name specified. If the namespace parameter is not specified, the entity parameter must include the glossary/namespace property. The entity/type need not be specified. 2. The glossary name (entity/name), display name (entity/label), namespace prefix, and namespace URI must all be unique within the context of the Loom registry. See Also: GET glossaries: to get all defined glossaries

Get Registered Glossaries

Get all glossaries registered with Loom. All contained subject areas are also returned. However, contained terms are not returned. (The focus of this method is to retrieve the ‘organizational scheme’ consisting of glossaries and subject areas).

Path: glossaries Method: GET Parameters: (none) Returns: array of Glossary entities, with contained SubjectArea entities. Notes: 1. When no glossaries are defined, will return empty array. See Also: POST glossaries: to create a new glossary GET glossaries/: to get a specific glossary

Get a Glossary

Get the glossary with the specified entity ID. The returned instance will include all subject areas in the glossary. It will include all terms in the glossary (linked to their respective subject areas) only if requested.

Path: glossaries/ Method: GET Parameters: include_terms: If true, the glossary terms will be included in the response; default is false. Returns: a Glossary entity, with contained SubjectArea entities; contained terms are also included if the include_terms parameter is true. Notes: 1. The ‘related’ section should contain the following sections: ● contentSummary, with counts of the total number of terms (‘nTerm’) and the total number of subject areas (‘nSubjectArea’). ● governanceSummary, an array of structures closely aligned with the actual TermStatus instances (with their name for identification, label for display, description for tooltip, and index for ordering), along with a "count" field with the number of terms in the glossary with that status. See the attached JSON example. The array elements should be ordered in the same order as the glossary/statusIndex values. See Also: POST glossaries: to create a new glossary GET glossaries: to get all glossaries GET glossaries/statuses: to get all term statuses

Update a Glossary

Update the properties of the glossary with the specified ID, with the new or updated attributes specified. This does not change the contents of the glossary: the subject areas and terms.

Path: glossaries/ Method: PATCH Parameters: entity: a set of glossary properties (e.g., entity/name, entity/description, glossary/import, etc), which will replace the existing glossary properties; may include glossary/namespace if the namespace parameter is not specified, and may contain glossary/import if the import parameter is not specified. namespace: the namespace to assign to the glossary entity; optional, if there is a glossary/namespace property defined within the entity that is passed in, or if the namespace is not to be changed. imports: the namespaces of other glossaries (or subject areas) to import, so that terms within the glossary can reference terms in other glossaries; optional, if a glossary/import property is defined within the entity that is passed in, or if the imports are not to be changed. Returns: ID of the updated glossary entity. Errors: 1. If the glossary’s display name (entity/label) matches that of another glossary in Loom. (Would be confusing in user interfaces). 2. If the glossary’s namespace URI matches that of another glossary in Loom. Notes: 1. Changing the entity name or the namespace prefix is not allowed. See Also: POST glossaries to create a new glossary instance GET glossaries/ to get a specific glossary instance GET glossaries: to get all defined glossaries

Delete a Glossary

Delete the glossary with the specified ID. This will delete all the subject areas and all the terms in the glossary.

Path: glossaries/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing

______

Add a Subject Area to a Glossary

Create a new subject area within a glossary. The subject area will be owned by the glossary; if the glossary is deleted, the subject area will be deleted also.

Path: glossaries//areas Method: POST Parameters: entity: properties to assign to the new subject area; must include glossary/namespace if the namespace parameter is not specified, and may contain glossary/import if the import parameter is not specified. namespace: the namespace to assign to the subject area entity; optional, if there is a glossary/namespace property defined within the entity that is passed in. imports: the namespaces of other glossaries (or subject areas) to import, so that terms associated with the subject area can reference terms in other glossaries; optional, if a glossary/import property is defined within the entity that is passed in. Returns: ID of the new SubjectArea entity. Errors: 1. If the subject area name (entity/name) matches that of another subject area in the glossary. 2. If the subject area’s display name (entity/label) matches that of another subject area in the glossary. (Would be confusing in user interfaces). 3. If the subject area’s namespace prefix or URI match those of another subject area in the the glossary. Notes: 1. The entity parameter is required, and must have at least the entity/name specified. If the namespace parameter is not specified, the entity parameter must include the glossary/namespace property. The entity/type need not be specified. 2. Subject areas are owned by their containing glossary; if the glossary is deleted, the subject area will be deleted also. 3. The subject area name (entity/name), display name (entity/label), namespace prefix, and namespace URI must all be unique within the context of the glossary. See Also: GET glossaries/areas: to get all subject areas in the glossary

Get all Subject Areas in a Glossary

Get all subject areas within a specified glossary. The terms associated with each subject area are not returned by this method.

Path: glossaries//areas Method: GET Parameters: (none) Returns: an array of SubjectArea entities. Notes: 1. When no subject areas are defined, will return empty array. See Also: POST glossaries//areas: to add a new subject area to a glossary GET glossaries//areas/: to get a specific subject area

Get a Subject Area and its Terms

Get a specific subject area from the glossary. The returned instance will include the subject area and all its terms.

Path: glossaries//areas/ Method: GET Parameters: (none) Returns: a SubjectArea entity under “subjectArea”, and an array of Terms under “terms”. Errors: 1. If the ID does not match the entity/id of a SubjectArea in the glossary. Notes: 1. When a subject area that doesn’t have any terms (such as one newly created) is retrieved, there should be no terms returned in the results. See Also: POST glossaries//areas: to add a new subject area to a glossary GET glossaries//areas: to get a all the subject areas in the glossary

Update a Subject Area in a Glossary

Update a glossary subject area with the specified ID, with the new or updated attributes specified. This does not change the terms associated with the subject area.

This method cannot be used to rename the subject area, or to change its namespace. (Renaming will not be supported in Loom 2.2, although changing the display name will be permitted).

Path: glossaries//areas/ Method: PATCH Parameters: entity: a set of subject area properties (e.g., entity/name, entity/description, glossary/import, etc), which will replace the existing subject area properties; may include glossary/namespace if the namespace parameter is not specified, and may contain glossary/import if the import parameter is not specified.

namespace: the namespace to assign to the subject area entity; optional, if there is a glossary/namespace property defined within the entity that is passed in, or if the namespace is not to be changed. imports: the namespaces of other glossaries (or subject areas) to import, so that terms within the subject area can reference terms in other glossaries; optional, if a glossary/import property is defined within the entity that is passed in, or if the imports are not to be changed. Returns: ID of the updated subject area entity. Errors: 1. If the subject area’s display name (entity/label) matches that of another subject area in the glossary. (Would be confusing in user interfaces). 2. If the subject area’s namespace URI matches that of another subject area in the the glossary. Notes: The name and namespace prefix cannot be modified by this method. That is because Terms reference their associated subject areas using these properties.

Delete a Subject Area from a Glossary

Delete the subject area with the specified ID from the glossary. This will also delete all the terms associated with the subject area from the glossary.

Path: glossaries//areas/ Method: DELETE Parameters: term_disposition: Indicates how to deal with terms that are associated with the subject area being deleted. Options are ‘delete’, which will delete the terms; or ‘leave’, which will leave the terms, but set their subject area to null. Returns: nothing Errors: 1. Calling this for an entity that does not exist will yield an error.

______

Add a Term to a Glossary

Create a new term in a glossary. Typically, a term will be associated with a subject area in the glossary. The term will be owned by the glossary; if the glossary is deleted, the term will be deleted also. The term is tightly tied to, but not owned by, the associated subject area if one is defined; if the associated subject area is deleted, it is optional to also delete all the associated terms.

Path: glossaries//terms Method: POST Parameters: entity: Properties to assign to the new term. E.g., entity/name (required), entity/label, entity/description, etc. May contain a glossary/subjectAreaRef property, if the area_ref parameter is not specified, whose value must match the name of a subject area in the glossary. May contain a glossary/termStatusRef property, if the

status_name parameter is not specified, whose value must match the name of a termStatus in Loom. May contain glossary/property array, if the composite_properties parameter is not specified, which denotes the term is a composite, and specifies the properties to embed in the composite (may be null if no properties specified for a composite upon create). composite_properties: An array of business properties for a composite term. Optional; if the glossary/property property is specified or if the term is a simple term. It is acceptable that this parameter (or the glossary/property property) is specified without a value, indicating the term is composite but that it does not currently have any properties. area_ref: The name of the subject area the term will be in. Optional, if the glossary/subjectAreaRef property is specified as part of the entity parameter. status_ref: The name of the term status that will be assigned to the term. Optional, if the glossary/termStatusRef property is specified as part of the entity parameter. Returns: ID of the new Term entity. Errors: 1. If the combination of the term name (entity/name) and subject area matches that of another term in the glossary (i.e., same name in same or ‘default’ subject area). 2. If the term display name (entity/label) matches that of another term in the glossary. (That would be confusing for UI displays). 3. If the glossary/termStatusRef has a value that does not match the name (entity/name) of a valid TermStatus in the registry. 4. If the area_ref or the glossary/subjectAreaRef property has a value that does not match the name of a SubjectArea in the glossary. 5. If an attempt is made to create a new term without a valid technical name (entity/name). 6. If an attempt is made to add a term with the combination of technical name and subject area as another term in the glossary. Notes: 1. If not explicitly specified using entity/type, the type is inferred from the presence or absence of the ‘glossary/property’ property or ‘composite_properties’ parameter (either of which may have value ‘null’ (or missing for parameter), which indicates the term is a composite with no properties defined (currently)). 2. Terms are owned by their containing glossary; if the glossary is deleted, the terms will be deleted also. 3. Terms are not owned by their associated subject areas; if the subject area is deleted, it is optional whether or not to delete the associated terms. 4. The entity parameter is required, and must have at least the entity/name specified. The entity/type need not be specified. 5. If the display name (entity/label) is not specified upon input, then the entity name should be copied into the entity/label field for use as the display name. 6. Term names within a given SubjectArea must be unique. Term names in different SubjectAreas can be the same, as their qualified names (namespace:name) will be unique. 7. Term display names (entity/label) must be unique within the entire glossary. This is more conservative than the restriction on technical names (which must be unique

within a subject area), for usability reasons (seeing different terms named the same thing in a glossary might be confusing). 8. It is permissible to add a term to the glossary that has the same technical name as another term, as long as its subject area is different. See Also: GET glossaries/terms: to get all terms in the glossary

Get all Terms in a Glossary Matching some Constraints

Get all terms within a specified glossary that match some input constraints.

Path: glossaries//terms Method: GET Parameters: area_ref: The subject area to restrict the terms to. If not specified, terms matching all subject areas are returned. If the special name ‘default’ is specified, then only terms associated with the glossary directly (no subject area references) are returned. status_ref: The status to restrict the terms to. If not specified, then terms with any status are returned. Returns: an array of Term entities matching the specified input constraints. Notes: 1. When no terms are defined, or when no defined terms match the input constraints, this method will return an empty array.

See Also: POST glossaries//terms: to add a new term to a glossary GET glossaries//terms/: to get a specific term

Get a Term from the Glossary

Get a specific term from the glossary.

There are multiple possible variations of this method, depending on how the get a term using its Path: glossaries//terms/ Method: GET Parameters: (none) Returns: a Term entity Errors: 1. If the ID does not match the entity/id of a term in the glossary. See Also: POST glossaries//terms: to add a new term to a glossary GET glossaries//terms: to get a set of terms

Update a Term in a Glossary

Update a specific term in the glossary, with the new or updated attributes specified.

This method can be used to change the subject area that a term is associated with. However, it cannot be used to rename the term. (Renaming will not be supported in Loom 2.2, although changing the display name will be permitted).

Path: glossaries//terms/ Method: PATCH Parameters: entity: A set of term properties (e.g., entity/name, entity/label, entity/description, etc), which will replace the current set of properties composite_properties: An array of properties for a composite term, which will replace the current set of composite properties. Optional; if the glossary/property property ios specified or if the term is a simple term. It is acceptable that this parameter (or the glossary/property property) is specified without a value, indicating the term is composite but that it does not currently have any properties. area_ref: The name of a different subject area the term will be in. Optional, if the glossary/subjectAreaRef property is specified as part of the entity parameter, or if the term will remain tied to the original subject area. status_ref: The name of the term status that will be assigned to the term. Optional, if the glossary/termStatusRef property is specified as part of the entity parameter, or if the term status will remain unchanged. Returns: ID of the updated term entity. Errors: 1. Attempting to change the name of the term will yield an error. 2. Attempting to update a term with a area_ref or glossary/subjectAreaRef that does not match the name of a subject area in the glossary will yield an error. Notes: 1. The term’s name cannot be modified by this method. That is because CompositeTerms reference their properties via qnames, formed with the SubjectArea prefix and Term name. So changing term names would break those property bindings. 2. The subject area that a term is associated with can be changed with this method by specifying a new (valid) value for the glossary/subjectAreaRef property of the entity that is passed in. 3. Updating a term without the area_ref or glossary/subjectAreaRef specified will result in the term having the same subject area association it had before the call.

Delete a Term from a Glossary

Delete the glossary term with the specified ID from the glossary.

Path: glossaries//terms/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing Error: 1. Calling this for an entity that does not exist should yield an error.

______

Relationships

Relationship Entity Attributes

A relationship is an entity that can relate any two entities. Relationships have a reference to a RelationshipType, which defines the constraints for relationships.

The Relationship entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description relate/relationshipType The relationship type, defining constraints on the relationship. reference to RelationshipType relate/role1 The first role (of two) defining one of the ends of the Role struct relationship. relate/role2 The second role (of two) defining one of the ends of the Role struct relationship.

RelationshipType Attributes

RelationshipTypes define the constraints for relationships.

The RelationshipType entity has the following attributes, in addition to the core entity attributes.

Name [Type] Description relate/acronym An acronym by which the type might be known. string relate/alternateName A set of alternate names by which the relationship type can be array of string known. relate/role1 The first role type (of two) defining constraints on one of the RoleType struct ends of relationship instances. relate/role2 The second role type (of two) defining constraints on one of the RoleType struct ends of relationship instances.

Role Attributes

Roles define the ends of a relationship. A role has an implied reference to a RoleType; i.e., the Role at ‘end 1’ is associated with the RoleType at ‘end 1’ of the associated relationship type; similarly for the role at ‘end 2’.

The Role struct has the following attributes, in addition to the core entity attributes such as entity/name, entity/label, and entity/description.

Name [Type] Description relate/entity The entity that the role is bound to. reference

RoleType Attributes

Role types define the constraints on the ends of relationships that reference the relationship ttype to which the role type is part of.

The RoleType struct has the following attributes, in addition to the core entity attributes such as entity/name, entity/label, and entity/description.

Name [Type] Description relate/isNavigable Whether the end of the relationship denoted by the role boolean is navigable or not. relate/constrainedType Constraint on type that can fill a role in a relationship string instance; null for 'any'.

Requests

Request Description GET relationships Get all relationships matching the provided filters. POST relationships Create a new relationship. GET relationships/ Get the relationship with the specified entity ID. PATCH relationships/ Update the relationship with the specified entity ID. DELETE relationships/ Delete the relationship with the specified entity ID. GET relationships/types Get all relationship types matching the provided filters.

Request Details

Create a Relationship between Two Entities

Create a new relationship between two entities, optionally overriding the default relationship name and role names

Path: relationships Method: POST

Parameters: rel_type_id: the entity/id of a relationship type (relate/RelationshipType) registered with the system; optional - if not specified, the generic 'relatedEntity' relationship will be used related1_id: the entity/id of the first entity to relate (required) related2_id: the entity/id of the second entity to relate (required) rel_entity: a bucket of properties (entity/name, entity/label, entity/description, etc) to put on the relationship entity itself; optional - if not specified, the relationship properties will be set from the associated relationship type role1: a bucket of properties (entity/name, entity/label, entity/description, etc) describing the role of the first entity in the relationship; optional - if not specified, the role properties will be set from the associated role type role2: a bucket of properties (entity/name, entity/label, entity/description, etc) describing the role of the second entity in the relationship; optional - if not specified, the role properties will be set from the associated role type Returns: ID of the new Relationship entity. Errors: 1. If exactly two valid entity IDs are not passed in. 2. If the relationship type is specified, but is invalid. 3. If the specified entities at the ends of the relationship do not correspond to the type constraints defined bty their respective role types (relate/constrainedType). Notes: 1. If the relationship type is not specified, the generic 'RelatedEntity' relationship will be used. 2. If the relationship entity properties are not specified, the corresponding properties from the relationship type will be used. 3. If the role for an end is not specified, the properties from the associated RoleType (from the RelationshipType) will be used. 4. The type of an entity at a relationship end (related1_id, related2_id), must be compatible (accounting for inheritance) with the type specified by the relate/constrainedType field of the corresponding role type. 5. If a label is specified for any of the Relationship or Roles, but no name, then the label will be ‘sanitized’ and used to generate the name. See Also: GET relationships: to get all relationships, given some constraints

Get all Relationships Matching some Constraints

Get all relationships, optionally constrained by relationship type or name, or additionally by entity ID (at either end) or related entity type (at either end if no entity ID specified, else at the opposite end)

Path: relationships Method: GET Parameters: rel_type_id: the entity/id of a relationship type (relate/RelationshipType) to constrain the relationships returned to; optional . rel_name: a name to restrict the relationships returned to; optional. Can be used independent of or in conjunction with rel_type_id. related_id: the entity/id of an entity at one end of the relationship (can be at either end); optional.

related_type: the type of entities at one end of the relationship (can be either end, if no related entity ID is specified, otherwise at the opposite end); optional. If not specified, there is no constraint on the type of the related entities. Can use the special values ‘all-technical’, ‘all-glossary’, and ‘all’ (which is the same as ‘core/Entity’) Returns: an array of Relationship entities. Errors: 1. If the specified relationship type ID is invalid. 2. If the specified related ID is invalid. Notes: 1. The relationship name may be more restrictive than the relationship type (specified via the rel_type_id). That is because a relationship can be named independently of the relationship type’s name. Or, in the case where the relationships were given the same name as the relationship types, then specifying this is just another way to identify the relationship type (effectively). 2. There is no restriction on what a relationship can be named. The same name can be used for multiple instances of the same type, or for instances of different relationship types. 3. The related type constraint should take inheritance into account. E.g., specifying data/DataContainer should return all instances that have that type, or source/Source, or dataset/Dataset. See Also: POST relationships: to create a new relationship GET relationships/: to get a relationship

Get a Relationship

Get a relationship given its unique identifier.

Path: relationships/ Method: GET Parameters: none (except for the ID in the URL) Returns: a Relationship entity Errors: 1. If the specified relationship ID is invalid. See Also: POST relationships: to create a new relationship

Update a Relationship

Update a relationship. This performs a partial update, not a full replace.

Path: relationships/ Method: PATCH Parameters: rel_type_id: the entity/id of a relationship type (relate/RelationshipType) registered with the system; optional - if not specified, the existing relationship type is left unchanged related1_id: the entity/id of the first entity to relate; optional - if not specified, the existing entity on ‘role 1’ is left unchanged

related2_id: the entity/id of the second entity to relate; optional - if not specified, the existing entity on ‘role 1’ is left unchanged rel_entity: a bucket of properties (entity/name, entity/label, entity/description, etc) to put on the relationship entity itself; optional - if specified, the existing relationship properties will be fully replaced with the new values; if not specified, the relationship properties will be left unchanged role1: a bucket of properties (entity/name, entity/label, entity/description, etc) describing the role of the first entity in the relationship; optional - if specified, the existing role properties will be fully replaced with the new values; if not specified, the role properties will be left unchanged role2: a bucket of properties (entity/name, entity/label, entity/description, etc) describing the role of the second entity in the relationship; optional - if specified, the existing role properties will be fully replaced with the new values; if not specified, the role properties will be left unchanged Returns: ID of the updated Relationship entity. Errors: 1. If an invalid entity ID is passed in. 2. If the relationship type is specified, but is invalid. Notes: 1. For any of the parameters that is specified, their contents will fully replace the corresponding values of the current relationship. 2. If a new relationship type is specified, but one or more of the rel_entity and role1 and role2 are not specified, then the existing values of those will be replaced with the ‘default’ values, obtained from the new relationship type. So, the relationship name, label, description will be replaced with the corresponding values from the new relationship type. Similarly for the role name, etc being replaced with the corresponding values from the role types (for each role). 3. If a entity is specified for one or both ends of the relationship (related1_id and/or related2_id), the type of the entity must be compatible (accounting for inheritance) with the type specified by the relate/constrainedType field of the corresponding role type. (I.e., related1 must be compatible with roleType1, and relalted2 must be compatible with roleType2). This applies whether a new relationship type is specified or not. Error: 1. If the specified entities at the ends of the relationship do not correspond to the type constraints defined by their respective role types (relate/constrainedType). See Also: POST relationships: to create a new relationship

Delete a Relationship

Delete a relationship given its unique identifier.

Path: relationships/ Method: DELETE Parameters: none (except for the ID in the URL) Returns: nothing Errors: 1. If the specified relationship ID is invalid.

See Also: POST relationships: to create a new relationship

______

Get Relationship Types

There are a fixed set of ‘global’ RelationshipType instances in the Loom registry. This method retrieves all the RelationshipType instances that are registered, optionally constrained by those types that allow for a specific type of entity to be related.

Path: relationships/types Method: GET Parameters: related_type: the type of entities allowed at one end of the relationship (can be either end); optional - if not specified, all relationship types are returned. Returns: array of RelationshipType entities. Notes: 1. This method will return all the RelationshipType instances registered if no parameters are specified.

Entities

This is a ‘generic’ API for accessing instance-related information about the Loom registry model. Requests

There is currently only one ‘polymorphic’ (type-agnostic) method available. In the future, there will be more methods for interacting with entity instances without declaring the type.

Request Description DELETE entities/ Delete the entity with the specified entity ID. DELETE entities Delete all entities in the specified folder; optionally recursive.

Delete an entity

Delete the entity with the specified ID. This performs type-specific delete processing where appropriate (i.e., delegates to DELETE /sources, etc).

Path: entities/ Method: DELETE Parameters: none, except for entity/id that is built into the URL Returns: nothing

Delete all entities in a folder

Deletes all entities ‘in’ the specified folder. Optionally recurses down the virtual folder hierarchy.

Path: entities Method: DELETE Parameters: folder: The starting folder; use a single slash (‘/’) or an empty string (“”) for the ‘root’ folder. recurse: Indicates whether to delete recursively or not (default false). Returns: nothing

______

This is a ‘generic’ API for accessing type-related information about the Loom registry model.

Requests

Request Description GET types/extensions Get extension attributes for specified entity type. GET types/attributes/values Get allowed values for the specified attribute.

Request Details

Type Extensions

Returns all the extension attributes for a data type. Results are an array of attribute definitions.

Path: types/extensions Method: GET Parameters: type: The entity type to retrieve extension attributes for. Returns: An array of attribute definitions associated with the entity type; e.g., [{"meta.attribute.ref/type":"dataset/Dataset", "meta.attribute/doc": "Related To", "meta.attribute/cardinality":"many", "meta.attribute/valueType":"uuid", "meta.attribute/name":"dataset.extension/relatedTo"}, {"meta.attribute/doc":"Source", "meta.attribute/fulltext":"true", "meta.attribute/cardinality":"one", "meta.attribute/valueType":"string", "meta.attribute/name":"dataset.extension/source"}] Attribute Allowed Values

Returns the allowed values for a specific attribute. This assumes the attribute holds enumerated values.

Path: types/attributes/values Method: GET Parameters: entity: The groups of attributes of interest, based on an entity type. This parameter is ignored if attribute is set.Must be a single value or array of: source/Source, process/Process or data.table/Column attribute: Defines the attributes of interest. Must be a single value or array of: persist/storageType, data/structuralForm, process/processType, data.table/dataType

Returns: Map of attribute-name to [value,label] pairs; E.g., {"data/structuralForm": [["table","Table"]],"persist/storageType":[["file/text","Text Files"],["rdb/generic","Relational Database"]]}

______

Activity API

The Activity API focuses on general user activities, such as executing transforms, managing sessions, etc. These are independent of metadata-manipulation operations performed through the Resource API.

Connection Operations

These operations are related to establishing, checking, and terminating connections to Loom.

Note that in Loom 1.0, connection establishes a cookie so that entity creation and modification events are tied to a specific user (via the entity/createdBy and entity/updatedBy fields). Loom does not have formal secure authentication in 1.0.

User Entity Attributes

The connection operations deal with users, and as such use User entity information. See the Users section above for attribute values.

Requests

Request Description POST connect/login Log user in to Loom. POST connect/logout Log user out of Loom. GET connect/ping Checks if the connection is still valid. If it is, returns information for the user associated with the session.

Request Details

Provide user credentials to get a session cookie for that user. Both the username and password parameters are strings. The password is sent in the clear, and will rely on the transport layer for security.

Path: connect/login Method: POST Parameters: username, password Returns: entity/id: The entity ID of the user who logged in. Headers: Set-Cookie with session-id set.

Logout

Releases the user’s connection. This resets the user’s session cookie.

Path: connect/logout Method: POST Parameters: none - retrieves current user from the session-id in cookies. Returns: empty

Checks if the caller is currently connected to Loom, and if so, returns the entity containing all of the user data for the currently logged in user. The current user is determined from the session information (see /connect/login).

Path: connect/ping Method: GET Parameters: session cookie in headers Returns: User entity.

______

Search Operations

Related to search - getting filters, defining filters, etc. Requests

Request Description GET search/filters/values Get the values for specific named filters, used for the multi- value ‘GET’ methods in the Resource API. POST search/text Search through text fields for matching entities. POST search/text/glossaries Search through all glossaries or a single glossary for term matching the search value. (New in 2.2) GET search/folders Get all entities under the specified folder. GET search/lineage/data Get lineage for a specific dataset, source, or data unit. GET search/lineage/process Get lineage starting from a specific process or process use. GET search/related/entities Get entities of some type that are traversable in a specified direction from a single instance. GET search/related/processes Get all the processes and process uses that a specific data unit has been used in. GET search/related/dataflow Get all the ‘dataflow segments’ that a specified entity is part of.

GET search/totals Get the total number of entities of each of the main types exposed by Loom: source/Source, dataset/Dataset, process/Process, job/Job

Request Details

Paths are all relative to the root of the API, including version number. Get Filter Values for a Filter

Get the values for specific named filters, used for the multi-value ‘GET’ methods in the Resource API.

Path: search/filters/values Method: GET Parameters: context: The name of a filter grouping (e.g., ‘dataset’, ‘source’, etc). filter: The name of a filter (e.g., ‘filter_recency’) Returns: Map of filter key to filter value. E.g., { : [ ["", ""], ... ] ... } The filter key is passed into Loom API methods, whereas the value is used for display purposes. See Also: GET /types/attributes/values: To get the values for a specific attribute. Filters often are comprised of the values of an enumerated attribute, plus the value ‘all’ (and sometimes ‘none’).

Perform Full-text Search over all Entities

Search through text fields for matching entities. The following core entity fields can be included in these searches: entity/name, entity/description, entity/label, entity/tags. In addition, some domain model attributes (those with the meta-attribute of meta.attribute/fulltext=true) can be included in these searches.

Path: search/text Method: POST Parameters: types: entity types to return if their properties, or the properties of a related type, is matched. One of ‘all’, ‘technical’, ‘glossary’, or a list of specific types. properties: properties to search over (all of the specified properties may not be on all the instances being matched against; just ignore the ones that are missing for a given type). By default, all of the 'core' properties for an entity are included: entity/name, entity/description, entity/label, entity/tags match_terms: list of search words or phrases, to be matched individually and AND’d together match_type: whether exact or wildcard (default wildcard); one of "exact" or "wildcard", with the latter being default; wildcard search can also be baked into search terms with wildcard (‘*’) character

related_types: entity types of related entities to search over; if not specified, relationships will not be followed. One of ‘all’, ‘technical’, ‘glossary’, or a list of specific types. _offset: Provides an offset into the results. _limit: Specifies how many results should be returned, starting at the offset position. Returns: An array of SearchResults, each containing the following information: ● returnEntity: the entity that was returned as a result of the match; must be compatible with the ‘types’ parameter ○ entity/id ○ entity/name ○ entity/label ○ entity/type ○ entity/modifiedAt ○ entity/modifiedBy ○ containerEntity ■ entity/id ■ entity/name ■ entity/label ■ entity/type Notes: If multiple properties are specified, then a match will occur if the match term is found in any of those properties, on the target types (types parameter) or on any related types (if related_types parameter is specified).

Get Entities in Folder

Lists all entities of a given type that appears within a folder. This recursively includes all sub-folders under the specified one. If a folder is not specified, then the root folder is presumed. Note that the type parameter is unnamed in the request, and appears at the end of the path.

Path: search/folders Method: GET Parameters: type: The type of entities to get; ‘all’ for all folder: The folder to look in. ‘/’ for the root folder. recurse: If true (default), recurse through the folder hierarchy Returns: entities: An array of entities of the requested type, in the folder or one of it’s sub-folders. folders: A sorted array of the folders that the objects appear in.

Get Lineage starting from a Data Container or Unit

Get lineage starting from a specific dataset, source, or data unit within a dataset or source.

Path: search/lineage/data (previous method search/lineage is deprecated) Method: GET

Parameters: container_id: The ID of the dataset or source; if specified and no dataunit information is provided, than container-level lineage will be computed, otherwise data unit-level lineage will be computed data_unit_name: The name of the data unit within the container (container_id must be specified); if specified, then data unit-level lineage will be computed data_unit_id: The ID of the data unit (container_id is optional, but must be consistent if specified); if specified, then data unit-level lineage will be computed direction - specifies direction. Optional. If excluded then both directions are given. format: The format the lineage will be returned in. One of ‘graph’ (default) or ‘nested’. The ‘graph’ format returns a node-link structure with all node and link details in the ‘related’ section of the response. The ‘nested’ format returns a deeply nested structure. direction: The direction to compute lineage for; one of ‘up' (for upstream), 'down' (for downstream), or 'both' (default). steps_max: the maximum number of steps taken in the graph traversal; default is ‘unlimited’ Returns: ‘graph’ option: nodes: an array of the nodes in the graph, representing entities that are connected via the links; the ‘node’ key contains the entity ID ■ node node is the UUID of the data entity (dataset, source, or data unit) ■ node label is the name of the node ■ node type is one of ‘data’ or ‘process’ ■ node group is the UUID of the container for data nodes, or of the process for process (use) nodes ■ node state is the entity state links: an array of links, with the ‘source’ and ‘target’ keys containing the IDs of the entities at the start and end of each link. ■ link source is the UUID of the node the directed link comes from ■ link target is the UUID of the node the directed link goes to ■ link label is a string of the form ‘source_type->target_type’ ■ link type is ‘lineage’ Returns: ‘nested’ option: up: the lineage upstream (‘backward’ direction) of the starting entity down: the lineage downstream (‘forward’ direction) of the starting entity Notes: The combinations possible on input are: ■ container_id only - container-level lineage ■ data_unit_id only - data unit-level lineage ■ container_id and data_unit_name - server looks up data_unit_id, computes data unit-level lineage ■ container_id and data_unit_id - same as previous, but data unit must be in the container or it is an error Notes: Example return structure for ‘graph’ option (with details keyed off UUIDs in the ‘related’ part of the response):

{"nodes": [ {"node": , "label": , "type": , "group": } , ... ],

"links": [ {"source": , "target": , "label": , "type": , ... ]}

Get Lineage starting from a Process

Get lineage starting from a specific process or process use.

This method always returns lineage in the ‘graph’ format. (The ‘data’ lineage has an option to do ‘nested’ also, but that does not make as much sense when starting from a process.)

Path: search/lineage/process Method: GET Parameters: process_id: The ID of the process or process use (required) granularity: The granularity at which lineage will be computed. One of ‘container’ or ‘data_unit’ (optional; default is container if not specified). steps_max: the maximum number of steps taken in the graph traversal; default is ‘unlimited’ Returns: nodes: an array of the nodes in the graph, representing entities that are connected via the links; the ‘node’ key contains the entity ID ■ node node is the UUID of the data entity (dataset, source, or data unit) ■ node label is the name of the node ■ node type is one of ‘data’ or ‘process’ ■ node group is the UUID of the container for data nodes, or of the process for process (use) nodes ■ node state is the entity state links: an array of links, with the ‘source’ and ‘target’ keys containing the IDs of the entities at the start and end of each link. ■ link source is the UUID of the node the directed link comes from ■ link target is the UUID of the node the directed link goes to ■ link label is a string of the form ‘source_type->target_type’ ■ link type is ‘lineage’ Notes: Example return structure for ‘graph’ option (with details keyed off UUIDs in the ‘related’ part of the response):

{"nodes": [ {"node": , "label": , "type": , "group": } , ... ], "links": [ {"source": , "target": , "label": , "type": , ... ]}

Get Related Entities

Get entities of some type that are traversable in a specified direction from a single instance.

Path: search/related/entities Method: GET Parameters: entity_id: the ID of the starting entity; currently, this must be of type source/Source related_type - the type of the target entities; currently, this must be of type dataset/Dataset steps_max: is the maximum number of steps taken in the graph traversal; default 1 Returns: related_entities: array of all entities of the specified type that are related to the starting entity steps_taken: the number of steps taken in the graph traversal; will be less than or equal to steps_max

Get Processes

Get Processes and Process Uses that have a Dataset or DataUnit in their input or output contexts. NOTE: This method will be merged into /search/related/entities in an upcoming release.

Path: search/related/processes Method: GET Parameters: dataset_id: The ID of the dataset; if specified and no dataunit information is provided, then all processes involving the dataset will be returned. data_unit_name: The name of the data unit within the dataset (dataset_id must be specified); if specified, then only processes using the specific data unit will be returned. data_unit_id: The ID of the data unit (dataset_id is optional, but must be consistent if specified); if specified, then only processes using the specific data unit will be returned. direction: The direction; either ‘in’ (for input to process) or ‘out’ (for output from process) or 'both' (default) class: The class of processes to return; one of either ‘process’, ‘process_use’, or ‘all’ (with ‘all’ being the default if not specified). Returns: in: array of Processes and ProcessUses that the dataset / data unit are input to out: array of Processes and ProcessUses that the dataset / data unit are output from Notes: The combinations possible are: ■ dataset_id and data_unit_name - server looks up dataunit_id, returns stats ■ data_unit_id only - ok, will return statistics for that ■ dataset_id and data_unit_id - same as previous, but data unit must be in the dataset or it is an error ■ dataset_id only - error ■ data_unit_name only - error

Get Related Dataflows

Get all ‘dataflow segments’ that involve a specified entity. This is a useful way to find not only which entities are related (within 2 graph links) to a given entity, but also how they are related.

A dataflow segment is of the form:

/-> process / data-input* -> process-use -> data-output* \ \-> job

where: ● data-input is a set of ‘Source [DataUnit]’ or ‘Dataset [DataUnit]’ pairs, input to and output from a ProcessUse ○ DataUnit does not have to be present for all process uses (e.g., source-> dataset) ○ There can in general be multiple input pairs and multiple output pairs ● process-use is a ProcessUse instance ● data-input is a Source/DataUnit or Dataset/DataUnit pair, input to a ProcessUse ● process is the Process entity that the ProcessUse is a snapshot of ● job contains Job information for the executing or executed ProcessUse (empty if not executable)

The main segment contains the main data flow information: data input to process use and data output. The process and job information are secondary info, and can be optionally omitted.

Path: search/related/dataflow Method: GET

Parameters: ● entity_id - the ID of the reference entity; must be of type Source, Dataset, Process, ProcessUse, or Job. ● include_secondary - whether to include ‘secondary’ entities (process and job entities), or not; default is true, to include the secondary entities.

Returns: ● an array containing structures with the the following fields: ○ use-role - the role that the starting entity plays in the segment (one of ‘data-input’, ‘data-output’, ‘process-use’, ‘process’, or ‘job’) ○ use-date - the modifiedAt date from the process-use (copied up for convenience) ○ data-input - the data input entity (and contained entity if there is one) ○ process-use - the process use ○ data-output - the data output entity (and contained entity if there is one) ○ process - the process that the process use is a snapshot of ○ job - the job tracking execution info (empty if not an executable process) ● Each path field (data-input, etc) contains the following core properties:

○ entity/id ○ entity/name ○ entity/type ○ entity/modifiedAt ○ entity/modifiedBy ● In addition, the ‘data’ fields (data-input and data-output) will have an additional sub-field, called ‘contained-entity’, if the ProcessUse context had a dataUnitName property. In that case, this which will have the same 5 core properties for the data unit. ● All other ‘auxiliary’ information should be in the ‘related’ section: ○ for all types: description, modifiedByUser, entityState ○ for sources: storageType (technical and human readable), location, # tables (calc) ○ for datasets: # tables (calc) ○ for processes: processType, processClass, processScope, # uses (calc) ○ for process uses: job ID, job name, job status, job duration (raw and display) ○ for jobs: process use ID, job name, job status, job duration (raw and display)

Notes: ● If include_secondary is false, the ‘process’ and ‘job’ fields will not be included in the response. ● The ‘use-role’ and ‘use-date’ are used to extract out particular segments. For example, a data entity can be an input to or an output from a process use, or both. The role allows the client to ‘filter’ based on these use contexts. (Similar to the ‘in’ and ‘out’ grouping output from search/related/processes). The use-date is a copy of the process-use modifiedAt, and is placed at the top-level to assign that same timestamp to the overall segment, and provide an easy way to sort segments by timestamp.

Get Entity Totals

Get the total number of entities of each of the main types exposed by Loom: source/Source, dataset/Dataset, process/Process, job/Job.

Path: search/totals Method: GET Parameters: none Returns: map of type to count

______

Data Access Operations

These operations focus on basic data access. These do not deal with any kind of filtering, processing, or transformations; those kinds of operations are handled through the Processing API. Requests

Request Description GET data/file/read_lines Get the first rows from a text file. GET data/file/read_parsed Get the first parsed lines from a text file. GET data/dataset/head Get the first rows from an individual data unit within a dataset. GET data/dataset/stats Get the statistics for an individual data unit within dataset.

Request Details

Read Lines from Text File

Get the first rows from text file in HDFS.

Path: data/file/read_lines Method: GET Parameters: location: The absolute path to the file in the file system. nrow: The number of rows to return; default 10. Returns: An array of strings, each of which is a line of text from the file.

Read Lines from Text File

Get the first rows from text file in HDFS.

Path: data/file/read_parsed Method: POST Parameters: location: The absolute path to the file in the file system. file_format: The Format struct, to interpret the bits on disk nrow: The number of rows to return; default 10. Returns: rows: An array of rows from the file. Each row is an array of strings that are the columns parsed from the lines in the file. columns: An array of strings that gives the column names used in the file. The column names are parsed from the file's header if possible, else the columns are assigned auto-generated names.

Get the First Rows from a Data Unit

Get the first rows from an individual data unit within a dataset.

Path: data/dataset/head

Method: GET Parameters: dataset_id: The ID of the dataset; optional if data_unit_id is specified data_unit_name: the name of the data unit within the dataset (dataset_id must be specified); optional if data_unit_id is specified data_unit_id: The ID of the data unit (dataset_id is optional, but must be consistent if specified); optional, if data_unit_name is specified nrow: The number of rows to return; default 10. Returns: records: the records column_names: the names of the columns Notes: The combinations possible are: ■ dataset_id and data_unit_name - server looks up data_unit_id, returns stats ■ data_unit_id only - ok, will return statistics for that ■ dataset_id and data_unit_id - same as previous, but data unit must be in the dataset or it is an error ■ dataset_id only - error ■ data_unit_name only - error

Get the Statistics for a Data Unit

Get the statistics for an individual data unit within dataset.

Path: data/dataset/stats Method: GET Parameters: container_id: The ID of the dataset; optional if data_unit_id is specified data_unit_name: the name of the data unit within the dataset (container_id must be specified); optional if data_unit_id is specified data_unit_id: The ID of the data unit (container_id is optional, but must be consistent if specified); optional, if data_unit_name is specified Returns: A ‘scan metadata’ structure, consisting of (only primary fields shown): scan.table/numRecords: the number of records scan.table/columnMetadata: the columns: ■ scan.table/column - the column that the metadata is for, with sub-fields of ● entity/type ● entity/name ● data.table/dataType ■ scan.table/columnType - one of 'string', 'number', or 'object' ■ scan.table/nullValues - number of null values ■ scan.table/emptValues - number of empty string values, for string columns ■ scan.table/minValue - the minimum value, for numeric columns ■ scan.table/maxValue - the maximum value, for numeric columns ■ scan.table/meanValue - the mean, for numeric columns ■ scan.table/stdDev - the standard deviation, for numeric columns Notes: The possible parameter combinations are: ■ container_id and data_unit_name - server looks up data_unit_id, returns stats ■ data_unit_id only - ok, will return statistics for that

■ container_id and data_unit_id - same as previous, but data unit must be in the dataset or it is an error ■ container_id only - error ■ data_unit_name only - error

Calculate the Statistics for a Data Unit

Calculate the statistics for an individual data unit within dataset.

Path: data/dataset/stats Method: POST Parameters: container_id: The ID of the dataset; optional if data_unit_id is specified data_unit_name: the name of the data unit within the dataset (container_id must be specified); optional if data_unit_id is specified data_unit_id: The ID of the data unit (container_id is optional, but must be consistent if specified); optional, if data_unit_name is specified Returns: ID of the data unit the statistics were computed for. Notes: The possible parameter combinations are: ■ container_id and data_unit_name - server looks up data_unit_id ■ data_unit_id only - ok ■ container_id and data_unit_id - same as previous, but data unit must be in the dataset or it is an error ■ container_id only - error ■ data_unit_name only - error

______

Execution Operations

These operations deal with processing data -- executing transformations and tracking job progress. Requests

Request Description POST execute/transform Execute the specified transformation. GET execute/status Get the status of an executed job.

Request Details

Execute a Transform

Execute the specified transformation.

Path: execute/transform Method: POST Parameters: process_id: The ID of the process to execute. contexts: The input and output data contexts; not required if process has a default context with both dataset and data unit name defined. Returns: id: The ID of a Job to track the progress of the execution Notes: If a context is not provided, the process being executed must have a default context defined. If that is the case, the input context will be the default input context, and the output context will be automatically generated as follows: the dataset will be the same as the input dataset, and the output data unit (table) name will be auto- generated.

Get Execution Status

Get the status of an executed job. The job may be in-progress, or may have completed (or failed).

Path: execute/status Method: GET Parameters: job_id: The ID of the job (appended to URL) Returns: The job status and progress, as a list with 2 fields: ■ job/status: the job's execution status ■ job/progress: the details about the job's execution, a JobProgress

______

Environment Operations

These operations deal with the external environment which Loom interacts with. Environment Struct Attributes

FileInfo Attributes

A FileInfo is a structure that describes a file or directory in HDFS.

Name [Type] Description path The full path of the file or directory. string isDir If true, is a directory; otherwise is a file.

boolean length The length of the file in bytes. integer blockSize The number of blocks in a file block. integer modificationTime When the file or directory was last modified. instant owner Username in HDFS of user who owns the file or directory string group Name of group in HDFS that owns the file or directory. string permission The file permissions, in string format (e.g., “644”). string replication How many times the file is replicated across a cluster. integer

Requests

Request Description GET environ/fs/home_dir Get the home directory of the file system. GET environ/fs/list_info Get a listing of files in a directory. This is not recursive. GET environ/fs/file_info Get information about a specific file. POST environ/fs/files Upload one or more files to a specified directory in HDFS. GET environ/hive/dblist Get a list of databases in a Hive instance.

Request Details

File System Home Directory

Gets the name of the file system home directory. The file system can either be the native file system on the Loom server, or a Hadoop File System (HDFS) managed by the server. See also Apache WebHDFS documentation.

Path: environ/fs/home_dir Method: GET Parameters: none Returns: The path of the home directory

File System List Files

Gets all file system details for everything in the provided path. If location is a directory, then lists the details for everything in that directory. If location is a file, then lists the details for just that file. The file system can either be the native file system on the Loom server, or a Hadoop File System (HDFS) managed by the server. See also Apache WebHDFS documentation.

Path: environ/fs/list_info

Method: GET Parameters: location: The path to get details for. May be a file or a directory. Returns: An array containing FileInfo objects (see above). E.g., [{"path":"file:///tmp/20130710194447","group":"wheel","blockSize":33 554432,"modificationTime":"2013-07- 10T23:44:47Z","length":102,"owner":"root","isDir":true,"replication" :1,"permission":"rwxr-xr-x"}, {"path":"file:///tmp/geColladaModelCacheLock","group":"wheel","block Size":33554432,"modificationTime":"2013-07- 23T02:30:47Z","length":0,"owner":"pag","isDir":false,"replication":1 ,"permission":"rw-r--r--"}]

File System File Information

Get the file system details for a specified path. If path is a directory, then returns just the description of that directory. See also Apache WebHDFS documentation.

Path: environ/fs/file_info Method: GET Parameters: location: The path to get details for. May be a file or a directory. Returns: A FileInfo object (see above). E.g. {"path":"file:///tmp","group":"wheel","blockSize":33554432,"modifica tionTime":"2013-07- 23T19:44:29Z","length":476,"owner":"root","isDir":true,"replication" :1,"permission":"rwxrwxrwt"}

File Upload

Upload one or more files to a specified directory in HDFS.

Path: environ/fs/files Method: POST Parameters: file: The fully-qualified path to the file to upload. target_directory: The destination directory in HDFS, where the file will be uploaded to. Returns: No return value.

Hive Database Listing

Get the databases in a Hive instance.

Path: environ/hive/dblist Method: GET Parameters: location: If present, list the tables in the database named by the path

all: If true, include databases that are Loom datasets; optional, defaults to false Returns: A map containing tables, and if location specified, names of databases {tables: [ { dbname: name: owner: tableType: /* "MANAGED" | "EXTERNAL" */ parameters: viewOriginalText: viewExpandedText: }, ... ] paths: [ "dbname_1", "dbname_2", ... ] /* only if location="" or empty */ }

______

System Operations

System operations deal with the Loom system itself. System Struct Attributes

SystemVersion Attributes

A SystemVersion is a structure that describes the Loom version.

Name [Type] Description versionDate The date and time when the Loom version was released instant versionNumber The Loom release identifier (e.g., 1.0.5) string buildNumber The Loom build identifier; mostly for internal Revelytix string use versionAndBuild The Loom release identifier with the build identifier string appended productName The name of the product (always ‘Loom’) string productEdition The name of the product edition (e.g., ‘Standard’) string

SystemConfig Attributes

A SystemConfig is a structure that describes the Loom configuration. These are the properties defined in the ‘loom.properties’ file under the ‘config’ directory in your Loom installation.

Name [Type] Description persist.mode The mechanism that Loom uses for persisting datasets. string If ‘loom’, then Loom manages persistence in directories of HDFS files. If ‘hive’, then Loom uses Hive to persist datasets as databases in Hive. Default: loom dataset.persist.dir The location in HDFS where Loom-managed datasets string are created, when running Loom persist mode. Defaults to a directory named 'loom-datasets' in the HDFS working directory. activeScan.hdfs.enabled If true, active scanning of potential sources in HDFS is boolean enabled. Default: false activeScan.hdfs.baseDir Comma-separated list of directories under which to scan string for potential sources in HDFS. Directories may be specified as an absolute hdfs:// URL or a relative path that will be resolved against the Loom working directory. The scan is recursive, so all sub-directories of each configured directory will be scanned. Defaults to Loom working directory. activeScan.hdfs.scanIntervalMinutes The interval, in minutes, at which Loom will scan HDFS integer for potential sources. Default: 60 activeScan.hdfs.parseLines The number of records to parse from a file in HDFS to integer determine whether it's a potential source. Default: 50 activeScan.hdfs.scoreThreshold The threshold above which the confidence level must be float for a file in HDFS to be considered a potential source. The confidence level is a computed value between 0 and 1. Default: 0.25 activeScan.hdfs.maxBufferSize The maximum amount of data to read into memory from long an HDFS file to determine whether it's a potential source. Default: 8388608 security.enabled Enables or disables Loom security. boolean If security is enabled, user impersonation is performed. Default: false security.authentication Configures how authentication is done: does the user boolean exist and have permission? Username and password will always be requested in order to get a session. -- jaas - (default) Use JAAS. JAAS is configured in security-unix.conf. Must have a valid session to access API. -- loom - Use Loom username/password for a valid session. API rejects requests without a valid session. -- disabled - Use Loom username/password to get a

session. Valid session not required to access API. ssl.enabled Enables SSL (https) support. Disabled by default. boolean ssl.port Configures the SSL port. Defaults to 8443. long

Requests

Request Description GET system/version Get system information, such as Loom version. GET system/config Get system configuration information. POST system/log_message Writes a message to the system log. GET system/backup Get the contents of the registry; for release migration. POST system/restore Fill the registry with the backed up contents from another registry.

Request Details

Get Version

Returns version information for the instance of Loom.

Path: system/version Method: GET Parameters: none Returns: SystemVersion struct (see above)

Get Configuration

Writes a message to the Loom system log. This is useful for ‘tagging’ activities before or after they are performed. Will emit the username from the current session.

Path: system/config Method: GET Parameters: none Returns: SystemConfig struct (see above); e.g. {"activeScan.hdfs.enabled":true,"activeScan.hdfs.baseDir": ["/data/dataset"],"activeScan.hdfs.scanIntervalMinutes":60,"activeSc an.hdfs.parseLines":50,"activeScan.hdfs.scoreThreshold":0.25,"active Scan.hdfs.maxBufferSize":8388608,"persist.mode":”hdfs”,"dataset.pers ist.dir":"data/loom-datasets","security.enabled": false,"security.authentication": "disabled","ssl.enabled": false,"ssl.port": 8443,"jobService.threadPool.size": 10}

Log a Message

Writes a message to the Loom system log. This is useful for ‘tagging’ activities before or after they are performed. Will emit the username from the current session.

Path: system/log_message Method: POST Parameters: level: The log level. One of ‘info’, ‘debug’, ‘trace’, ‘fatal’, ‘warn’. message: The message to write Returns: none; writes a message to the txnlog, e.g. 2013-08-08 10:41:03,701- INFO - [fabric.txnlog] - [nREPL-worker-36] - Logged by : THIS IS A MESSAGE

Backup Registry

Get the contents of the registry; for release migration. This can be restored to a new registry using /system/restore.

Path: system/backup Method: GET Parameters: none Returns: array of entity objects

Restore a Backup

Fill the registry with the backed up contents from another registry. The backup contents are obtained from /system/backup.

Path: system/restore Method: POST Parameters: array of entity objects wrapped in ‘results’ (output from backup) Returns: array of entity/id’s of restored objects

______

Registry Model Notes

Registry Model Overview

The following figure shows a high-level view of the Loom registry model. The model is comprised of a set of ‘domain’ models, which define a set of connected Entity Types. The models themselves have dependencies based on cross-model relationships between entity types in the domains. Some domain models are extensible, and have specific sub-models to expose specific functionality.

The primary 3 entity types are Source, Dataset, and Process. Source and Dataset models represent unmanaged and managed sets of data, respectively; they both are derived from the base DataContainer, which contains DataUnits (e.g., tables) with schema information. Data containers and data units have underlying persistent storage, which is proxied by Storage (container) and storage units. The Process entity represents processing performed on data entities, in a generic form.

Lineage is derived from the inter-connected instances of data entities (containers or data units) and processes. In order to compute valid lineage, a ‘snapshot’ of a process must be taken when it is used (executed, or just used in a relationship). This provides the immutability of processes from a lineage perspective. From a data perspective, immutability is provided by making dataset units (i.e., tables) non-modifiable once they have been used.

Core Entity Attributes

All entities have the following core attributes. In addition to these, entities of each type have their own type-specific attributes.

Name [Type] Description entity/id Unique identifier for the entity. In the form of a UUID. Entity string IDs are used for references between entities. entity/name Name of the entity. string entity/description Description of the entity. string entity/folder Registry folder in which the entity is organized. string entity/tags Tags applied to the entity by users. array of string? entity/createdAt Timestamp when the entity was created. instant (long) entity/createdBy User ID of the person who created the entity. string entity/modifiedAt Timestamp when the entity was last modified. instant (long) entity/modifiedBy User ID of the person who last modified the entity. string

Usage Scenarios

Basic Entity/Metadata Operations

Create a Source

A data source is a set of data whose lifecycle is not managed and controlled by Loom. These may be registered with Loom as a Source. This yields the following benefits:

1. visibility of the source to a wider set of users 2. definition and management of metadata describing the source 3. determination of how to read and parse the data 4. understanding of the data characteristics, through descriptive statistics and other measurements 5. first step in creating a dataset for more thorough data preparation and analysis 6. ability to define relationships between sets of data

Here is how you create a Source entity in Loom, to represent a data source in its ‘native’ form. This version allows for user interaction; there is a more streamlined version if the user knows all information up-front. The example is for a text file-based source.

Step Description Notes/Methods 1 Identify location Input to API 2 Identify storage type, and format type Inputs to API 3 Read source metadata, return structures GET /sources/default 4 Identify which parts of the source to include Set ‘containsData’ 5 Review raw data in files GET /data/file/read-lines 6 Review parsed data based on current format GET /data/file/read-parsed 7 Modify format characteristics to best parse data Change Format properties 8 Register the source with Loom POST /sources

Create a Dataset

A dataset is a set of data that is created by Loom, and whose lifecycle is controlled and managed by Loom. These are represented by Datasets in Loom. Datasets may be initially created in Loom from Sources; thereafter, any processing that is performed on a dataset in Loom will result in the automatic creation and registration of another dataset. Datasets are used for data preparation -- and subsequently data analysis -- using Loom.

The benefits of defining a dataset are:

1. visibility of the dataset to a wider set of users 2. definition and management of metadata describing the source

3. Loom controls how the data are persisted, and can optimize this for efficient processing. 4. understanding of the data characteristics, through descriptive statistics and other measurements 5. used for data preparation and cleansing in Loom 6. relationships between datasets automatically captured when processing occurs

Here is how you create a Dataset entity in Loom from an existing Source, to represent a managed set of data. This version involves user interaction; there is a more streamlined sequence of steps if no user interaction is required.

Step Description Notes//Methods 1 Get default dataset from registered source GET /datasets/default 2 Change which data units to include 3 Change schema information Edit data unit Schemas 4 Register the dataset with Loom POST /datasets

Create a Process

A process represents some processing of data. Data resides in Sources and Datasets, so processing acts upon those (specifically, upon the data in tables in them). A process is represented by a Process entity in Loom. A process is similar to a function or method in a programming language -- it has a name, some input parameters, and some output parameters. In the case of Loom, there is also the notion of data contexts, which arguments may bind to during execution.

There are a variety of types of processes (including any type a user wants to define). The most common when using Loom for data processing is a SQL Query.

Here is how you create a Process entity in Loom from an existing Source, to represent a managed set of data. This version involves user interaction; there is a more streamlined sequence of steps if no user interaction is required.

Step Description Objects//Methods 1 Define the input arguments (name-value pairs) Arguments 2 Define the input and output data contexts Contexts 3 Define the local Process Process 4 Register the Process with Loom POST /processes

Overlay Metadata on Corporate Resources

Another set of use cases involves using Loom primarily as a metadata registry, to attach metadata, manage resource, define relationships, and determine lineage between data resources that are generated and used outside of Loom.

Here is the basic sequence of steps. The full sequencing of each activity is consolidated down to one primary method for each; the additional details are covered above.

Step Description Objects//Methods 1 Register Sources for each set of data POST /sources 2 Register Processes POST /processes 3 ‘Use’ Processes to link Sources together POST /processes//uses 4 Compute lineage relative to a Source GET /search/lineage

API Examples Source

The following is an example of the composite structure that is returned from and passed into the various /sources methods. JSON Composite Structure

{ “entity”: { "entity/type": [ "data/DataContainer", "source/Source" ], "entity/name": "ModifiedName", "entity/folder": "test/test2/test3", "entity/description": "Created by RLoom", "entity/tags": "RLoom", "data/structuralForm": "table", "source/dataAccessible": true, "source/metadataAccessible": true, "source/entityState": "active", "source/expandable": true "data/dataUnit": [], }, “storage”: { "persist/format": { "persist.file.text/headerRow": true, "entity/type": [ "persist/StorageFormat", "persist.file/DelimitedFormat" ], "persist/formatType": "text/delim", "persist.file.delim/delimiter": ",", "persist.file.delim/quoteChar": "\"", "persist.file.text/skipRows": 0 }, "entity/type": [ "persist/Storage", "persist.file/FileSet"

], "persist/storageType": "file/text", "persist/location": "/data/datasets/earthquakes", "persist/application": "", "persist.file/isSingleFile": true, "persist/storageUnit": [] }, “storage_units”: [ { "entity/type": [ "persist/StorageUnit", "persist.file/FileSetFile" ], "persist/location": "file:///data/datasets/earthquakes/earthquakes.ddl", "persist/relativeLocation": "earthquakes.ddl", "persist/containsData": false, "persist.file/fileExtension": "ddl", "persist.file/isLogicalFile": false, "persist.file/isBinary": false }, { "entity/name": "eqs7day", "entity/type": [ "persist/StorageUnit", "persist.file/FileSetFile" ], "persist/location": "file:///data/datasets/earthquakes/eqs7day.csv", "persist/relativeLocation": "eqs7day.csv", "persist/containsData": true, "persist.file/fileExtension": "csv", "persist.file/isLogicalFile": false, "persist.file/isBinary": false }, { "entity/type": [ "persist/StorageUnit", "persist.file/FileSetFile" ], "persist/location": "file:///data/datasets/earthquakes/README.txt", "persist/relativeLocation": "README.txt", "persist/containsData": false, "persist.file/fileExtension": "txt", "persist.file/isLogicalFile": false, "persist.file/isBinary": false

Process

The following is an example of the structure that is returned from and passed into the various /processes methods. JSON for SQL Transform with no Default Input Context

The following is an example of the JSON that can be POSTed to /processes to define an executable ad-hoc SQL process that can be executed through Loom. This particular example does not have a default input context defined.

{ "entity/description": "Created by RLoom", "process/argument": [ { "entity/type": [ "process/Argument", "process/ConfigArgument" ], "entity/name": "transformText" } ], "entity/type": [ "process/Process" ], "process/processClass": "transform", "process/processType": "sql-query", "process/processScope": "dataunit", "entity/name": "SQLProcess_NoContexts", "entity/folder": "test", "process/isExecutable": true, "entity/tags": "RLoom" }

JSON for Linking 3 Sources

The following is an example of the JSON that can be POSTed to /processes to define a ‘descriptive’ (non-executable) process to link 2 sources as inputs with 1 as output.

{ "entity/type": [ "process/Process" ], "entity/name": "ProcessLinkingSources", "entity/description": "2 input sources, 1 output",

"entity/folder": "test1/test2", "entity/tags": "tag1, tag2", "process/processClass": "descriptive", "process/processType": "lineage-process", "process/processScope": "container", "process/isExecutable": false, "process/argument": [ { "entity/type": [ "process/Argument", "process/ConfigArgument" ], "entity/name": "relationship", "process.arg/value": "link" }, { "entity/type": [ "process/Argument", "process/ConfigArgument" ], "entity/name": "source1", "process.arg/value": "Source1" }, { "entity/type": [ "process/Argument", "process/ConfigArgument" ], "entity/name": "source2", "process.arg/value": "Source2" }, { "entity/type": [ "process/Argument", "process/ConfigArgument" ], "entity/name": "source3", "process.arg/value": "Source3" } ], "process/context": [ { "entity/type": [ "process/Context" ], "entity/name": "Source1", "process.context/inout": "in", "process.context/container": "51ee9a72-af33-4798-97dd-15b71ea86d49" }, { "entity/type": [ "process/Context" ], "entity/name": "Source2", "process.context/inout": "in", "process.context/container": "51ee9a74-fdc2-47c9-a9c3-203f165f353d" }, { "entity/type": [ "process/Context" ], "entity/name": "Source3", "process.context/inout": "out",

"process.context/container": "51ee9a77-e4d6-4135-9160-2ea07d565927" } ] }

Executing/Using Processes

Processes are used, or realized, either through execution (for executable processes) or by explicitly defining a ‘use’ (for non-executable processes). In both cases, a ProcessUse is created. A ProcessUse is a snapshot of the Process at the instant the process was used.

When using a process, the data contexts must be specified. (Default input contexts from the Process may be used, or new input contexts may be specified; output contexts must always be specified.) A context represents a single set of data involved in the processing. If the process is scoped at the container level, then the contexts will contain only data container references (i.e., references to Sources and Datasets, one per context). If the process is data unit scoped, then the contexts will have not only a container reference, but also the name of a data unit (i.e., table) within the container. In addition to a name, a container, and an optional data unit name, each context must specified whether it is used for input or output.

When executing or using a process through the API, an array of Contexts is provided.

JSON for Array of Contexts, Container-Level Process

This is an example of a JSON for the ‘contexts’ parameter for POST /execute/transform and POST /processes//uses.

{ "contexts": [ { "entity/type": [ "process/Context" ], "entity/name": "Source1", "process.context/inout": "in", "process.context/container": "51ee9a72-af33-4798-97dd-15b71ea86d49" }, { "entity/type": [ "process/Context" ], "entity/name": "Source2", "process.context/inout": "in", "process.context/container": "51ee9a74-fdc2-47c9-a9c3-203f165f353d" }, { "entity/type": [ "process/Context" ],

"entity/name": "Source0", "process.context/inout": "out", "process.context/container": "51ee9a77-e4d6-4135-9160-2ea07d565927" } ] }

JSON for Array of Contexts, DataUnit-Level Process

This is an example of a JSON for the ‘contexts’ parameter for POST /execute/transform and POST /processes//uses. This would be similar to contexts used in single-table SQL query executions (1 table in, 1 table out).

{ "contexts": [ { "entity/type": [ "process/Context" ], "entity/name": "input", "process.context/inout": "in", "process.context/container": "520d1f28-35e2-47e8-aa80-6509d674bda1", "process.context/dataUnitName": "eqs7day" }, { "entity/type": [ "process/Context" ], "entity/name": "output", "process.context/inout": "out", "process.context/container": "520d1f28-35e2-47e8-aa80-6509d674bda1", "process.context/dataUnitName": "Result01" } ] }

API Notes

Notes on HTTP calls

Each API is accessed by an HTTP URL whose path starts with the root path of the API. The individual API operations are addressed both by extensions to this root path, and by the HTTP methods used to access to the URL.

Methods

Some HTTP clients (e.g. browsers) may not support all of the methods used by an API. In this case, a POST operation may be used in conjunction with a URL parameter named _method. The value of the _method parameter will be used to override the POST method.

As an example, if it is not possible to send a PATCH request to the URL:

http://somehost.com/path/operation then a POST may be sent instead to the URL:

http://somehost.com/path/operation?_method=PATCH

Host and Port

When referencing the Loom Server via its URL, the host name must be resolvable through DNS to the server that Loom is running on, according to the machines listed in the local hosts file of that machine (or may be an explicit IP address).

The default port is 8080, although that can be changed when the Loom Server is started. Firewalls should be configured to allow the host machine to accept connections to this port number. Input Parameters

Most API operations require several parameters. For GET requests, these parameters are provided as part of the URL. For other HTTP requests (POST, PUT, PATCH), the parameters are provided in the body, as a JSON map of parameter names to values. Occasionally an unnamed parameter may appear in the path, as is often seen in REST operations. This is the case, for example, when performing operations on a specific entity instance, in which case, the entity ID is part of the URL.

_method Parameter

Due to URL length limitations, it is sometimes not possible to put the required input parameters in the URL for a GET request. In those cases, the special _method parameter should be placed in the operation URL, and a POST operation should be used. The back-end service will interpret the request as an HTTP HGET, but will look for the parameters in the request body, rather than in the URL.

Responses

All responses are JSON map structures in the HTTP body. This is consistent even when a response holds a single value or an array.

Reference properties

Many entities contain references by ID to other entities. For instance, a job currently contains a reference to a query, an input dataset, and an output dataset. Invariably, a client (e.g. the UI) requires the name and folder path of the referred-to entity in order to display the reference to the user. In addition, the ID is used to construct a hyperlink to the referred-to entity. The API provides a way to return this information, in the ‘related’ part of the standard response structure.

References and composites

Certain kinds of references are not to named entities but are instead references to internal entities. For instance a DataUnit contains a reference to a Schema entity, which contains references to Column entities, and so on. In most cases, those kinds of references to internal objects are expanded into nested JSON objects when passed through the API.

Serialization Entities are serialized as JSON.

In general, a sub-graph is serialized as a nested JSON structure, where references (UUID ‘pointer’ attributes) and composite structs are treated similarly, as nested values under a parent.