Alinex Data Store
Total Page:16
File Type:pdf, Size:1020Kb
Alinex Data Store Read, work and write data structures to differents stores Alexander Schilling Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> Table of contents Table of contents 1. Home 6 1.1 Alinex Data Store 6 1.1.1 Usage 6 1.1.2 Debugging 6 1.1.3 Module Usage 7 1.1.4 Chapters 7 1.1.5 Support 7 1.2 Command Line Usage 8 1.2.1 Input 8 1.2.2 Output 8 1.2.3 Transform Files 9 1.2.4 Using Definition 9 1.2.5 Examples 9 1.3 Last Changes 10 1.3.1 Version 1.16.0 - (12.05.2021) 10 1.3.2 Version 1.15.0 - (02.01.2021) 10 1.3.3 Version 1.13.0 - (16.06.2020) 10 1.3.4 Version 1.12.0 - (27.01.2020) 10 1.3.5 Version 1.11.0 - (13.01.2020) 11 1.3.6 Version 1.10.0 - (22.11.2019) 11 1.3.7 Version 1.9.1 - (13.11.2019) 11 1.3.8 Version 1.8.0 - (31.10.2019) 11 1.3.9 Version 1.7.0 - (13.10.2019) 11 1.3.10 Version 1.6.0 - (01.10.2019) 11 1.3.11 Version 1.5.0 - (28.08.2019) 12 1.3.12 Version 1.4.0 - (15.08.2019) 12 1.3.13 Version 1.3.0 - (6.08.2019) 12 1.3.14 Version 1.2.0 - (22.06.2019) 13 1.3.15 Version 1.1.0 - (17.05.2019) 13 1.3.16 Version 1.0.0 - (12.05.2019) 13 1.3.17 Version 0.7.0 (29.04.2019) 13 1.3.18 Version 0.6.0 (26.04.2019) 14 1.3.19 Version 0.5.0 (19.04.2019) 14 1.3.20 Version 0.4.0 (17.04.2019) 14 1.3.21 Version 0.3.0 (15.04.2019) 14 - 2/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> Table of contents 1.3.22 Version 0.2.0 (12.04.2019) 14 1.3.23 Version 0.1.0 (0t.04.019) 14 1.4 Roadmap 16 1.4.1 Add Protocols 16 1.4.2 Multiple sources 16 1.5 Privacy statement 17 2. API 18 2.1 Application Programming Interface 18 2.1.1 Basic Usage 18 2.1.2 TypeScript 18 2.2 Loading from Persistent Store 19 2.2.1 Initialization 19 2.2.2 Options 21 2.2.3 Load 22 2.2.4 Save 23 2.2.5 Reload 23 2.3 Transform 25 2.3.1 Parse 25 2.3.2 Format 25 2.4 Access and Manipulate Data 26 2.4.1 Has Element 26 2.4.2 Get Element 26 2.4.3 Filter 27 2.4.4 Coalesce 27 2.4.5 Set Element 27 2.4.6 Insert into Array 28 2.4.7 Push to Array 28 2.4.8 Empty Element 28 2.4.9 Delete Element 29 3. Specifications 30 3.1 Specifications 30 3.1.1 URI 31 3.2 Protocols 32 3.2.1 File Protocol Handler 32 3.2.2 SSH File Transfer Protocol Handler 33 3.2.3 FTP and FTPS Handler 39 3.2.4 HTTP and HTTPS Handler 41 3.2.5 PostgreSQL Database Handler 43 - 3/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> Table of contents 3.2.6 Shell Execution Protocol Handler 44 3.3 Compression 45 3.3.1 GZip Compression 45 3.3.2 Raw Deflate Compression 46 3.3.3 Brotli Compression 47 3.3.4 BZip2 Compression 48 3.3.5 LZMA Compression 49 3.3.6 TAR Archive 50 3.3.7 TAR + GZip Archives 53 3.3.8 TAR + BZip2 Archives 54 3.3.9 TAR + LZMA Archives 55 3.3.10 ZIP Archives 56 3.4 Formats 57 3.4.1 JSON Format 57 3.4.2 BSON Format 58 3.4.3 MessagePack Format 59 3.4.4 JS Format 60 3.4.5 CSON Format 61 3.4.6 CoffeeScript Format 62 3.4.7 YAML Format 63 3.4.8 INI Format 65 3.4.9 Properties Format 66 3.4.10 TOML Format 67 3.4.11 XML Format 68 3.4.12 HTML Format 70 3.4.13 CSV Format 71 3.4.14 Excel 97-2004 Workbook Format 73 3.4.15 Excel 2007+ XML Format 74 3.4.16 Plain Text Format 75 3.4.17 Binary Format 77 4. Internal 78 4.1 Extending this Module 78 4.1.1 Structure 78 4.1.2 Processing 79 4.1.3 DataStore 79 4.1.4 Statistics 79 4.1.5 Download Documentation 79 4.1.6 License 79 - 4/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> Table of contents 4.2 Test 80 4.2.1 SFTP 80 4.2.2 FTP 80 - 5/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> 1. Home 1. Home 1.1 Alinex Data Store The data store is a in memory store for small to medium sized data structures. It allows you to read, work and write in different formats and therefore also to transform between them. Big Features: • read, write and transform data structures • methods to access and work with data structures • 9 protocols supported (file, sftp, HTTP, https...) • 17 file formats supported • 10 compression or archive formats are also supported • integrated access filter language • multi source loading for single structure See more on the specification page. 1.1.1 Usage It can be used from command line but also as module in other NodeJS scripts. The main use cases are: • reading configuration files • pre-processing configuration into other format to easily import later • communicating with REST services • data provider for the Validator module since version 3 Personally I seldom use it standalone and in most times use the additional checking from the Validator. See also the examples there. 1.1.2 Debugging To see some more information you may enable the debug logging using the environment variable DEBBUG=datastore* . The following finer debug elements are alternatively usable: • datastore:protocol:* - output uri accessed by specific handler • datastore:details - output a lot of verbose information • datastore:compression - output size changes • datastore:compression:* - output uri processed through specific handler • datastore:format - output parsed data • datastore:format:* - output uri processed through specific handler • datastore - output general problems like ignore because source not found • datastore* - output all A lot more may be available using the additional core debugging of NodeJS using NODE_DEBUG=xxx . If there is something available the datastore:details logging will inform you about it. - 6/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> 1.1.3 Module Usage 1.1.3 Module Usage Because of the big list of supported alternatives this module depends on a lot of other modules but only the ones really used are loaded on demand. Only the following are loaded always: • @alinex/core • @alinex/data • debug • object-path All others are only loaded at the time the specific format is used. But it will install a bigger collection of modules on your system to be prepared if specific formats are used. 1.1.4 Chapters Read all about the DataStore in the chapters: • API - How to use it in your own script. • Specifications - Detailed format, protocol and file specifications. • Extending - Structure of this module for those extending it. 1.1.5 Support I don't give any paid support but you may create GitLab Issues: • Bug Reports or Feature: Please make sure to give as much information as possible. And explain how the system should behave in your opinion. • Code Fixing or Extending: If you help to develop this package I am thankful. Develop in a fork and make a merge request after done. I will have a look at it. • Should anybody be willing to join the core team feel free to ask, too. • Any other comment or discussion as far as it is on this package is also kindly accepted. Please use for all of them the GitLab Issues which only requires you to register with a free account. - 7/80 - Copyright © 2019 - 2021 <a href="https://alinex.de">Alexander Schilling</a> 1.2 Command Line Usage 1.2 Command Line Usage The CLI allows to work in two major modes and also can generate a bash completion script: • working on a single input source • use input as definition to allow multiple sources datastore [options] # work with single input datastore definition [options] # use input as definition (allow multiple sources) datastore bashrc-script # generate completion script The general options are: --version Show version number [boolean] --quiet, -q only output result [boolean] --help Show help [boolean] 1.2.1 Input The input can be a defined file or if not specified STDIN is used as for pipes. If no input file or URI the --input-format is always needed because the filename is unknown to the DataStore. The other options are used to specify how to read this input: --input, -i input path or URI to read from [string] --input-format format used to parse input [choices: "bson", "coffee", "cson", "csv", "ini", "js", "json", "msgpack", "properties", "text", "toml", "xml", "yaml"] --input-compression compression method to be used [choices: "gzip", "brotli", "bzip2", "lzma", "tar", "tgz", "tbz2", "tlz", "zip"] --records flag to read the CSV always as with records [boolean] Protocol specific options: --proxy HTTP method to use for call [choices: "get", "delete", "head", "options", "post", "put", "patch"] --http-method HTTP method to use for call [choices: "get", "delete", "head", "options", "post", "put", "patch"] --http-header send additional HTTP header [array] --http-data send additional HTTP POST or PUT data [string] --ignore-error ignore HTTP error response codes [boolean] --sftp-privatekey private key file [string] --sftp-passphrase passphrase for private key, if needed [string] Info See the specifications for details to the supported protocols, compression, archive and file formats.