LOGIQ Platform Overview

LOGIQ platform Overview

LOGIQ Insights

LOGIQ Insights is a complete observability platform for monitoring, log aggregation, and analytics with inﬁnite storage scale.

The LOGIQ platform comprises of a User interface, a command line toolkit, a monitoring stack for time series metrics and a log analytics stack for log data.

Monitoring

LOGIQ's monitoring stack is powered by Prometheus. LOGIQ has it's own user interface and does not use Grafana. LOGIQ's monitoring stack also allows data from additional data sources like Elasticsearch, Postgres, MySQL, MariaDB, MongoDB, Athena etc. to be queried, analyzed and visualized.

Log Insights

LOGIQ Log Insights is a comprehensive log aggregation stack. It allows users to ingest, organize, analyze and, search through log data. The Log Insights stack is unique in its ability to use any S3 compatible store as its primary data store.

It is best suited for ingesting log and time-series data directly into your S3 compatible store. The ingested data is written in an open format for ready consumption by other software. Examples of common uses are ingesting log and time-series data from Rsyslogd, Syslog, Syslog-ng, Logstash, Fluentd, Docker and ECS logging drivers, Kafka, inﬂuxDB etc.

The LOGIQ log ingest server is natively written in the Go programming language and compiles into a single binary. This makes it light enough to be bundled with any application stack. The server also can be conﬁgured to run in clustered, HA mode that can tolerate 1 or 2 node failures making it ideal for creating larger clusters. Communities

LOGIQ is a member of the CNCF Releases

2.1.11 - 2020-12-13

Log aggregation

Support for and expressions in search Event rules designer support for && and || for individual parameters Performance and memory improvements

Data convergence

Support for Apache DRUID connector Optionally deploy Grafana with the LOGIQ stack

Logs compare view to select 2 logs to be viewed side by side Easy toggle for activating/deactivating periodic queries

2.1.9 - 2020-11-28

Added

Log aggregation

Support for full data/metadata recovery on service restarts

Log analytics

Support for application and process/pod context in log and search views

Deployment and Infrastructure Support for node selectors. Both taints and node selectors are supported Support for using spot instances on EKS/AWS Support for using S3 compatible buckets directly without a caching gateway to optimize for region optimized deployments

Multi cluster support License management UI Ingest conﬁguration settings exposed in the UI Logs page and search now have application and process/pod level contexts

2.1 - 2020-09-28

Added

Log aggregation

Parquet with Snappy compression for data at REST

Log analytics

Log view supports full JSON view for ingested log data like Search view Performance improvements for faster search, logs, and tailing

Monitoring

Event deduplication that can reduce event data by up to 1000x at peak data rates Deduplication of monitoring events at Namespace granularity

Deployment and Infrastructure

Separation of LOGIQ server into microservices for data ingestion, ML/UI and S3/Metadata management Support for taints in HELM chart for more control over large-scale deployments e.g. schedule ingest pods on dedicated nodes etc. Log tailing infrastructure using Redis switches to diskless replication/no persistence

2.0 - 2020-08-12

Added

Log aggregation

Support for AWS Fargate, Firelens, Fluent forward Protocol LOGIQ Fluent-bit daemon-set for K8S clusters Data extraction via Grok patterns, compatible with Logstash Grok patterns using the Grokky library

Log analytics

Redesigned - Elastic/Kibana like search UI that scales to inﬁnite data from S3 compatible object store Real-time alertable events and alerts from log data Real-time extraction of log data facets using Grok expressions 1-Click conversion of log data events to time series visualization

Logiqctl

Logiqctl command-line toolkit Works with SAML users via API Key

Monitoring

Prometheus alert manager integration into LOGIQ alerts for uniﬁed alerting across logs and metrics Built-in Logiq dashboard for LOGIQ cluster performance and health monitoring

LOGIQ Data platform

Connect numerous popular data sources into the LOGIQ platforms such as Postgres, MySql, Elasticsearch, Athena, MongoDB, Prometheus, and more. JSON Data source for easily converting arbitrary JSON data into tables, widgets, and alerts

Role-based access control

Namespace RBAC for log data from K8S namespaces SAML Integration for RBAC allowing SAML Attributes to map to RBAC groups

Security and Compliance

Fully secured HELM deployment using Role, RoleBindings, ServiceAccounts and Pod Security policies for all service Cryptographically veriﬁed JWT token for API communication Built-in audit logging for the entire product and infrastructure

1.2.1 - 2020-05-11

Added

K8S

Add support for ingress with http and optionally have https ServiceMonitor for ingest server if prometheus is installed

Logs modal ordering to match how developers view logs from a ﬁle Highlight log line from search

Misc

Bug ﬁxes for performance, graceful failure handling/recovery

1.2.0 - 2020-04-28

Oﬃcial GA of LOGIQ's complete Observability platform with support for metrics and log analytics

Added

K8S

Scale-out and HA deployment for Kubernetes via HELM 3 chart ( https://github.com/logiqai/helm-charts )

Monitoring of time series metrics New Log viewer Log viewer integration with faceted search Log time machine to go back in time to log state

CLI

logiqctl is now GA with support for log tailing, historical queries and search

1.1.0 - 2020-02-27

Fixed

Fluentd sends error logs as info - ﬁxed with grok patterns to extract proper severity strings from incoming messages

Added

Anomaly detection via Eventing with built-in and custom rules Built-in UI Help with Intercomm chat Expand and collapse search facets

Changed

New AMI's for AWS marketplace

1.0.0 - 2020-01-21

Oﬃcial GA of LOGIQ's Log Insights platform

Added

AWS Marketplace AMI for all regions including Gov cloud regions AWS CloudFormation 1-click deployment Rsyslog, Syslog protocol support for data ingest via Rsyslogd, syslogd, syslog-ng, logstash, ﬂuentd, ﬂuentbit, docker logging syslog driver. Built-in UI with SQL Queries, Faceted search, Alerts, Dashboards EULA End User License Agreement

IMPORTANT: USE OF THE LOGIQ SOFTWARE CONSTITUTES ACCEPTANCE OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THIS AGREEMENT, DO NOT USE THE SOFTWARE.

This END USER LICENSE Agreement (the “Agreement”) is made effective by and among end-customer (“Customer”), and Logiq.ai Inc. with a place of business at 2216 Emerald Hills Cir, San Jose, CA

The parties agree as follows:

1. DEFINITIONS

1.1. “Intellectual Property Rights” means worldwide common law and statutory rights associated with (i) patents and patent applications; (ii) works of authorship, including copyrights, copyright applications, copyright registrations and “moral” rights; (iii) the protection of trade and industrial secrets and confidential information; (iv) other proprietary rights relating to intangible intellectual property (specifically excluding trademarks, tradenames and service marks); (v) analogous rights to those set forth above; and (vi) divisions, continuations, renewals, reissuances and extensions of the foregoing (as applicable) now existing or hereafter filed, issued or acquired.

1.2. "Cloud Provider" means the provider of computing resources for enabling convenient, on-demand network access to a shared pool of conﬁgurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. The deﬁnition "Cloud Provider" includes both public and private Cloud Provider.

1.3. “Software” means the LOGIQ.ai software, LOGIQ Log Insights, LOGIQ Insights via Cloud Provider to Customer including all Updates related thereto.

1.4. “Update” means a release of the Software containing substantially only Error Corrections, minor new features, functionality and/or performance enhancements.

1.5. “Customer” is a user who subscribes to or deploys Logiq.ai Software using any Cloud Provider. 1.6. “Customer data” is all data created by Customer using the Software and/or stored on Cloud Provider machines operating the Software.

2. OWNERSHIP AND PROPRIETARY NOTICES

2.1. Ownership. The Software and associated elements are licensed, not sold – they remain property of LOGIQ.ai Inc.

2.2. Notices. The parties will not remove, alter, or obscure any copyright or intellectual property notice on the intellectual property of the other party.

2.3. Customer data. Customer data is property of Customer and cannot be accessed by Logiq.ai Inc. unless access rights are granted by Customer.

3. PAYMENTS AND ACCOUNTING

3.1. Fees. Customer shall pay Logiq.ai Inc. the fees set in accordance with the conditions published on the Cloud Provider Marketplace. Customer may use multiple instances of the Software paying for each instance separately.

3.2. Taxes. The Customer shall pay VAT, howsoever designated, to the extent attributable to this Agreement or to any part, service or material furnished hereunder.

3.3. Refunds. The customer may terminate the Cloud Provider instances at anytime to stop incurring charges. Annual subscription cancellations or downgrades are not supported.

4. MAINTENANCE AND SUPPORT SERVICES

4.1. Updates. Logiq.ai Inc. shall provide to Customer the Updates as they become available without additional charges.

4.2. Support. Support is offered to Customer via Email and serves the purpose of resolving product defects. At Logiq.ai Inc. discretion, initial installation support for a Customer may be offered at extra charges. Please contact [email protected] for further details.

5. LICENSE GRANTS AND RESTRICTIONS 5.1. Grants. Logiq.ai Inc. grants to Customer a limited, non-exclusive, non-transferable license under Logiq.ai Inc. Intellectual Property Rights to the Software. This agreement enables Customer to use the LOGIQ log analytics solution in the Cloud Provider environment so as to provide service offerings directly and indirectly to Customer’s subscribers, customers and clients.

5.2. Term and Termination. The license is granted for the duration of Cloud Provider subscription to the Licensed Software.

5.3. Limitations on Use. Customer may not attempt to modify, reverse engineer or disassemble, distribute, sublicense or transfer the Software out of the licensed Cloud Provider environment.

5.4 Open Source Components. The Software incorporates numerous open-source components made available by third parties under their own license terms. The respective licenses apply to these components. A list of the components and their respective license terms is available in Software documentation.

6. DISCLAIMERS AND LIMITATION OF LIABILITY

6.1. Warranty disclaimer. THE SOFTWARE IS PROVIDED AS IS. IN NO EVENT DOES Logiq.ai Inc. WARRANT THAT THE SOFTWARE IS ERROR FREE. EXCEPT AS PROVIDED IN THIS AGREEMENT, THE PARTIES DO NOT MAKE ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING, BUT NOT LIMITED TO, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND TITLE.

6.2. Liability limitation. EXCEPT AS OTHERWISE PROVIDED IN SECTION 6.4, IN NO EVENT WILL COMPANY OR ANY OF ITS LICENSORS, SERVICE COMPANY’S, OR SUPPLIERS BE LIABLE UNDER OR IN CONNECTION WITH THIS AGREEMENT OR ITS SUBJECT MATTER UNDER ANY LEGAL OR EQUITABLE THEORY, INCLUDING BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY AND OTHERWISE, FOR ANY: (a) LOSS OF PRODUCTION, USE, BUSINESS, REVENUE OR PROFIT OR DIMINUTION IN VALUE; (b) IMPAIRMENT, INABILITY TO USE OR LOSS, INTERRUPTION OR DELAY OF THE SERVICES, OTHER THAN FOR THE ISSUANCE OF ANY APPLICABLE SERVICE CREDITS PURSUANT TO, (c) LOSS, DAMAGE, CORRUPTION OR RECOVERY OF DATA, OR BREACH OF DATA OR SYSTEM SECURITY, OR (d) CONSEQUENTIAL, INCIDENTAL, INDIRECT, EXEMPLARY, SPECIAL, ENHANCED OR PUNITIVE DAMAGES, REGARDLESS OF WHETHER SUCH PERSONS WERE ADVISED OF THE POSSIBILITY OF SUCH LOSSES OR DAMAGES OR SUCH LOSSES OR DAMAGES WERE OTHERWISE FORESEEABLE, AND NOTWITHSTANDING THE FAILURE OF ANY AGREED OR OTHER REMEDY OF ITS ESSENTIAL PURPOSE.

6.3 CAP ON MONETARY LIABILITY. EXCEPT AS OTHERWISE PROVIDED IN SECTION 6.4, IN NO EVENT WILL THE AGGREGATE LIABILITY OF EITHER PARTY UNDER OR IN CONNECTION WITH THIS AGREEMENT OR ITS SUBJECT MATTER, UNDER ANY LEGAL OR EQUITABLE THEORY, INCLUDING BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY, AND OTHERWISE, EXCEED ONE TIMES (1X) THE AGGREGATE AMOUNT OF REVENUES PAID OR TO BE PAID BY CUSTOMER TO COMPANY FOR SERVICES PROVIDED UNDER THIS AGREEMENT IN THE THREE (3) MONTH PERIOD PRECEDING THE EVENT GIVING RISE TO LIABILITY. CUSTOMER ACKNOWLEDGES THAT THE AMOUNTS PAYABLE HEREUNDER ARE BASED IN PART ON THESE LIMITATIONS. THE PARTIES AGREE THAT THESE LIMITATIONS WILL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.

6.4 Exceptions. THE EXCLUSIONS AND LIMITATIONS IN SECTION 6.2 AND SECTION 6.3 DO NOT APPLY TO ANY LIABILITY FOR COMPANY'S FRAUD, GROSS NEGLIGENCE, OR WILLFUL MISCONDUCT.

6.5. Data ownership. ALL CUSTOMER DATA REMAINS CUSTOMER’s, AND SHALL BE MAINTAINED BY CUSTOMER. IN NO EVENT SHALL Logiq.ai Inc. BE LIABLE FOR LOSS OF CUSTOMER DATA.

7. MISCELLANEOUS

7.1. Severability. In the event that any part of this Agreement is found to be unenforceable, the remainder shall continue in effect, to the extent permissible by law and consistent with the intent of the parties.

7.2. Relationship of the Parties. No employees, consultants, contractors or agents of one party shall, as a result of this Agreement, be considered agents, employees, partners, franchisees or joint venturers of the other party, nor do they have any authority to bind the other party by contract or otherwise to any obligation. They will not represent to the contrary, either expressly or implicitly. 7.3. Choice of Law: Jurisdiction and Venue. This Agreement will be governed by and interpreted in accordance with the laws of the State of Delaware without giving effect to its conﬂicts of law rules. Each of the Parties to this Agreement consents to the exclusive jurisdiction and venue of the state and federal courts located in Santa Clara County in the State of California.

7.4. Assignment. Neither party may assign or otherwise transfer any of its rights or obligations under this Agreement, without the prior written consent of the other party, except in the event of a merger, acquisition or sale of all or substantially all of its assets, except that neither party may assign or transfer this agreement to a direct competitor of the other party. Log Insights Quickstart guide

Please read and agree with the EULA before proceeding.

Install Docker compose

You can spin-up LOGIQ using docker-compose . Install guide for docker-compose can be found here - https://docs.docker.com/compose/install/

NOTE: the docker-compose quick-start YAML ﬁles are intended for demo and trial use only. If you want to run a production deployment, we recommend you use Kubernetes with HELM to deploy the LOGIQ stack. Please contact us at : s a l e s @ l o g i q . a i

Running LOGIQ

NOTE: LOGIQ services use approximately 2GB of memory. Please have suﬃcient memory in your system before proceeding

The ﬁrst step is to get the appropriate docker-compose YAML ﬁle from the URL below.

Download LOGIQ compose ﬁle

The latest version of LOGIQ quickstart docker compose image is 2.2.11 ⬇ Download the YAML here - https://logiqcf.s3.amazonaws.com/2.2.11/docker- compose.quickstart.yml

You are now ready to bring up the LOGIQ stack.

Download container images

$docker-compose -f docker-compose.quickstart.yml pull

Bring up the stack

$docker-compose -f docker-compose.quickstart.yml up -d

NOTE: If you have been running previous versions of LOGIQ docker-compose, you should stop and remove the existing containers by running docker-compose -f docker-compose.quickstart.yml down and remove any older docker volume via docker-compose -f docker-compose.quickstart.yml rm && docker-compose -f docker-compose.quickstart.yml rm -v

Test using LOGIQ UI

Once the LOGIQ server is up and running, the LOGIQ UI can be accessed as described above on port 80 on the server docker-compose. You will be presented with a login screen as shown below. Use ﬂ[email protected] / ﬂash-password to login

Ingesting data

Please refer to section on data ingestion for more details.

Prometheus monitoring and alerting

The LOGIQ quickstart ﬁle includes Prometheus and Alertmanager services.

Firewall ports and URLs

Ports

LOGIQ exposes the below ports

7514 - Syslog / TCP - TLS 514 - Syslog / TCP - Non TLS 2514 - RELP/Rsyslog / TCP - TLS 20514 - RELP/Rsyslog / TCP - Non TLS 9998 - Server administration web cli 9999 - REST API 8081 - GRPC ( needed for logiqctl CLI ) 80 - http 443 - https 24224/24225 - FluentForward protocol port / TLS 3000 - Grafana instance (optional), not available with quickstart

The ports used to bind are configurable via the server configuration file. See Server options for more details. K8S Quickstart guide

1 - Prerequisites

LOGIQ K8S components are made available as helm charts. The instructions below assume you are using HELM 3. Please read and agree to the EULA before proceeding.

1.1 Add LOGIQ helm repository

1.1.0 Adding LOGIQ's helm repository to your HELM repositories

helm repo add logiq-repo https://logiqai.github.io/helm-charts

The HELM repository will be named logiq-repo . For installing charts from this repository please make sure to use the repository name as the preﬁx e.g.

helm install logiq-repo/

You can now run helm search repo logiq-repo to see the available helm charts

1 $ helm search repo logiq-repo 2 NAME CHART VERSION APP VERSION DESCRIPTION 3 logiq-repo/logiq 2.2.11 2.1.11 LOGIQ Observability HE

1.1.1 Update your HELM chart

If you already added LOGIQ's HELM repository in the past, you can get updated software releases using helm repo update 1 $ helm repo update 2 $ helm search repo logiq-repo 3 NAME CHART VERSION APP VERSION DESCRIPTION 4 logiq-repo/logiq 2.2.11 2.1.11 LOGIQ Observability HE

1.2 Create namespace where LOGIQ will be deployed

NOTE: Namespace name cannot be more than 15 characters in length

kubectl create namespace logiq

This will create a namespace logiq where we will deploy the LOGIQ Log Insights stack.

If you choose a different name for the namespace, please remember to use the same namespace for the remainder of the steps

1.3 Prepare your Values YAML ﬁle

Sample YAML ﬁles for small, medium, large cluster conﬁgs can be downloaded at the links below

values.small.yaml values.small.yaml - 6KB

values.medium.yaml values.medium.yaml - 6KB values.large.yaml values.large.yaml - 6KB

These YAML ﬁles can be used for deployment with -f parameter as shown below in the description.

1 helm install logiq --namespace logiq \ 2 --set global.persistence.storageClass= logiq-repo/logi

Please refer to Section 3.10 for sizing your LOGIQ cluster as speciﬁed in these YAML ﬁles.

1.4 Latest image tags

Image Tag

logiq-ﬂash 2.1.11.10

coffee 2.1.7

HELM 2.2.11

2. Install LOGIQ

1 helm install logiq --namespace logiq \ 2 --set global.persistence.storageClass= logiq-repo/logi

This will install LOGIQ and expose the LOGIQ services and UI on the ingress IP. If you plan to use an AWS S3 bucket, please refer to section 3.2 before running this step. Please refer to Section 3.4 for details about storage class. Service ports are described in the Port details section. You should now be able to go to http://ingress-ip/ The default login and password to use is [email protected] and

flash-password . You can change these in the UI once logged in. HELM chart can override the default admin settings as well. See section 3.7 on customizing the admin settings

Logiq Insights Login UI

LOGIQ server provides Ingest, log tailing, data indexing, query, and search capabilities. Besides the web-based UI, LOGIQ also offers logiqctl, LOGIQ CLI for accessing the above features.

3 Customizing the deployment

3.1 Enabling https for the UI

1 helm install logiq --namespace logiq \ 2 --set global.domain=logiq.my-domain.com \ 3 --set ingress.tlsEnabled=true \ 4 --set kubernetes-ingress.controller.defaultTLSSecret.enabled=true \ 5 --set global.persistence.storageClass= logiq-repo/logi

You should now be able to login to LOGIQ UI at your domain using https://logiq.my-domain.com that you set in the ingress after you have updated your DNS server to point to the Ingress controller service IP

The default login and password to use is [email protected] and

flash-password . You can change these in the UI once logged in.

The logiq.my-domain.com also fronts all the LOGIQ service ports as described in the port details section.

HELM Option Description

DNS domain where the LOGIQ service global.domain running. This is required for HTTPS

Enable the ingress controller to front H ingress.tlsEnabled services

kubernetes- Specify if a default certiﬁcate is enab ingress.controller.defaultTLSSecret.enabled ingress gateway

Specify the name of a TLS Secret for gateway. If this is not speciﬁed, a sec kubernetes- automatically generated of option ingress.controller.defaultTLSSecret.secret kubernetes- ingress.controller.defaultTLSSec above is enabled.

3.1.1 Passing an ingress secret If you want to pass your own ingress secret, you can do so when installing the HELM chart

3.2 Using an AWS S3 bucket

Depending on your requirements, you may want to host your storage in your own K8S cluster or create a bucket in a cloud provider like AWS.

Please note that cloud providers may charge data transfer costs between regions. It is important that the LOGIQ cluster be deployed in the same region where the S3 bucket is hosted

3.2.1 Create an access/secret key pair for creating and managing your bucket

Go to AWS IAM console and create an access key and secret key that can be used to create your bucket and manage access to the bucket for writing and reading your log ﬁles

3.2.2 Deploy the LOGIQ helm in gateway mode

Make sure to pass your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and give a bucket name. The S3 gateway acts as a caching gateway and helps reduce API costs. Create a bucket in AWS s3 with a unique bucket name in the region where you plan to host the deployment.

You do not need to create the bucket, we will automatically provision it for you. Just provide the bucket name and access credentials in the step below. If the bucket already exists, LOGIQ will use it. Check to make sure the access and secret key work with it. Additionally, provide a valid amazon service endpoint for s3 else the conﬁg defaults to https://s3.us-east-1.amazonaws.com

1 helm install logiq --namespace logiq --set global.domain=logiq.my-domain.c 2 --set global.environment.s3_bucket= \ 3 --set global.environment.awsServiceEndpoint=https://s3..amazonaws. 4 --set global.environment.s3_region= \ 5 --set global.environment.AWS_ACCESS_KEY_ID= \ 6 --set global.environment.AWS_SECRET_ACCESS_KEY= \ 7 --set global.persistence.storageClass= logiq-repo/logi

HELM Option Description Defaults

This helm option speciﬁes the supported cloudProvider that is hosting the S3 global.cloudProvider aws compatible bucket. Right now only aws is supported.

Name of the S3 bucket in global.environment.s3_bucket logiq AWS

S3 Service endpoint : https://s3 global.environment.awsServiceEndpoint https://s3.** 1.amazon **.amazonaws.com

AWS Access key for global.environment.AWS_ACCESS_KEY_ID No default accessing the bucket

AWS Secret key for global.environment.AWS_SECRET_ACCESS_KEY No default accessing the bucket

AWS Region where the global.environment.s3_region us-east-1 bucket is hosted S3 providers may have restrictions on bucket names for e.g. AWS S3 bucket names are globally unique.

3.3 Install LOGIQ server certiﬁcates and Client CA [OPTIONAL]

LOGIQ supports TLS for all ingest. We also enable non-TLS ports by default. It is however recommended that non-TLS ports not be used unless running in a secure VPC or cluster. The certiﬁcates can be provided to the cluster using K8S secrets. Replace the template sections below with your Base64 encoded secret ﬁles.

If you skip this step, the LOGIQ server automatically generates a ca and a pair of client and server certiﬁcates for you to use. you can get them from the ingest server pods under the folder /flash/certs

1 apiVersion: v1 2 kind: Secret 3 metadata: 4 name: logiq-certs 5 type: Opaque 6 data: 7 ca.crt: {{ .Files.Get "certs/ca.crt.b64" }} 8 syslog.crt: {{ .Files.Get "certs/syslog.crt.b64" }} 9 syslog.key: {{ .Files.Get "certs/syslog.key.b64" }}

Save the secret ﬁle e.g. logiq-certs.yaml . Proceed to install the secret in the same namespace where you want to deploy LOGIQ

The secret can now be passed into the LOGIQ deployment

1 helm install logiq --namespace logiq --set global.domain=logiq.my-domain.c 2 --set logiq-flash.secrets_name=logiq-certs \ 3 --set global.persistence.storageClass= logiq-repo/logi HELM Option Description Defaults

logiq- TLS certiﬁcate key pair and CA cert for TLS No flash.secrets_name transport default

3.4 Changing the storage class

If you are planning on using a speciﬁc storage class for your volumes, you can customize it for the LOGIQ deployment. By default, LOGIQ uses the standard storage class

It is quite possible that your environment may use a different storage class name for the provisioner. In that case please use the appropriate storage class name. E.g. if a user creates a storage class ebs-volume for the EBS provisioner

for their cluster, you can use ebs-volume instead of gp2 as suggested below

Cloud Provider K8S StorageClassName Default Provisioner

AWS gp2 EBS

Azure standard azure-disc

GCP standard pd-standard

Digital Ocean do-block-storage Block Storage Volume

Oracle oci Block Volume

1 helm upgrade --namespace logiq \ 2 --set global.persistence.storageClass= \ 3 logiq logiq-repo/logiq

3.5 Using external AWS RDS Postgres database instance To use external AWS RDS Postgres database for your LOGIQ deployment, execute the following command.

1 helm install logiq --namespace logiq \ 2 --set global.chart.postgres=false \ 3 --set global.environment.postgres_host= \ 4 --set global.environment.postgres_user= \ 5 --set global.environment.postgres_password= \ 6 --set global.persistence.storageClass= logiq-repo/logi

HELM Option Description Default

Deploy Postgres which is needed for LOGIQ metadata. Set this to global.chart.postgres true false if an external Postgres cluster is being used

Host IP/DNS for external global.environment.postgres_host postgres Postgres

global.environment.postgres_user Postgres admin user postgres

global.environment.postgres_password Postgres admin user password postgres

global.environment.postgres_port Host Port for external Postgres 5432

While conﬁguring RDS, create a new parameter group that sets autoVaccum to true or the value "1", associate this parameter group to your RDS instance.

Auto vacuum automates the execution of VACUUM and ANALYZE (to gather statistics) commands. Auto vacuum checks for bloated tables in the database and reclaims the space for reuse.

3.6 Upload LOGIQ professional license The deployment described above offers 30 days trial license. Send an e-mail to [email protected] to obtain a professional license. After obtaining the license, use the logiqctl tool to apply the license to the deployment. Please refer to logiqctl details at https://logiqctl.logiq.ai/. You will need API-token from LOGIQ UI as shown below

Logiq Insights Login Api-token

1 Setup your LOGIQ Cluster endpoint 2 - logiqctl config set-cluster logiq.my-domain.com 3 4 Sets a logiq ui api token 5 - logiqctl config set-token api_token 6 7 Upload your LOGIQ deployment license 8 - logiqctl license set -f=license.jws 9 10 View License information 11 - logiqctl license get

3.7 Customize Admin account 1 helm install logiq --namespace logiq \ 2 --set global.environment.admin_name="LOGIQ Administrator" \ 3 --set global.environment.admin_password="admin_password" \ 4 --set global.environment.admin_email="[email protected]" \ 5 --set global.persistence.storageClass= logiq-repo/logi

HELM Option Description Default

LOGIQ Administrator ﬂash- global.environment.admin_name name [email protected]

LOGIQ Administrator global.environment.admin_password ﬂash-password password

LOGIQ Administrator e- ﬂash- global.environment.admin_email mail [email protected]

3.8 Using external Redis instance

To use external Redis for your LOGIQ deployment, execute the following command.

NOTE: At this time LOGIQ only supports connecting to a Redis cluster in a local VPC without authentication. If you are using an AWS Elasticache instance, do not turn on encryption-in-transit or cluster mode.

1 helm install logiq --namespace logiq \ 2 --set global.chart.redis=false \ 3 --set global.environment.redis_host= \ 4 --set global.persistence.storageClass= logiq-repo/logi

HELM Option Description Default Deploy Redis which is needed for log global.chart.redis tailing. Set this to false if an external true Redis cluster is being used

redis- global.environment.redis_host Host IP/DNS of the external Redis cluster master

Host Port where external Redis service is global.environment.redis_port 6379 exposed

3.9 Conﬁguring cluster id

When deploying LOGIQ, conﬁgure the cluster id to monitor your own LOGIQ deployment. For details about the cluster_id refer to section Managing multiple K8S clusters

1 helm install logiq --namespace logiq \ 2 --set global.environment.cluster_id= \ 3 --set global.persistence.storageClass= logiq-repo/logi

HELM Option Description Default

Cluster Id being used for the K8S cluster global.environment.cluster_id running LOGIQ. See Section on Managing LOGIQ multiple K8S clusters for more details.

3.10 Sizing your LOGIQ cluster

When deploying LOGIQ, size your infrastructure to provide appropriate vcpu and memory requirements. We recommened the following minimum size for small. medium and large cluster speciﬁcation from Section 1.3 values yaml ﬁles.

LOGIQ Cluster vCPU Memory NodeCount

small 12 32 gb 3 medium 20 56 gb 5

large 32 88 gb 8

3.11 NodePort/ClusterIP/LoadBalancer

The service type conﬁgurations are exposed in values.yaml as below

1 flash-coffee: 2 service: 3 type: ClusterIP 4 logiq-flash: 5 service: 6 type: NodePort 7 kubernetes-ingress: 8 controller: 9 service: 10 type: LoadBalancer 11

For e.g. if you are running on bare-metal and want an external LB to front LOGIQ, conﬁgure all services as NodePort

1 helm install logiq -n logiq -f values.yaml \ 2 --set flash-coffee.service.type=NodePort \ 3 --set logiq-flash.service.type=NodePort \ 4 --set kubernetes-ingress.controller.service.type=NodePort \ 5 logiq-repo/logiq

3.12 Using Node Selectors

The LOGIQ stack deployment can be optimized using node labels and node selectors to place various components of the stack optimally

logiq.ai/node=ingest The node label logiq.ai/node above can be used to control the placement of ingest pods for log data into ingest optimized nodes. This allows for managing cost and instance sizing effectively.

The various nodeSelectors are deﬁned in the globals section of the values.yaml ﬁle

1 globals: 2 nodeSelectors: 3 enabled: true 4 ingest: ingest 5 infra: common 6 other: common 7 db: db 8 cache: cache 9 ingest_sync: sync

In the example above, there are different node pools being used - ingest , common , db ,

cache and sync

Node selectors are enabled by setting enabled to true for

globals.nodeSelectors

3.13 Installing Grafana

The LOGIQ stack includes Grafana as part of the deployment as an optional component. To enable Grafana in your cluster, follow the steps below

1 helm upgrade --install logiq --namespace logiq \ 2 --set global.chart.grafana=true \ 3 --set global.persistence.storageClass= logiq-repo/logi The Grafana instance is exposed at port 3000 on the ingress controller. The deployed Grafana instance uses the same credentials as the LOGIQ UI

4 Teardown

If and when you want to decommission the installation using the following commands

1 helm delete logiq --namespace logiq 2 helm repo remove logiq-repo 3 kubectl delete namespace logiq

If you followed the installation steps in section 3.1 - Using an AWS S3 bucket, you may want to delete the s3 bucket that was speciﬁed at deployment time. AWS Quickstart guide

Overview

LOGIQ can be deployed on AWS in a single AMI instance in a 1-Click fashion using our CloudFormation template and the LOGIQ AMI from the Amazon Marketplace. Please read and agree EULA before proceeding.

All the resources required to create and conﬁgure LOGIQ on AWS are taken care by the template. All you need to do is provide a few simple input parameters.

The CloudFormation template can be found in the software subscription at the AWS marketplace

https://aws.amazon.com/marketplace/pp/B083ZMYQNV

Please note that using the Marketplace AMI is not free and you will be charged per the marketplace published rates for LOGIQ

LOGIQ UI credentials

After the Cloud formation template is complete, it may take several minutes for the UI to be available on the AMI.

The deployment exposes the UI on an http port by default. You can install an ELB to front the UI via https. This is the recommended production setup. Once the LOGIQ instance is created, you can login to the instance using the below credentials user: [email protected] password: - e.g. if CloudFormation stack is called Foo and bucket is called Bar the password is Foo-Bar

AWS Quickstart video

https://www.youtube.com/watch? LOGIQ CloudFormation Deployment v=IDmJOF9y5Ac

Also see section on AMI using CloudFormation for more details.

Ingesting data

Please refer to section on data ingestion for more details. Ingesting data

Logstash

Syslog output plugin

1 input { 2 3 file { 4 path => "/var/log/syslog" 5 type => "syslog" 6 start_position => "beginning" 7 } 8 9 filter { 10 uuid { 11 target => "uuid" 12 } 13 } 14 15 output { 16 syslog { appname => "my-awesome-app" 17 host => "logiq-server-dns.my-domain.com" 18 protocol => "ssl-tcp" 19 msgid => "%{uuid}" 20 ssl_cert => "client.crt" 21 ssl_key => "client.key" 22 ssl_cacert => "logiq.crt" 23 ssl_verify => true 24 port => "7514" 25 rfc => "rfc5424" 26 id => "%{uuid}" 27 } 28 stdout { codec => rubydebug } 29 }

NOTE: Change "host" , "appname", "ssl_cert", "ssl_key", "ssl_cacert" above to suit your conﬁguration

HTTP output plugin 1 output { 2 http { 3 url => "https://logiq-dns-or-ip/v1/json_batch" 4 headers => { "Authorization" => "Bearer " } 5 http_method => "post" 6 format => "json_batch" 7 content_type => "json_batch" 8 pool_max => 300 9 pool_max_per_route => 100 10 socket_timeout => 60 11 } 12 }

You can additionally control the data organization by specifying additional ﬁelds

1 filter { 2 mutate { 3 add_field => { "cluster_id" => "demo-http-test" } 4 add_field => { "namespace" => "namespace_name" } 5 add_field => { "app_name" => "application_name" } 6 add_field => { "proc_id" => "process_or_pod_identifier" } 7 } 8 }

You can generate the Bearer token using logiqctl

$logiqctl get httpingestkey

Rsyslogd

Please see below on how to conﬁgure Rsyslog to send to LOGIQ server. Rsyslog can send data to LOGIQ using either TCP transport or RELP transport. The RELP module for Rsyslog is called omrelp and for the TCP forward is called omfwd LOGIQ strongly recommends sending data using the RELP transport to ensure packets are not lost or dropped. RELP relies on acknowledgements from the receiver to make sure packet is delivered. LOGIQ, for its part only sends the acknowledgements back once the data is written to persistent store.

Using omfwd

Update the syslog conﬁg in /etc/rsyslog.conf or /etc/rsyslog.d/50-default.conf

1 *.* action(type="omfwd" 2 queue.type="LinkedList" 3 action.resumeRetryCount="-1" 4 queue.size="10000" 5 queue.saveonshutdown="on" 6 target="logiq-server-syslog-host" Port="514" Protocol="tcp" 7 )

Using omrelp

Installation rsyslog RELP modules rsyslog is installed by default in most modern OS's, rsyslog needs the omrelp module to send data to a RELP aware endpoint such as LOGIQ. To enable RELP install packages listed below

rsyslog-relp, enables RELP protocol for rsyslog rsyslog-gnutls, enables rsyslog to communicate over a secure socket

1 sudo apt update 2 sudo apt install rsyslog-gnutls rsyslog-relp

For Redhat/CentOS/Fedora, use yum to install yum install rsyslog-gnutls rsyslog-relp

Conﬁguring rsyslog (TLS)

Update the syslog conﬁg in /etc/rsyslog.conf or /etc/rsyslog.d/50-default.conf

1 module(load="omrelp") 2 action(type="omrelp" 3 target="logiq-server-relp-host" 4 port="2514" 5 tls="on" 6 tls.caCert="/etc/ssl/LOGIQ/certs/LOGIQ.crt" 7 tls.myCert="/etc/ssl/LOGIQ/certs/client.crt" 8 tls.myPrivKey="/etc/ssl/LOGIQ/certs/client.key" 9 tls.authMode="fingerprint" 10 tls.PermittedPeer=["SHA1:BF:46:AB:9F:A3:77:46:AF:6B:D2:EC:A4:30:72 11 action.reportSuspensionContinuation="on" 12 action.resumeRetryCount="-1" 13 action.resumeInterval="10" 14 queue.type="fixedArray" 15 queue.size="250000" 16 queue.dequeueBatchSize="1024" 17 queue.workerThreads="4" 18 queue.workerThreadMinimumMessages="50000" 19 queue.spoolDirectory="/var/log/rsyslog" 20 queue.fileName="XXX_sock" 21 queue.maxFileSize="16m" 22 queue.maxDiskSpace="2G" 23 queue.highWaterMark="200000" 24 queue.lowWaterMark="100000" 25 queue.checkpointInterval="30000" 26 queue.saveOnShutdown="on" 27 queue.timeoutEnqueue="1" 28 )

NOTE: Change "target", "port", tls.caCert" , "tls.myCert", "tls.myPrivKey", "tls.PermitterPeer" above to suit your configuration. For non TLS config, set "tls" parameter as "off" and remove all tls.* parameters from above config file. E.g. of target=ec2-34-213-110-235.us- west-2.compute.amazonaws.com

Conﬁguring rsyslog (non-TLS) Update the syslog conﬁg in /etc/rsyslog.conf or /etc/rsyslog.d/50-default.conf

Rsyslog non-TLS uses port 20514 vs TLS which uses port 2514

1 module(load="omrelp") 2 action(type="omrelp" 3 target="logiq-server-relp-host" 4 port="20514" 5 tls="off" 6 action.reportSuspensionContinuation="on" 7 action.resumeRetryCount="-1" 8 action.resumeInterval="10" 9 queue.type="fixedArray" 10 queue.size="250000" 11 queue.dequeueBatchSize="1024" 12 queue.workerThreads="4" 13 queue.workerThreadMinimumMessages="50000" 14 queue.spoolDirectory="/var/log/rsyslog" 15 queue.fileName="XXX_sock" 16 queue.maxFileSize="16m" 17 queue.maxDiskSpace="2G" 18 queue.highWaterMark="200000" 19 queue.lowWaterMark="100000" 20 queue.checkpointInterval="30000" 21 queue.saveOnShutdown="on" 22 queue.timeoutEnqueue="1" 23 )

Fluentd K8S

If you are running a K8S cluster, you can use ﬂuentd to send data to the LOGIQ server. Please see below for instructions

Managing multiple K8S clusters in a single LOGIQ instance When deploying fluentd daemonset on K8S clusters, we recommend you use the fluentd daemon set container provided by LOGIQ. It is available at https://hub.docker.com/repository/docker/logiqai/fluentd-remote-syslog. It allows the administrator to pass a human readable CLUSTER_ID or cluster identifier with all the log data.

Providing a CLUSTER_ID allows LOGIQ to separate namespaces that may be conﬂicting in two separate K8S clusters.

It is also easier for the administrator to use human readable names vs LOGIQ using uuid's etc that it detects from the incoming stream.

Running the ﬂuentd daemonset

Clone the repository to get the kubectl YAML ﬁles to start your daemonset

git clone https://bitbucket.org/logiqcloud/client-integrations.git

The ﬁles needed are under folder fluentd

1 $ cd client-integrations/ 2 $ cd fluentd/ 3 $ ls -la 4 total 32 5 drwxr-xr-x 6 user staff 192 Oct 30 14:47 . 6 drwxr-xr-x 7 user staff 224 Oct 30 14:47 .. 7 -rw-r--r-- 1 user staff 645 Oct 30 14:47 README.md 8 -rw-r--r-- 1 user staff 1373 Oct 30 14:47 fluentd-logiq.yaml 9 -rw-r--r-- 1 user staff 1373 Oct 30 14:47 fluentd-logiq_non_tls.yaml 10 -rw-r--r-- 1 user staff 590 Oct 30 14:47 fluentd_rbac.yaml 11 -rw-r--r-- 1 user staff 210 Oct 30 14:47 secret.yaml

TLS Mode Edit the fluentd/secret.yaml to include your CA and Client pub/private keys in base64 encoded format

Edit the fluentd/fluentd-logiq.yaml and add your LOGIQ cluster IP/DNS. Also conﬁgure your CLUSTER_ID (e.g. RC, Prod, Dev-Test, QA).

1 .... 2 - env: 3 - name: SYSLOG_HOST 4 value: "YOUR_LOGIQ_SERVER_IP" 5 - name: CLUSTER_ID 6 value: "YOUR_CLUSTER_ID" 7 ....

Run the kubectl commands to create the kube-logging namespace. You can choose a different namespace as well. In case a different namespace is used please edit the YAML ﬁles to set the correct namespace before applying them

1 kubectl create namespace kube-logging 2 kubectl apply -f fluentd_rbac.yaml 3 kubectl apply -f secret.yaml 4 kubectl apply -f fluentd-logiq.yaml

Non-TLS Mode

Edit the fluentd/fluentd-logiq_non_tls.yaml and add your LOGIQ cluster IP/DNS. Also conﬁgure your CLUSTER_ID (e.g. RC, Prod, Dev-Test, QA)

1 .... 2 - env: 3 - name: SYSLOG_HOST 4 value: "YOUR_LOGIQ_SERVER_IP" 5 - name: CLUSTER_ID 6 value: "YOUR_CLUSTER_ID" 7 .... Run the kubectl commands to create the kube-logging namespace. You can choose a different namespace as well. In case a different namespace is used please edit the YAML ﬁles to set the correct namespace before applying them

1 kubectl create namespace kube-logging 2 kubectl apply -f fluentd_rbac.yaml 3 kubectl apply -f fluentd-logiq_non_tls.yaml

Fluent-bit K8S

If you are running a K8S cluster, you can use ﬂuent-bit to send data to the LOGIQ server. Please see below for instructions

Managing multiple K8S clusters in a single LOGIQ instance

LOGIQ has provided its own fluent-bit daemon for deploying on K8S clusters. It is available at https://bitbucket.org/logiqcloud/client-integrations/src/master/fluent-bit/. It allows the administrator to pass a human readable CLUSTER_ID or cluster identifier with all the log data.

Providing a CLUSTER_ID allows LOGIQ to separate namespaces that may be conﬂicting in two separate K8S clusters.

It is also easier for the administrator to use human readable names vs LOGIQ using uuid's etc that it detects from the incoming stream.

Running the ﬂuent-bit daemonset

Clone the repository to get the kubectl YAML ﬁles to start your daemonset git clone https://bitbucket.org/logiqcloud/client-integrations.git

The ﬁles needed are under folder fluent-bit

1 $ cd client-integrations/ 2 $ cd fluentd-bit/ 3 $ ls -la 4 total 64 5 drwxr-xr-x 8 user staff 256 Aug 9 05:47 . 6 drwxr-xr-x 9 user staff 288 Aug 9 05:20 .. 7 -rw-r--r-- 1 user staff 2446 Aug 9 05:47 README.md 8 -rw-r--r-- 1 user staff 8688 Aug 9 05:32 fluent-bit-config-logiq-forwa 9 -rw-r--r-- 1 user staff 1670 Aug 9 05:29 fluent-bit-daemonset-logiq-fo 10 -rw-r--r-- 1 user staff 269 Aug 9 05:26 fluent-bit-role-binding.yaml 11 -rw-r--r-- 1 user staff 194 Aug 9 04:49 fluent-bit-role.yaml 12 -rw-r--r-- 1 user staff 86 Aug 9 05:25 fluent-bit-service-account.ya

To get started run the following commands to create the namespace, service account and role setup:

1 $ kubectl create namespace logiq-logging 2 $ kubectl create -f fluent-bit-service-account.yaml 3 $ kubectl create -f fluent-bit-role-binding.yaml 4 $ kubectl create -f fluent-bit-role.yaml

Fluent Bit to LOGIQ

The next step is to create a ConﬁgMap that will be used by the Fluent Bit DaemonSet:

$ kubectl create -f fluent-bit-config-logiq-forward.yml

Fluent Bit DaemonSet is ready to be used with LOGIQ on a regular Kubernetes Cluster, conﬁgure the following in deamonset ﬂuent-bit-daemonset-logiq-forward.yml name: LOGIQ_HOST value: "YOUR_LOGIQ_SERVER_IP" name: LOGIQ_PORT value: "24224" name: CLUSTER_ID value: "YOUR_CLUSTER_ID"

For Kubernetes version < 1.17, please change the apiVersion: "extensions/v1beta1" from "apps/v1" and remove selector attached to DaemonSet spec selector: matchLabels: k8s- app: ﬂuent-bit-logging

$ kubectl create -f fluent-bit-daemonset-logiq-forward.yml

Docker Syslog log driver

Using the docker syslog driver to send logs to LOGIQ is quite simple. Details about the docker syslog driver can be found here https://docs.docker.com/conﬁg/containers/logging/syslog/

LOGIQ supports both TLS and non TLS syslog ports using TCP.

Required values

Following ﬁelds are required options to be passed to the logdriver

tag - User a human readable string for better readability otherwise the ﬁrst 12 chars of the container id will be used syslog-format=rfc3164

syslog-address

Optional values syslog-tls-cert syslog-tls-key syslog-tls-ca-cert syslog-tls-skip-verify

Using TCP and non TLS port

Sending data from docker to LOGIQ using TCP and non TLS port can be done as below. In the example below, we are going to run a mysql container and have all logs go to LOGIQ server hosted at logiqserver-devtest.example.com

1 docker run --log-driver syslog \ 2 --log-opt syslog-address=tcp://logiqserver-devtest.example.com:514 \ 3 --log-opt syslog-format=rfc3164 --log-opt tag=mysql --name mysql3 -d mysql

Using TCP and TLS port

When using to connect to LOGIQ TLS port in a secured setup, pass the client certiﬁcates to connect to the server

1 docker run --log-driver syslog \ 2 --log-opt syslog-address=tcp://logiqserver-devtest.example.com:514 \ 3 --log-opt syslog-tls-cert=client.pem --log-opt syslog-tls-key=key.pem \ 4 --log-opt syslog-tls-ca-cert=ca.pem --log-opt syslog-format=rfc3164 \ 5 --log-opt tag=mysql --name mysql3 -d mysql

Fluentd conﬁguration

Fluentd out-forward Buffered Output plugin forwards events to other fluentd nodes. Logiq has the capability to act as one of the fluentd nodes. The below code block defines the minimal changes to be added to fluentd configuration to start sending log events to flash. It is important to have the transformations while sending the data to Logiq.

1 2 @type tail 3 path /var/log/*.log 4 pos_file /var/log/tty.txt.pos 5 6 @type none 7 8 9 10 11 12 @type record_transformer 13 enable_ruby 14 15 hostname "#{Socket.gethostname}" 16 namespace "#{Socket.gethostname}" 17 cluster_id "hadoop-master" 18 log ${record["message"]} 19 20 21 22 23 24 25 @type forward 26 send_timeout 10s 27 recover_wait 10s 28 hard_timeout 20s 29 30 31 32 @type msgpack 33 time_type unixtime 34 utc 35 36 37 @type memory 38 timekey 2s 39 timekey_wait 1s 40 flush_mode interval 41 flush_interval 1s 42 retry_max_interval 2s 43 retry_timeout 10s 44 45 46 47 48 name logiq 49 host development.logiq.ai 50 port 24224 51 weight 100 52 53 54 55 @type secondary_file 56 directory /var/log/forward-failed 57 58

Fluent-bit conﬁguration

Forward is the protocol used by Fluentd to route messages between peers. The forward output plugin allows to provide interoperability between compatible systems, Logiq being one.

The below code block defines the minimal changes to be added to the fluent-bit configuration to start sending log events to flash.

1 [INPUT] 2 Name tail 3 Path /var/log/* 4 Path_Key On 5 Tag logiq 6 Buffer_Max_Size 1024k 7 Read_from_Head On 8 9 [FILTER] 10 Name record_modifier 11 Match logiq 12 Record cluster_id flash 13 14 [FILTER] 15 Name record_modifier 16 Match logiq 17 Record namespace mesos 18 19 [FILTER] 20 Name record_modifier 21 Match logiq 22 Record app_name fluentbit 23 24 25 [OUTPUT] 26 Name forward 27 Match logiq 28 host localhost 29 port 24224 30 Data extraction

Grok basics

Grok works by combining text patterns into complex pattern expressions that can match your incoming log data stream.

The typical syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The SYNTAX is the pattern identiﬁer that will match the text. For example,

[email protected] will be matched by the EMAILADDRESS pattern and 10.0.1.22 will be matched by the IPV4 pattern. The syntax can further be broken down into subpatterns composed of regular expressions.

The SEMANTIC is the identiﬁer you give to the piece of text being matched. For example,

[email protected] is an e-mail address so you could call it email . Further, a string

10.0.1.22 might identify the sender making a request.

Groks patterns offer a powerful, yet simple mechanism to compose patterns for complex log matching. LOGIQ's server is written in Go and uses the popular Grokky library for it Grok implementation

Composing Grok expressions

Here is a simple example of composing grok expressions. We are going to compose a regex for hostnames and a regex for matching user names to create the Grok for email address matching

1 "HOSTNAME" : `\b[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.[0-9A-Za-z][0-9A-Za-z-]{ 2 "EMAILLOCALPART" : `[a-zA-Z][a-zA-Z0-9_.+-=:]+` 3 4 "EMAILADDRESS": `%{EMAILLOCALPART}@%{HOSTNAME}`

Grok matching rules

Grok patterns can be applied in incoming logs data via matching rules. The matching rules allow for targeting expressions to speciﬁc applications, and namespaces. Let us look at an example below.

Example: Applying Grok matching to HTTPD logs

In the example below, the LOGIQ engine will apply the combined apache log format grok extraction rule to all application names that being with httpd or Httpd and running on namespaces that being with webservers-.

Omitting either the applications or the namespaces keyword or both, excludes matching on that criteria. For e.g. omitting applications means the grok pattern is applied to data from all applications from the webservers-.* namespaces

1 - name: combined_apache_log 2 type: extract 3 format: "%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent} 4 applications: [h|H]ttpd.* 5 namespaces: webservers-.*

The above rule would result in LOGIQ parsing out data as shown below. All the extracted values will automatically be made available as facets for ﬁltering search data

1 { 2 "AppName": "finite-test-engine", 3 "Facility": "3", 4 "FacilityString": "system daemon", 5 "Hostname": "watcher_test_8", 6 "Message": "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] \"GET /index. 7 "MsgID": "1g4Jt60f78HOYBk26BiJREdwBEG", 8 "Priority": "29", 9 "ProcID": "70958", 10 "Sender": "::1", 11 "Severity": "5", 12 "SeverityString": "notice", 13 "StructuredData": [ 14 { 15 "key": "auth", 16 "values": [ 17 "frank" 18 ] 19 }, 20 { 21 "key": "bytes", 22 "values": [ 23 "2481" 24 ] 25 }, 26 { 27 "key": "clientip", 28 "values": [ 29 "127.0.0.1" 30 ] 31 }, 32 { 33 "key": "httpversion", 34 "values": [ 35 "1.0" 36 ] 37 }, 38 { 39 "key": "ip", 40 "values": [ 41 "127.0.0.1" 42 ] 43 }, 44 { 45 "key": "uri", 46 "values": [ 47 "http://www.example.com/end.html" 48 ] 49 }, 50 { 51 "key": "ident", 52 "values": [ 53 "-" 54 ] 55 }, 56 { 57 "key": "request", 58 "values": [ 59 "/index.html" 60 ] 61 }, 62 { 63 "key": "response", 64 "values": [ 65 "200" 66 ] 67 }, 68 { 69 "key": "verb", 70 "values": [ 71 "GET" 72 ] 73 } 74 ], 75 "Timestamp": "2020-08-14T01:22:18Z", 76 "Namespace": "watcher_test_8", 77 "MsTimestamp": "1597368138" 78 }

Specifying Grok matching rules

The LOGIQ ingest server can watch a rules directory for rules ﬁles as described above. The server can handle new rules being added to the system dynamically without any downtime.

In a Kubernetes environment, the rules can be added to a running server via ConﬁgMaps Timestamp handling

Which timestamp to use?

Incoming log data streams can have timestamps deﬁned in the following ways

1. Sending agent sends a timestamp 2. Log data has its own timestamp 3. Ingest layer for e.g. LOGIQ adds its own timestamp 4. Log data has a non-standard timestamp format

LOGIQ handles the timestamps in the following order

1. Use the sending agent timestamp 2. Extract any timestamp in log data automatically that is non-ambiguous 3. If a user-deﬁned timestamp extraction rule is provided, use the extraction rule to get the timestamp

User-deﬁned timestamp rules

Users can specify timestamp extraction rules for log data using the LOGIQ data manipulation capabilities. Timestamps are handled by timestamp rules that are deﬁned as follows

1 - name: custom_timestamp 2 namespaces: http.* 3 applications: .* 4 type: timestamp 5 format: "(?P[0-9]{1,4})/(?P[0-9]{1,2})/(?P[0-9]{1,2})

Using the example rule above, a logline such as the one below will result in the proper timestamp being parsed and extracted from the log line. "2009/06/29 13:30:10.956+05:30 something interesting happened"

LOGIQ Monitoring Conﬁguring Prometheus

LOGIQ monitoring is powered by Prometheus. The steps below assume that your LOGIQ UI is up and running. To connect to your prometheus setup, head over the Data Sources in settings

Data Sources

We are going to click on the New Data Source button Next, let us select the prometheus Data source from the data sources screen

LOGIQ supports many data sources, LOGIQ's provides tighter integration with Prometheus and uses its time series database for monitoring.

Enter your prometheus data source URL and check the connection by clicking on Test Connection You are now ready to visualize your metrics Prometheus Data source

LOGIQ's Prometheus integration allows querying data from the query editor. Just enter a PromQL query and see your data and create visualizations in an instant

Query language

The query language is nothing but the PromQL expression and any additional parameters that would be sent to the Prometheus Query API. The query starts with a query= preﬁx and ends with optional URL parameters that are sent to the query API

Let's look at an example

query=go_gc_duration_seconds&duration=15m&step=60

In the above query, we are looking for the go_gc_duration_seconds metric, sampled at 60 second intervals and duration for which data is needed which is the last 15 minutes.

The Prometheus Query API expects start_time and end_time to be provided in queries.

LOGIQ has a simpliﬁed duration syntax that is compatible with the Prometheus Query API.

LOGIQ translates the duration values to appropriate start and end times before issuing the Query API call

Using the duration syntax allows you to construct dynamic time range queries without specifying start or end time.

1 example: instant query 2 query=http_requests_total 3 4 example: range query 5 query=http_requests_total&start=2018-01-20T00:00:00.000Z&end=2018-01-25T 6 7 example: until now range query 8 query=http_requests_total&start=2018-01-20T00:00:00.000Z&step=60s 9 query=http_requests_total&start=2018-01-20T00:00:00.000Z&end=now&step=60 10 11 example: using duration 12 # end is assumed now, start is end-duration 13 query=http_requests_total&duration=1h5m10s20ms&step=60s 14 # end is (start + duration) 15 query=http_requests_total&start=2018-01-20T00:00:00.000Z&duration=1h&ste 16 # start is (end - duration), end is now 17 query=http_requests_total&duration=1h&step=60s 18 #start is (end - duration), end is now 19 query=http_requests_total&end=2018-01-20T00:00:00.000Z&duration=1h&step=

PromQL compatibility

LOGIQ's query language is 100% compatible with PromQL, primarily because all query expressions are translated to appropriate Prometheus Query / Query range API calls.

Let's look at a more complicated expression below

query=(100-(avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))&dura In the above example we are using several prometheus constructs

Label based ﬁltering e.g. mode="idle"

Function such as irate , avg

Using the vector syntax [5m]

Mathematical operator like * / - JSON Data source

Create the JSON Data source

The ﬁrst step in to create the data source and provide basic auth credentials. Note that basic auth credentials are optional and you can provide a bearer token if that is your means of authenticating against the API

Creating a JSON data source

Writing queries

In the query editor, select the JSON data source created above and enter the query parameters. The query parameters use the YAML syntax. For E.g.

Providing HTTP Options

The following HTTP options are used for sending a query

The URL parameter is the only required parameter

url - This is the URL where the RESTful API is exposed

method - the HTTP method to use (default: get )

headers - a dictionary of headers to send with the request

auth - basic auth username/password (should be passed as an array:

[username, password] )

params - a dictionary of query string parameters to add to the URL

data - a dictionary of values to use as the request body

json - same as data except that it’s being converted to JSON

Filtering response data: path and ﬁelds The response data can be ﬁltered by specifying the path and fields parameters. The

path ﬁlter allows accessing attributes within the response for e.g. if a key foo in the response contains rows of objects you want to access, specifying path foo will convert each of the objects into rows.

In the example below, we are then selecting fields volumeInfo.authors, volumeInfo.title, volumeInfo.publisher and accessInfo.webReaderLink

1 url: https://www.googleapis.com/books/v1/volumes?q=isbn:0747532699 2 path: items 3 fields: ["volumeInfo.authors","volumeInfo.title","volumeInfo.publisher","a

The resulting data from the above query is a nicely formatted table that can be searched in LOGIQ or made available as a widget in a dashboard Elasticsearch Data source

Create the Elasticsearch data source

The ﬁrst step in to create the data source and provide the Elasticsearch cluster URL and optionally provide the basic auth login and password .

Conﬁguring the Elasticsearch data source

Writing queries

In the query editor view, select the Elasticsearch data source created above. On the left column, click on the refresh icon to refresh the schemas (indexes). The schemas are expandible and show the schema details. Refresh and lookup Elasticsearch indexes

You can then proceed to the query editor and run the search query. The query uses JSON as passed to the Elasticsearch search API Writing a search query against an Elasticsearch index Vewing Logs Terminology

Logs are organized by cluster_id, namespaces, applications, and their processes.

CLUSTER_ID

CLUSTER_ID is an optional label that can be applied to incoming data streams. E.g. it is a good practice to apply CLUSTER_ID to a K8S cluster. All namespaces from the K8S cluster will get preﬁxed with CLUSTER_ID so the user can distinguish between namespace name collisions across different clusters.

For Kubernetes clusters, CLUSTER_ID is an attribute added to the JSON payload by the sending agent for e.g. Fluentd or Fluent-bit. See here for on how to manage multiple clusters.

Namespace

The namespace is the top of the hierarchy. For a Kubernetes based deployment namespace maps to Kubernetes namespace and for non-Kubernetes based deployment it maps to the host-name of the virtual machine or physical host.

Application & Process

In the Kubernetes based deployments Application refers to deployment name and process refers to its pod names. For example for Nginx application (deployment/service) may have multiple pods which are its processes.

For the physical machine, the application refers to process names like Nginx or tomcat and process refers to process id.

There are multiple ways you can access the logs.

The Logs page hierarchically arranges logs by Namespace, application and process. It lets you see the most recent logs for the process The Search page allows you to search through logs. The logiqctl command line tool allows you to view logs in realtime. Logs Page

Logs Page

Use logs page to view the application logs by its processes. Logs table is sorted by the most recent ﬁrst. Clicking on the View Logs button opens a modal with the most recent logs for the selected process.

Logs Modal

The logs modal presents an inﬁnite view of application logs. The modal is loaded with the most recent logs, use the “Load More” button to see older logs. The severity drop-down helps to ﬁlter the logs based on log level like error, warning, info, debug etc. The search box helps to search through the logs loaded in the modal. The “Export as CSV” and “Export as JSON” buttons allows export of logs loaded in the modal.

Individual Actions

The dots available on each line will reveal additional options to copy individual log lines as json to clipboard and to deﬁne an alertable event for the selected log line. Refer to events documentation to see how to create events. See Alertable event section. Search Page

On the search Page, you can:

Access every log entry in the selected namespace that matches your search pattern. Make use of advanced search modal to build complex queries Filter the search results using Facets Get ﬁeld-level details about the entries that match your search Create a Metric Create a Report Export data

The search happens at the namespace level, select the namespace and applications, enter the search term to get started. By default, Search shows data for the last 15 minutes. If no data displays, try increasing the time range. Using the time ﬁlter, you can specify a common or recently-used time range, a relative time from now, or an absolute time range.

With the help of the advanced search modal, complex search expressions can be created. A successful search reveals a graph with a time-series view of logs, a table with facets and logs.

Filter data using Facets

Use facets to narrow your search results. Logiq's intelligent parsing engine extracts facets at the time of data ingestion. By default selecting a facet only ﬁlters the data loaded in the browser. If the data is not available, click on the "apply" button to fetch more results.

View Log Detail

Expand the caret available in each row to see the log in detail.

The dots available on each line will reveal additional options to export individual log lines as json, to deﬁne a metric and create a report. Context Logs

When you’ve narrowed down your results to a single log line and now you want to see the logs around it. You can do that by clicking the context logs option. This brings up the logs modal and you can view the logs from the selected application and process. The log line of interest will be highlighted so that you can look around it. Reports

Reporting feature comes in handy when required to periodically search and aggregate what is happening in a particular log set. You can create reports in multiple ways:

Creating a Report

From the logline, click on "create batch report"

From the favourites

From the queries page Onclick of any of the above options opens the. "Create Reports" modal.

Group By

Reporting feature lets you group the results by any ﬁelds available in the structured data or use any arbitrary regex named capture groups. To use ﬁelds from structured data, select the "Group Type" as "Field." To use regex named capture groups, select "Pattern." Logiq uses regular expression syntax accepted by RE2. Please refer to the RE2 wiki for detailed syntax.

Example 1

1 (?P\d{4})-(?P\d{2})-(?P\d{2}) 2 # This extract Year(2011), Month(03) and Day(12) from a message like 3 # flash test message 2021-03-12T19:40:08+05:30 #:25:# took=0.9647148 Example 2

1 filePath:\s(?P[/.a-zA-Z0-9]+) 2 # This exreact Path(/ecp/default.aspx) from a message like below 3 # ... count=18 filePath: /ecp/default.aspx abcd

Aggregation Operators

Following aggregation operators are available.

sum count mean min max group

View Results

Saved reports can be accessed from the Queries Page. If the query is running, this page will show the progress.

Click on any report to view the results. Click on the toggle to enable periodic refresh. Visualizations

The results can be used to create visualizations. Click "New Visualization" to start visualizing the results.

An example scatter plot logiqctl

The LOGIQ command line toolkit, logiqctl, allows you to run commands against LOGIQ Observability stack.

You can

Real-time streaming of logs Query historical application logs Search your log data. View Events Manage Dashboards Create event rules Manage license Conﬁguring logiqctl

Installation

The simplest way to try logiqctl is to download a pre-built binary from the release page. You can also build the binaries directly from source. Instructions are available here.

Conﬁguring logiqctl

logiqctl requires a running Logiq cluster and an api key. See how to get the API key here. After obtaining the API key, run the below commands to conﬁgure the logiqctl.

1 # Set cluster end point 2 > logiqctl config set-cluster your-logiq-cluster.com 3 4 # Set the API Key 5 > logiqctl config set-token r0q7EyIxNgVjAqLoIeDioJAWEhAR6wK4Y5XpPb3A 6 7 # Set the default namespace 8 > logiqctl config set-context ngnix

The default namespace settings can be overridden by passing -n ﬂag. Basic operations are covered below, for more detailed documentation refer to the user guide for logiqctl .

logiqctl logs

The logs command is used to view historical logs. The command supports an

interactive mode as well which lets the user select application and process. # tito @ tito-logiq in ~/logiqctl [18:40:48] C:127 $ ./logiqctl config init Enter cluster endpoint rc.logiq.ai Endpoint rc.logiq.ai:8081 configured succesfully.

Enter User Token **************************************** ✔ rc-logiq:rc

# tito @ tito-logiq in ~/logiqctl [18:41:04] $ ./logiqctl logs i Use the arrow keys to navigate: ↓ ↑ → ← Select an application (showing 'rc-logiq:rc' namespace)? » logiq-flash (Last Seen 14 seconds ago) logiq-kubernetes-ingress (Last Seen 15 seconds ago) redis (Last Seen 19 seconds ago) coffee-worker (Last Seen 19 seconds ago) postgres (Last Seen 48 seconds ago) ↓ flash-discovery (Last Seen 4 minutes ago)

logiqctl tail

The tail command can be used to view the logs from applications in realtime. The tail command runs an interactive prompt and lets the user choose the application and processes.

# tito @ tito-logiq in ~/logiqctl [18:52:29] C:127 $ ./logiqctl tail Use the arrow keys to navigate: ↓ ↑ → ← Select an application (showing 'rc-logiq:rc' namespace)? » * (Select All) logiq-flash (Last Seen 39 seconds ago) logiq-kubernetes-ingress (Last Seen 42 seconds ago) fluentd (Last Seen 1 minute ago) coffee-worker (Last Seen 1 minute ago) ↓ redis (Last Seen 2 minutes ago) Obtaining API Key

API Key

The API token required for accessing logiqctl and logiq api's is available under API Key Role-Based Access Control (RBAC)

Overview

Access to log data for users is managed by namespace controls. To learn more about what namespaces are in the LOGIQ product, please refer to the section on namespaces here. Log data from a single namespace can contain multiple application logs. If a user has access to a namespace, the user can view logs for all applications in the namespace.

UI Controls

When a user has restricted access to select namespaces, the UI will show only the namespaces the user can access for Logs, Search, and Events. In the example below, the user has restricted access to only 2 namespaces; customer-tooling and rc-logiq:rc

Logiqctl Controls

LOGIQ's cli provides a similar restriction when a user tries to access resources under Role- based access controls. In the example below, the same user can be seen to only access the same namespaces; customer-tooling and rc-logiq:rc

Conﬁguring RBAC

Overview

LOGIQ, not only supports access to logs but also to additional data sources. LOGIQ's RBAC enforcement works by restricting access to the data source. This in-turn translates to restricting user access to queries and dashboards created on those data sources.

In addition, LOGIQ's log collection model works by mapping incoming log data into namespaces. In LOGIQ, a namespace is treated as a data source, thus allowing a uniform model for applying access restrictions.

Creating a namespace data source

Namespace restrictions are managed by the namespace data source. Only the admin users have privileges to access and modify data sources.

In the data source configuration, provide the data source settings by listing out the namespaces that are part of the data source definition. Provide a name for the data sources and then Save your configuration. In the example below, we are creating a NonAdmin-Namespace data source that has access restriction to two namespaces customer-tooling and rc-logiq:rc

Assigning Namespace data source to users

The ﬁnal step is to attach the namespace data source to a group. This restricts all users in the group to the namespaces deﬁned in the data source.

Managing access to namespaces from multiple clusters and hosts

Namespaces in LOGIQ map to virtual hosts or namespaces in Kubernetes clusters. Resources such as ECS clusters can also be mapped to namespaces.

Let us look at how this works in practice. In the example above, the access restrictions are applied to two namespaces customer-tooling and rc-logiq:rc

The customer-tooling namespace is a virtual machine with hostname

customer-tooling . The rc-logiq:rc is a Kubernetes cluster with CLUSTER_ID set to

rc-logiq and namespace rc within the cluster.

Yes it is that simple! ANOMALY DETECTION Events

Events UI

Events are shown in the user interface in the Events tab.

It is easy to narrow down and search for events that you care about using search and ﬁltering

Event rules

Events are captured based on event rules. Many event rules are built into LOGIQ. For customizing this further, we provide a simple UI to add new rules into the system.

See section on Event Rules for further information.

Event De-duplication

Events generated within a namespace are deduplicated by the LOGIQ platform. This reduces the amount of data stored at rest over time. At peak data rates, it is possible to get a 1000x reduction in the amount of events generated. Event Rules

Event Rules UI

To access event Rules UI, click on "Events" menu and pick "Rules" option as shown in the screenshot below.

Viewing Event Rules

The Event Rules page lists all the available rules in the LOGIQ server. Each rule consists of the following:

Group: Denotes logical grouping of event rules. Typically, but not necessarily, it is the application or service name. Name: Unique name to identify the event rule. Level: It can be one of "Info", "Warning" or "High".

Application Match: The condition on which the event will be triggered. Supports regex matches. Description: A human-readable description of the event. Condition: An expression applied to ingested data. Events are triggered when the boolean condition evaluates to "True". Please see refer to Writing condition expressions. Active: This field indicates whether the event rule condition is enabled and has to be applied to the incoming data. Rules that are inactive are ignored. Alert Configured: If an alert is configured on the event, this field will have a link to that alert. Actions icon group: This offers a quick way to delete, edit or configure an alert on the event Out of the box, the LOGIQ server comes with commonly needed system defined Event Rules. The built-in rules cannot be modified or deleted. However, they can be made active / inactive if required.

Creating a new Event Rule

The Event rule creation screen is launched by clicking on "New rule" button. Please refer to Viewing Event Rules section above for details about each of the parameters needed to create new event rule. If the rule does not belong to any of the existing groups, a new group is created.

Using Filters

Event Rules can be searched and narrowed down by using ﬁlters such as Level and Group Name.

Creating alerts from event rules An Alert can be created from an Event rule by simply clicking the bell icon.

Writing condition expressions

Condition expressions can be created using the below attributes and operators

Syslog data attributes for condition expressions

The attribute values are of type string unless otherwise speciﬁed

severity, Severity facility, Facility app_name, appname, Appname, AppName timestamp, Timestamp message, Message host_name, hostname, Hostname, HostName groupings tag

Condition expression syntax Modiﬁers: + - / * & | ^ ** % >> <<

Comparators: > >= < <= == != =~ !~

Logical ops: || &&

Numeric constants, as 64-bit ﬂoating point ( 12345.678 )

String constants (single quotes: 'foobar' ) Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or unix date; date parsing is automatically tried with any string constant) Boolean constants: true false

Parenthesis to control order of evaluation ( )

Arrays (anything separated by , within parenthesis: (1, 2, 'foo') )

Preﬁxes: ! - ~

Regex comparators =~ !~

Examples

1 - 2 name: BreakInAttempt 3 description: POSSIBLE BREAK-IN ATTEMPT! 4 condition: message =~ 'POSSIBLE BREAK-IN ATTEMPT' 5 level: high Alertable Events

Alerts can be conﬁgured from log data from either the Search or the Logs pages. For example, while scrolling through context logs or search logs, if we want to get alerted on a line of log that is of interest, user can directly create an alertable event from the log line.

E.g., we want to receive an alert when a particular event (i.e. occurance of the text Reusing worker in above example) occurs multiple times say in any 5 minute duration.

On clicking the on the log line, user can open the Alertable event rule editor.

Creating an alertable event Fields

Name: A name for the alert, should be alphanumeric. Level: one of the: low , medium , high or critical Group: Select a predefined group or add a new. Description: A human-readable description of the alert. Namespace: Auto populated and non-editable field. Application Match: Auto populated field, editable, and match could be a regex as well. Message: The expression on which events will get generated. Supports RE2 regex. Application: Auto populated from the logline. Severity String: Auto populated from the logline. Facility String: Auto populated from the logline. Data Sources: Prometheus instance that is used for monitoring the event counters. This is non-editable Destination: This is a user-specified selection where the alerts will be delivered. Only user-defined alert destinations will be available for selection. Operation: One of the comparators: > >= < <= == != =~ !~ Occurrences: number of times the event must occur, must be a valid number. Period: time over which the event occurred, e.g, 5m , 10m , 1h , 1d , 1w . Period

should be greater than 5m ( 300s ) and should be greater than or equal to Refresh schedule. Rearm: how frequently you will receive notiﬁcations when your query meets the Alert criteria and does not change, must be a valid number (seconds) minimum 300 seconds. Refresh Schedule: how frequently the query needed to be refreshed in seconds, must be a valid number. Until: select when to stop the alerts. If not selected, the alert will never expire.

All the events created can be accessed at the events page and if the alert is conﬁgured on that event, it will be present as a column in the alert row.

Clicking on the conﬁgured alert will open up the respective alert page where it can be modiﬁed further for e.g. change the alert rearm duration, add additional alert destinations etc. Viewing Alerts

All the conﬁgured alerts are viewable when navigating to the alerts tab. An Individual alerts' conﬁguration can be edited on this page

LOGIQ includes alerts from it's Prometheus alert manager instance that is included with the LOGIQ install. NOTE that the editing of the Prometheus alert rules cannot be done via the UI and must use alert manager CRD's to change those alert rules. Logs to time series event visualization

With LOGIQ insights in a matter of a few clicks, you can create time series visualizations for events that matter for effective monitoring. This section describes the details on how to create visualizations from logs or search pages.

You may choose to create a dedicated dashboard for event time series visualizations. Click on "create" and choose "dashboard".

Proceed to logs tab or search tab. Once the log-line is narrowed down either from the search page or from logs page, LOGIQ Insight’s event user interface makes it almost effortless to generate events, alerts and visualizations for eﬃcient monitoring of your infrastructure.

From the search UI, choose a log-line that is of interest to generate events, and/or alerts and/or visualization. On the right hand side of the log-line click on “create new event” Similarly, You may choose to ﬁnd the log-line of interest from Logs UI. On the left hand side of the log-line click on “create new event”

This pops up the “Create alertable event rule” modal window. It enables the user to

Create an event rule Create the corresponding Visualization Create an Alert trigger (optional)

Events are generated when a log-line satisﬁes event rule criteria. Event rule consists of a list of key, operator and value combinations that could be connected by AND conditions. Each combination consists of standard parameters such as Message, Application, Severity String, Facility String, Sender or some facet parameters which are speciﬁc to that log-line. Create alertable event rule modal window

By pressing the "Create event" Button, LOGIQ generates an event rule, a query & visualization graph for the time series DB and an alert trigger if the "create and activate alert on event rule" checkbox is checked. In this example, the event is generated when the message matches the regular expression “GET /V1/license” and other additional parameter criteria are met. Each triggered event is recorded by LOGIQ insights in the time series database. It helps to generate a visualization graph of the frequency of the events with respect to time.

In this example, the query and time series visualizations can be found in the queries page as license_check query and license_check-vis visualization. Since the dashboard “EventTimeSeriesCharts” is selected in the above example the visualization will be embedded in that dashboard. Click on the dashboard and pick ”EventTimeSeriesCharts”

Time series visualization

If required by selecting the checkbox “create and activate alert on event rule” and specifying a preconﬁgured destination for notiﬁcations the alert trigger is conveniently created. COMPLIANCE Audit Trail

Audit logging is automatically enabled in Logiq. You can go to events page and look for logiq-audit group to ﬁnd the available logiq events. LOGIQ STREAMING Query API

Getting data is a two step process

Create a Query Request

POST /v1/query will respond with a QueryId. Use that to access data

1 $ curl --location --request POST 'http://cluster-1.logiq.ai/v1/query' \ 2 --header 'Accept: application/json' \ 3 --header 'Content-Type: application/json' \ 4 --data-raw '{ 5 "applicationNames": [ 6 "tomcat" 7 ], 8 "filters": { 9 "Message": { 10 "values": [ 11 "user-369" 12 ] 13 } 14 }, 15 "namespace": "production", 16 "pageSize": 100, 17 "startTime": "2020-05-10T17:55:20+05:30" 18 }'

1 { 2 "queryId": "4c977d08-4acf-52b5-a99d-d8c0eb41fe4b-lgq" 3 }

Use GET /v1/data/{queryId}/next and GET /v1/data/{queryId}/previous to retrieve data.

1 $ curl 'https://cluster-1.logiq.ai/v1/data/4c977d08-4acf-52b5-a99d-d8c0eb4 2 [...data] 3 $ curl 'https://cluster-1.logiq.ai/v1/data/4c977d08-4acf-52b5-a99d-d8c0eb4 4 [...data] 5

Query API Documentation

Version: 1.0

/v1/query

POST

Parameters

Name Located in Description Required Schema

body body Yes queryQueryProperties

Responses

Code Description Schema

200 A successful response. queryGetQueryResponse

/v1/data/{queryId}/next

GET

Parameters

Name Located in Description Required Schema

queryId path Yes string Responses

Code Description Schema

200 A successful response. queryGetDataResponse

400 Invalid Request.

403 Token Expired.

404 Returned when the resource does not exist.

/v1/data/{queryId}/previous

GET

Parameters

Name Located in Description Required Schema

queryId path Yes string

_internal query No boolean (boolean)

Responses

Code Description Schema

200 A successful response. queryGetDataResponse

400 Invalid Request.

403 Token Expired.

404 Returned when the resource does not exist.

protobufAny A

1 if (any.UnpackTo(&foo)) { 2 ... 3 }

Example 2: Pack and unpack a message in Java.

1 Foo foo = ...; 2 Any any = Any.pack(foo); 3 ... 4 if (any.is(Foo.class)) { 5 foo = any.unpack(Foo.class); 6 }

Example 3: Pack and unpack a message in Python.

1 foo = Foo(...) 2 any = Any() 3 any.Pack(foo) 4 ... 5 if any.Is(Foo.DESCRIPTOR): 6 any.Unpack(foo) 7 ...

Example 4: Pack and unpack a message in Go

1 foo := &pb.Foo{...} 2 any, err := ptypes.MarshalAny(foo) 3 ... 4 foo := &pb.Foo{} 5 if err := ptypes.UnmarshalAny(any, foo); err != nil { 6 ... 7 } The pack methods provided by protobuf library will by default use 'type.googleapis.com/full.type.name' as the type URL and the unpack methods only use the fully qualiﬁed type name after the last '/' in the type URL, for example "foo.bar.com/x/y.z" will yield type name "y.z".

JSON

The JSON representation of an Any value uses the regular representation of the deserialized, embedded message, with an additional ﬁeld @type which contains the type URL. Example:

1 package google.profile; 2 message Person { 3 string first_name = 1; 4 string last_name = 2; 5 } 6 7 { 8 "@type": "type.googleapis.com/google.profile.Person", 9 "firstName": , 10 "lastName": 11 }

If the embedded message type is well-known and has a custom JSON representation, that representation will be embedded adding a ﬁeld value which holds the custom JSON in addition to the @type ﬁeld. Example (for message [google.protobuf.Duration][]):

1 { 2 "@type": "type.googleapis.com/google.protobuf.Duration", 3 "value": "1.212s" 4 }

queryFilterValues Name Type Description Required

values [ string ] No

queryGetDataResponse

Name Type Description Required

data [ querySysLogMessage ] No

Status string No

remaining integer No

queryGetQueryResponse

Name Type Description Required

queryId string No

info [ queryQueryInfo ] No

errors [ queryQueryErrors ] No

meta object No

queryOrderBy

Name Type Description Required

queryOrderBy string

queryQueryErrors

Name Type Description Required

Key string No message string No

queryQueryInfo

Name Type Description Required

Key string No

message string No

queryQueryProperties

Name Type Description Required

applicationNames [ string ] No

ﬁlters object No

namespace string No

pageSize long No

startTime string No

endTime string No

keyWord string No

querySysLogMessage

Name Type Description Required

ID string No

AppName string No

Facility string No

FacilityString string No Hostname string No

Message string No

MsgID string No

PartitionID string No

Priority string No

ProcID string No

Sender string No

Severity string No

SeverityString string No

StructuredData string No

Tag string No

Timestamp string No

Namespace string No

runtimeError

Name Type Description Required

error string No

code integer No

message string No LOGIQ Conﬁguration E-Mail Conﬁguration

LOGIQ insights can be conﬁgured to send emails to notify alerts. It uses smtp mail and requires following information for conﬁguration:

1. Mail Server DNS/IP 2. Mail service/relay port for SMTP e.g. 25 or 465/587 (TLS/SSL) 3. Username for the mail service/relay 4. Password for the user above in 3 5. Default Sender E-Mail: The send e-mails will show up as from this sender . Depending on what port you are using and support at the mail service/relay, select either SSL/TLS to be enabled

The E-mail conﬁguration is a global conﬁguration and is available only to the admin user

The default sender e-mail in most cases will need to be a veriﬁed e-mail address from the Mail service/relay provider

The following attributes can be setup by LOGIQ admin on UI. Logiq Insights Email Setup Alert Destinations

When an alert triggers, LOGIQ Insights sends alert details to its designated alert destinations. LOGIQ Insights supports following types of alert destinations.

E-mail Slack PagerDuty Generic Webhook

Conﬁguring Destinations

To conﬁgure alert destinations navigate to settings and open “Alert Destinations” tab

Alert Destinations

It’s required to conﬁgure the e-mail server to receive e-mail notiﬁcations.

1. Click on + New Alert Destination”, pick Email. 2. Specify, recipient’s name & email address. 3. Subject is conﬁgurable. It’s recommended to use format: Alert {alert_name} changed status to {state}. alert_name and state are template parameters and will be replaced by the urls to give more information about the alert that will have occurred.

James Smith's Email conﬁguration

Email destination is created by clicking the save button.

Email Alert Destination Slack

In your slack console, pick or create a channel such as #alerts-events in this example and create an incoming webhook by clicking on "Add Incoming Webhooks Integration"

Slack Webhooks integration

In your LOGIQ Insight's UI, Open “Alert Destinations” tab in the settings screen, and click on ”+ New Alert Destination”. Pick “Slack” as the type. Set the name, channel, etc. and provide a “Slack Webhook URL”, from above Slack Destination Conﬁguration

Pager Duty

Obtain the PagerDuty Integration Key from your PagerDuty service. Use Events API v2 Integration Type. After obtaining the Integration Key:

1. Open “Alert Destinations” tab in the settings screen, and click on ”+ New Alert Destination”. 2. Pick “PagerDuty” as the type. 3. Populate mandatory ﬁelds are Name and Integration Key obtained earlier. PagerDuty Conﬁguration

Associating alert destinations to alert

Once the Alert destinations are created, one or more alert destinations can be configured in alert create or edit ui. Each configured destination is notified whenever that alert triggers. Single Sign-On with SAML

LOGIQ can be set up for user login using Single Sign-On (SSO) with SAML by conﬁguring LOGIQ as Service Provider(SP) and OKTA, Google, or in general any SAML2.0 compliant identity provider (IDP). This is a two-step process.

Enabling SAML

Login with your admin credentials. Click on Settings menu. Enable "SAML conﬁguration" checkbox. Add SAML Metadata URL, Entity ID, NameID Format.

Check below on speciﬁc steps for your Identity provider

LOGIQ (Service Provider) conﬁguration If user-groups are conﬁgured on IDP side, create the identical user groups in LOGIQ. This can be done by clicking on "Settings" menu and going to Groups tab. This example shows creating "NonAdmin" user group.

This concludes the LOGIQ side conﬁguration.

Logout as LOGIQ admin. In the login Screen, "SAML Login" Button should be available to login with the user's SSO credentials. By clicking the button browser is redirected to the IDP screen where user can login using its IDP credentials.

IDP conﬁguration

This document provides detailed information to conﬁgure OKTA and Google as Identity providers. For other identity providers, please refer to identity providers' documentation. In your IDP application, provide the SAML Assertion Consumer Service (ACS) URL for your LOGIQ environment and attribute mappings

The following attributes are required. The LOGIQ mappings for each of the attributes are in brackets. Please use the correct attribute name otherwise LOGIQ will not be able to recognize the SAML assertion

First name (FirstName) and Last name (LastName) Group name (LogiqGroups)

Use following SAML Assertion Consumer Service (ACS) url

https:///saml/callback?org_slug=default

With this you should be able to access a SAML metadata URL or SAML metadata ﬁle.

Okta Conﬁguration

This section describes Okta conﬁguration in detail. Users should assume the Okta admin role and start in the Okta control panel by clicking the button to add a new application. Choose Web as the platform. The sign-on method is SAML 2.0.

Create a New App

On the next screen OKTA has ﬁelds for a few URLs:

Single Sign-On URL Recipient URL Destination URL Audience Restriction

Use your LOGIQ endpoint url in following format: https:///saml/callback?org_slug=default

Set Name ID format: EmailAddress Application username: Email

ACS URL

Conﬁgure Attribute statements:

Name Name Format Value

FirstName Unspecified user.firstName LastName Unspecified user.lastName

By default, any user that is created with SAML/SSO will join the default user-group in LOGIQ. It’s possible to conﬁgure OKTA to pass groups the user should join by setting the LogiqGroups parameter with the intended group name. For example, if the SAML user is a member of the NonAdmin group in Okta, at the user login, the user will be authenticated and added to "NonAdmin" group.

The default group in LOGIQ has access to all data sources. It is highly recommended to create a group assignment for your users and conﬁgure LogiqGroups as described above. This allows RBAC policies and limits access to what data a user can access

Conﬁgure Attribute statements:

Group Name Name Format Value

LogiqGroups Basic Equals: NonAdmin Attribute Setup

Continue to create the application as guided by OKTA instructions. Once the application is successfully created, take note of the following information. This is needed to conﬁgure LOGIQ.

1. SAML Metadata URL: "Identity Provider Metadata" URL depicted below in blue can be clicked to ﬁnd out SAML metadata URL.

2. Entity ID: By navigating to "View Setup Instruction" shown in the above snapshot, You can ﬁnd Entity ID. 3. NameIDFormat: NameID can be found in the SAML metadata by searching NameIDFormat, shown as selected gray text in the picture below.

NameIDFormat in saml metadata

Navigate back to the app and edit "Audience Restriction" and set it with the IDP issuer described in #2 above. If already not there, create users and user groups such as NonAdmin in this example. Users and user groups can be also be brought in with inbound federation with ADFS or other identity providers. Assign the users to the Application or group such as NonAdmin in this example to the Application. This concludes the IDP side conﬁguration.

Google GSuite Conﬁguration

How to conﬁgure LOGIQ to use SAML for https://www.youtube.com/watch? single sign-on v=pTVHkxcp4mg

LOGIQ with Google as SAML2.0 IDP Conﬁguration Log Ingest conﬁguration Terminology

This section covers commonly used terminology in our documentation, ui, source code when referring to the LOGIQ conﬁguration ﬁle

Credential

A credential gives access details to the S3 storage resource so LOGIQ can drain the logs to your S3 compatible persistent store. A credential is referenced in the destination section.

Destination

A destination is a store for the data. Incoming data streams are processed and eventually land into one or more of the destinations specified in the configuration file.

Filter

A Filter performs a boolean expression evaluation based on the incoming data fields. It returns a "true" or "false" evaluation. A filter is attached to a rule. The filter condition must evaluate to "true" for the data to be stored at the destination specified in the rule.

Grouping

A Grouping is a key value pair that tags data ingested by the LOGIQ server. It is deﬁned on a destination.

Partition

A partition speciﬁes a partitioning scheme for the incoming data for a given destination. A user can specify a partition scheme using any of the attributes in the incoming data.

Rule

A Rule determine how incoming data is mapped to a bucket. LOGIQ server can manage multiple buckets and manage data ﬂowing into them simultaneously via multiple rules.

Source

A source is one or more clients sending data to the LOGIQ server. Minimal server conﬁguration

Creating a minimal conﬁguration

LOGIQ server needs a configuration file that describes how to handle incoming data and where to store it. See below for a minimal configuration to get started. In the below example we are connecting LOGIQ to ingest data into an AWS backed S3 compatible store. The data stored will be partitioned by day.

NOTE: Change the below config to work with your environment. If you are using AWS deployment using CloudFormation, the config file is automatically generated during the install. If you are using a Kubernetes cluster to run LOGIQ, the configuration is passed as a config map.

1 options: 2 ca: "/etc/ssl/logiq/certs/logIQ.crt" 3 cert: "/etc/ssl/logiq/certs/logiq-server.crt" 4 key: "/etc/ssl/logiq/certs/logiq-server.key" 5 host: 0.0.0.0 6 credentials: 7 - 8 name: aws_logging_bucket 9 s3: 10 secret_key: logiq_secret 11 access_key: logiq_access 12 partitions: 13 - 14 name: p_by_day 15 fields: 16 - year 17 - month 18 - day 19 destinations: 20 - 21 name: default_log_store 22 partition: p_by_day 23 s3: 24 bucket: logiq-bucket-1fbc-1 25 region: us-east-1 26 credential: aws_logging_bucket 27 rules: 28 - 29 destination: default_log_store The above conﬁguration deﬁnes a simple rule that all incoming data needs to go to the destination default_log_store.

See the section on LOGIQ server conﬁguration on additional conﬁguration options. Server options

Options

The options section in the LOGIQ configuration file is for changing default server parameters. Below are the commonly used options for typical deployments. Options can be either specified as quoted strings / non-quoted e.g. "2514" or 2514 ca [optional]

CA Certiﬁcate ﬁle path cert [optional]

Server Certiﬁcate ﬁle path host [optional]

Host IP's which server binds to - default is 0.0.0.0 key [optional]

Server certiﬁcate key relp_port [optional]

The RELP port where server listens for RELP connection. Default port is 20514 relp_port_tls [optional]

The RELP port where server listens for RELP connection. Default port is 2514 syslog_port [optional] The syslog port where server listens for syslog connection. Default port is 514 syslog_port_tls [optional]

The syslog port where server listens for syslog connection. Default port is 7514 cli_port [optional]

The port where the LOGIQ CLI is accessed. Default port is 9998 cli_user [optional]

Password for LOGIQ CLI console glue_iam_service_role [optional]

ARN of the glue service role. See here for more details. This only applies if you are trying to run LOGIQ in AWS.

Example options:

1 ca: "/etc/ssl/logiq/certs/logIQ.crt" 2 cert: "/etc/ssl/logiq/certs/syslog.crt" 3 host: 0.0.0.0 4 key: "/etc/ssl/logiq/certs/syslog.key" 5 relp_port_tls: "2514" 6 relp_port: "20514" 7 cli_port : "9998" 8 syslog_port_tls: "7514" 9 syslog_port: "514" 10 cli_user : "logiq_access" 11 cli_password : "logiq_secret" 12 glue_iam_service_role: "arn:aws:iam::199500300922:role/LogIQ-Glue_role Sources

Sources provide matching rules for sender ip addresses. Sources can specify a single IP, a range of IP addresses or a list of IP addresses. A source deﬁnition can specify one or more of single IP, range of IP addresses or a list of IP addresses. In this case the source condition evaluates true if any of these source deﬁnitions are a match.

1 sources: 2 - 3 name: localhost 4 ipv4: 127.0.0.1 5 - 6 name: 3164_ip_1 7 ipv4: 192.168.55.20 8 - 9 name: 3164_ip_2 10 ipv4: 192.168.55.21 11 - 12 name: qa_lab 13 ipv4: 192.168.40.3 14 ipv4_range: 10.0.1.5/24 15 ipv4_list: 192.168.1.1, 192.168.1.10

Sources are defined with the "sources" keyword in the config file. All source definitions must have a "name". Sources are referred in rules using their name. The config file validator will flag an error for names that are not found and or source definitions that have a missing name.

name [required]

This key is used to specify a unique name for source matching rules.

1 - 2 name: qa_lab 3 ipv4_list: 192.168.1.1, 192.168.1.10 The source matching conditions are referenced in message rules using their names

1 rules: 2 - 3 source: localhost 4 destination: t_debug

Client IP/Network details [optional]

At least one of ipv4, ipv4_list and ipv4_range is required when selecting a Client IP/Network based source

ipv4 - Specify a single ip address

1 sources: 2 - 3 name: qa_lab_dns 4 ipv4: 192.168.4.21

ipv4_list - Specify a list of ip addresses

An IP address list is speciﬁed by individual ip addresses separated by commas. Note that in the "ipv4_list" keyword, only individual ip addresses are allowed and not CIDR ranges.

1 sources: 2 - 3 name: qa_lab 4 ipv4_list: 192.168.1.1, 192.168.1.10 ipv4_range - Specify a range of ip addresses

The IP address range is speciﬁed using the CIDR notation. E.g. 10.0.1.5/24 represents a range of ip's starting from 10.0.1.0 - 10.0.1.255 or a total of 256 addresses.

1 sources: 2 - 3 name: qa_lab 4 ipv4_range: 10.0.1.5/24

Cloud S3 storage buckets

Stackdriver logs from GCP cloud storage

LOGIQ can directly ingest from Stackdriver's Google cloud storage sink. You do not need to incur additional pub sub costs for this. Create a google_cloud_storage source in LOGIQ conﬁg for this.

1 - 2 name: stackdriver_gcp_storage_bucket 3 s3: 4 type: google_cloud_storage 5 credential: > 6 { 7 "type": "service_account", 8 "project_id": , 9 "private_key_id": , 10 "private_key": , 11 "client_email": , 12 "client_id": , 13 "auth_uri": "https://accounts.google.com/o/oauth2/auth", 14 "token_uri": "https://oauth2.googleapis.com/token", 15 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth 16 "client_x509_cert_url": 17 } 18 bucket: Destinations

Every LOGIQ server conﬁguration ﬁle must have one or more destination for the data to be written to.

1 destinations: 2 - 3 name: t_webservers 4 s3: 5 bucket: production 6 prefix: webservers/ 7 credential: production 8 groupings: 9 - 10 name: Environment 11 value: Production 12 - 13 name: Tier 14 value: Web

NOTE: A missing destination in a rule is equivalent to evaluating the rule but dropping the incoming packet once a rule is matched. However, it is possible other rules with properly conﬁgured destinations may still process and successfully write the incoming packet.This also implies that an incoming data message can go to multiple destinations simultaneously.

Destinations are defined using the "destinations" keyword in the configuration file. All destination definitions must have a "name", a destination specific section and and optional "partition" section defining data partitioning scheme.

The config validator will flag an error for destination names that are not defined but referenced in rules. In addition, depending on the type of the destination e.g. AWS S3 or On- prem S3, additional mandatory fields may be required when defining a destination.

name [required] This key is used to specify a unique name for a destination deﬁnition

1 - 2 name: qa_lab 3 ipv4_list: 192.168.1.1, 192.168.1.10

The destination for an incoming message is speciﬁed in the message rules using their names

1 rules: 2 - 3 source: debug_instances 4 destination: t_debug

partition [optional]

An optional partition reference can be provided which tells the LOGIQ server on how to organize the data at the destination.

In the example below, there are two destinations defined: t_webservers and t_debug. A partition reference p_by_day is specified for the destination t_webservers _but not for t_debug. The p_by_day is a named reference to a partition definition. See section on Partitions on how to specify a partitioning scheme for destination data.

1 destinations: 2 - 3 name: t_webservers 4 partition: p_by_day 5 s3: 6 bucket: production 7 prefix: webservers/ 8 credential: production 9 groupings: 10 - 11 name: Environment 12 value: Production 13 - 14 name: Tier 15 value: Web 16 - 17 name: t_debug 18 s3: 19 bucket: debug 20 credential: debug

Destination types

A destination definition must specify one of the below defined types. A destination with no subsection for one of the types, is not a valid destination and the config parser will issue an error.

s3 [required]

An S3 destination is used for AWS S3 and AWS S3 compatible object stores.

NOTE: For AWS S3 compatible object store, the AWS SDK should work on the object store and support bucket listing, creation and multi-part uploads.

Below are two examples. The first destination t_webservers is a definition of a bucket in AWS S3. The second destination t_minio_webservers is a definition of a bucket hosted in an S3 compatible object store. An S3 compatible object stores supports an optional endpoint key in the s3 definition that points to the http endpoint where the object store is hosted.

1 - 2 name: t_webservers 3 partition: p_by_day 4 s3: 5 bucket: logiqf978a 6 region: us-east-1 7 prefix: webservers 8 credential: athena_and_s3_access 9 groupings: 10 - 11 name: Environment 12 value: Production 13 - 14 name: Tier 15 value: Web 16 - 17 name: t_webservers 18 partition: p_by_day 19 s3: 20 bucket: logiq-f978b 21 endpoint: http://10.0.1.49:9000 22 region: us-east-1 23 prefix: webservers 24 credential: compatible_s3 25 groupings: 26 - 27 name: Environment 28 value: Development 29 - 30 name: Tier 31 value: Web

bucket [required]

The bucket key speciﬁes the bucket in the S3 compatible store where LOGIQ will upload the log data. This is a required parameter and the bucket must exist. LOGIQ currently doesn't support creating the bucket automatically. credential [required]

Every S3 destination must provide a credential to use. The credential key is a reference to a credential name that is deﬁned in the credentials section of the conﬁg. Please refer to the Credentials section for more details on how to specify a credential. endpoint [required for s3 compatible object stores] If you want to write data to any S3 compatible object store thats not AWS S3 e.g. Nutanix Objects, Cloudian, Azure blob etc., you need to specify the endpoint for the s3 compatible object store. The endpoint URL will either be an http or and https URL along with a port e.g. http://10.1.0.49:9000.

If an endpoint is not provided, it is implicitly assumed that we are talking with AWS S3. groupings [optional]

Groupings provide a way to group incoming data streams that share similar characteristics. E.g. a user may want to tag all traﬃc coming from 10.0.20.0/24 subnet to be Production data stream. See section on Groupings for more details. preﬁx [optional]

The prefix key is an optional key that specifies a prefix to use for all the objects being created in S3. E.g. if bucket is foo and prefix is bar/, all of the uploaded data objects will be under foo/bar/

NOTE: If a prefix is not specified, the destination name is used as a prefix. Filters

Incoming data streams can be filtered before they are written to the final store in the S3 / S3 compatible bucket. Filter expressions allow fine grained matching on key attributes from the ingested data.

Filters are defined using the "filters" keyword in the LOGIQ configuration file. A filter definition consists of a name and a condition. E.g.

1 filters: 2 - 3 name: f_debug 4 condition: severity == 'debug' 5 - 6 name: f_macbook_ubuntu_vm 7 condition: app_name == 'macbook_ubuntu_vm'

name [required]

This name key is used to specify a unique name for the ﬁlter

1 filters: 2 - 3 name: f_ubuntu_vm 4 condition: app_name == 'ubuntu_vm'

Filter for a rule is specified by including the filter name in the rule definition.

1 rules: 2 - 3 source: s_ubuntuvm 4 filter: f_ubuntu_vm 5 destination: t_default_destination Once an incoming stream matches the source definition in the rule, the filter, if one is specified, is applied to determine a rule match.

condition [required]

The condition key is used to specify a boolean expression that is used for computing a rule match. The boolean condition is speciﬁed on key ﬁelds that are extracted from the ingested data.

1 filters: 2 - 3 name: f_ubuntu_vm 4 condition: app_name == 'ubuntu_vm'

Syslog data attributes for condition expressions

The attribute values are of type string unless otherwise speciﬁed

severity, Severity facility, Facility priority, Priority app_name, appname, Appname, AppName timestamp, Timestamp message, Message host_name, hostname, Hostname, HostName boolean: tag.Exists, Tag.Exists string: tag.Value, Tag.Value (RFC3164 only) boolean: protocol.RFC5424, Protocol.RFC5424, Protocol.RFC3164, protocol.RFC3164 MsgId, Msgid, msg_id (RFC5424 only )

Condition expression syntax Modiﬁers: + - / * & | ^ ** % >> <<

Comparators: > >= < <= == != =~ !~

Logical ops: || &&

Numeric constants, as 64-bit ﬂoating point ( 12345.678 )

Parenthesis to control order of evaluation ( )

Arrays (anything separated by , within parenthesis: (1, 2, 'foo') )

Preﬁxes: ! - ~

Regex comparators =~ !~ Groupings

Incoming data can be tagged with user deﬁned key value pairs for facilitating analytics. The key value pair is called a grouping in the LOGIQ platform. A destination may specify one or more groupings to be applied to the incoming data message.

1 destinations: 2 - name: t_webservers 3 partition: p_by_day 4 s3: 5 bucket: logiqcloud 6 region: us-east-1 7 prefix: dev-logs 8 credential: athenas3 9 groupings: 10 - name: Environment 11 value: Production 12 - name: Tier 13 value: Web

The LOGIQ platform provides the below built-in groupings that are common. Additional groupings can be defined by the user when setting up the configuration file.

1 groupings: 2 - 3 name: Environment 4 values: 5 - Production 6 - Staging 7 - Development 8 - QA 9 - UAT 10 description: The Environment for which the log is being collected 11 - 12 name: Tier 13 values: 14 - Database 15 - Web 16 - Cache 17 - LoadBalancer 18 - Queue 19 - Analytics 20 - Logger 21 - Unknown 22 description: The Tier of the app typically in a multi-tier application

Built-in groupings can be extended by adding extra values. This is done by creating a grouping definition in the configuration file with the same name. The user defined values are merged with the built-in values.

Custom groupings are deﬁned using the "groupings" keyword and related attributes.

1 groupings: 2 - 3 name: CustomGrouping 4 values: 5 - foo 6 - bar 7 - baz

The attribute details for deﬁning custom groupings are deﬁned below.

name [required]

Every grouping deﬁnition needs to have a name. The name deﬁnes a key for the grouping.

values [required]

A grouping key can have one or more possible values. The values keyword enumerates all possible values allowed for a grouping key.

description [required]

A description for a grouping is a human readable text that explains the purpose of the grouping/classiﬁcation being done.

Cardinality

When specifying a grouping for a destination, multiple values for the same key are allowed. Rules

Every LOGIQ configuration file must have at least one rule. A rule specifies how the incoming data streams are separated and organized into buckets and objects.

Rules are defined with the "rules" keyword in the config file. All rule definitions must have a "destination" keyword referring to a destination by name. Optionally, a rule may specify a "source" keyword referring to a source by name and a "filter" keyword referring to a filter by name. The config file validator will flag an error for destination, source, filter names that are not found but referenced in a rule definition

1 rules: 2 - 3 source: s_webservers 4 filter: f_debug 5 destination: t_webservers_debug

A minimal rule is deﬁned with just a destination keyword and nothing else. This matches any client sending data and all data packets that are received.

1 rules: 2 - 3 destination: t_default_bucket

destination [required]

The "destination" keyword speciﬁes the name of a destination where data will be stored. Please refer to the destinations section for more details on how to deﬁne a valid destination.

source [optional]

The "source" keyword specifies the name of a source definition to match a sender of the data. This is an optional field. If the source keyword is not specified, all clients that send data to the server are allowed to match against the rule. Please refer to the sources section for more details on how to define a valid source.

ﬁlter [optional]

The "filter" keyword specifies the name of a filter definition that is used to specify a filtering rule. The destination is selected if the incoming data stream matches the filter. The filter is applied after the source match is made.

If a filter is not specified, the destination is selected based on the source match or if no source if specified for the destination. Please refer to the filters section for more details on how to define a valid filter. Credentials

A credential gives access details to the S3 storage resource so LOGIQ can drain the logs to your S3 compatible persistent store.

A credential is referenced in the destination section. For e.g.

One or more credentials can be deﬁned using the "credentials" keyword.

1 credentials: 2 - s3: 3 name: production 4 secret_key: wJalrXUtnFA9I/K7BDENA/bPxRfiCYEXNMPLEKEY 5 access_key: AKIAIOSFODNN7EXAMPLE

The attributes for deﬁning a single credential are deﬁned below.

s3 [required]

The "s3" keyword is used to specify credentials for any S3/S3 compatible object store. name [required]

Every credential requires a unique name. It is referenced by this name in the configuration file when specified in a destination. The "name" keyword is used to specify the name for the credential.

secret_key [required]

The "secret_key" keyword provides a secret or password that is part of the Access key used by the LOGIQ server to sign the request when it uses the S3 SDK to connect to an S3 compatible service.

access_key [required]

The "access_key" keyword provides an access or user identiﬁer that is part of the Access key used by the LOGIQ server to sign the request when it used the S3 SDK to connect to an S3 compatible service. Partitions

A partition speciﬁes a partitioning scheme for the incoming data for a given destination. A user can specify a partition scheme using any of the attributes in the incoming data. Partitioning of data is important to ensure good query performance.

Partitioning schemes are defined in the configuration file using the "partitions" keyword.

1 partitions: 2 - name: p_by_day 3 fields: 4 - year 5 - month 6 - day

Once deﬁned, the partition scheme can be referred in a destination to control data layout.

1 destinations: 2 - name: t_webservers 3 partition: p_by_day 4 ......

The attributes to deﬁne a partitions are detailed below.

name [required]

The "name" keyword defines a unique name for the partition scheme. This will be used to refer to the partitioning scheme in the rest of the configuration file

fields [required] The "fields" keyword is a list of attributes that are part of the incoming data. Any of the data attributes can be used in the partition field. The list of keys in an ordered list and gives the order in which the partition values are created.

NOTE: Once a partition scheme is attached to a destination, it cannot be changed and neither can the partition deﬁnition be altered.

Valid attributes in ﬁelds

Following attributes are valid in ﬁelds

namespace appname year month day RUNNING ON AWS Getting started

Create an S3 bucket

LOGIQ stores all your ingested logs in an S3 bucket. You will need to create a bucket in a region which has Athena access. Please refer to the AWS documentation for Athena regions.

Generate Programmatic access keys for LOGIQ

Go to your AWS IAM console and add a user for programmatic access. Download and save the Secret and Access key for the user. This will be later used to access your S3 bucket where log data will be ingested.

Proceed to AWS IAM Resources to conﬁgure permissions for the user.

Running the LOGIQ cluster

You should now have the following information created and ready for using and running a LOGIQ cluster.

1. AWS S3 bucket name and ARN 2. Access/Secret key for S3 and Athena access 3. Glue service role and ARN

Self hosted

If you are hosting LOGIQ instances in your AWS account, you can setup using Cloud formation instructions in the AWS Quickstart guide.

Managed service If LOGIQ is running as a managed cluster for you, please provide us with the above information to get up and running. AWS IAM Resources

NOTE: LOGIQ can use Athena and Glue on AWS optionally to power the SQL queries. Instructions below for Athena/Glue are only needed if you choose to use those services.

LOGIQ User Role

You need to provide AWS access key and secret in LOGIQ server conﬁguration with the following permissions.

1 { 2 "Version": "2012-10-17", 3 "Statement": [ 4 { 5 "Sid": "VisualEditor0", 6 "Effect": "Allow", 7 "Action": [ 8 "s3:PutObject", 9 "s3:GetObject", 10 "s3:ListBucket" 11 ], 12 "Resource": [ 13 "arn:aws:s3:::", 14 "arn:aws:s3:::/*" 15 ] 16 }, 17 { 18 "Sid": "VisualEditor1", 19 "Effect": "Allow", 20 "Action": [ 21 "iam:PassRole", 22 "s3:ListAllMyBuckets", 23 "glue:CreateCrawler" 24 ], 25 "Resource": "*" 26 } 27 ] 28 }

You also need to provide AWS access key and secret in LOGIQ server conﬁguration with AmazonAthenaFullAccess (arn:aws:iam::aws:policy/AmazonAthenaFullAccess) policy.

IAM Service Role For Glue

You need to grant your IAM role permissions that AWS Glue can assume when calling other services on your behalf. This includes access to Amazon S3 for any sources, targets, scripts, and temporary directories that you use with AWS Glue. Permission is needed by crawlers, jobs, and development endpoints.

Please refer to the following guide for creating a service role for glue

Create an IAM Role for AWS Glue

Please refer to the following guide for creating a service role for glue

Step 2: Create an IAM Role for AWS Glue - https://docs.aws.amazon.com/en_pv/glu AWS Glue e/latest/dg/create-an-iam-role

Use the below Inline policy for Glue Service Role. You need to update the policy with your actual S3 Bucket name. Alternately you can use the Cloud formation template below to create the Glue Service Role

1 { 2 "Version": "2012-10-17", 3 "Statement": [ 4 { 5 "Effect": "Allow", 6 "Action": [ 7 "glue:*", 8 "s3:GetBucketLocation", 9 "s3:ListBucket", 10 "s3:ListAllMyBuckets", 11 "s3:GetBucketAcl", 12 "iam:ListRolePolicies", 13 "iam:GetRole", 14 "iam:GetRolePolicy", 15 "cloudwatch:PutMetricData" 16 ], 17 "Resource": [ 18 "*" 19 ] 20 }, 21 { 22 "Effect": "Allow", 23 "Action": [ 24 "s3:GetObject", 25 "s3:PutObject", 26 "s3:DeleteObject" 27 ], 28 "Resource": [ 29 "arn:aws:s3:::/*" 30 ] 31 }, 32 { 33 "Effect": "Allow", 34 "Action": [ 35 "logs:CreateLogGroup", 36 "logs:CreateLogStream", 37 "logs:PutLogEvents" 38 ], 39 "Resource": [ 40 "arn:aws:logs:*:*:/aws-glue/*" 41 ] 42 } 43 ] 44 }

AWS Cloud formation template for Glue Service Role Glue-Service-Role-Cloudformation.yml

1 AWSTemplateFormatVersion: '2010-09-09' 2 Description: >- 3 This template will build out the IAM Roles for Logiq Glue Crawler 4 5 Parameters: 6 S3BucketName: 7 Description: Name of the Logiq S3 bucket 8 Type: String 9 10 Resources: 11 LogiqGlueCrawlerPolicy: 12 Type: 'AWS::IAM::ManagedPolicy' 13 Properties: 14 ManagedPolicyName: Logiq-Glue-Crawler-Policy 15 Description: Policy for Logiq Glue Crawler 16 Roles: 17 - !Ref LogiqGlueCrawlerRole 18 PolicyDocument: 19 Version: 2012-10-17 20 Statement: 21 - Effect: Allow 22 Action: 23 - 'glue:*' 24 - 's3:GetBucketLocation' 25 - 's3:ListBucket' 26 - 's3:ListAllMyBuckets' 27 - 's3:GetBucketAcl' 28 - 'iam:ListRolePolicies' 29 - 'iam:GetRole' 30 - 'iam:GetRolePolicy' 31 - 'cloudwatch:PutMetricData' 32 Resource: 33 - '*' 34 - Effect: Allow 35 Action: 36 - 's3:DeleteObject' 37 - 's3:PutObject' 38 - 's3:GetObject' 39 Resource: 40 - !Sub 'arn:aws:s3:::${S3BucketName}/*' 41 - Effect: Allow 42 Action: 43 - 'logs:PutLogEvents' 44 - 'logs:CreateLogStream' 45 - 'logs:CreateLogGroup' 46 Resource: 47 - 'arn:aws:logs:*:*:/aws-glue/*' 48 49 LogiqGlueCrawlerRole: 50 Type: 'AWS::IAM::Role' 51 Properties: 52 AssumeRolePolicyDocument: 53 Version: '2012-10-17' 54 Statement: 55 - 56 Effect: 'Allow' 57 Principal: 58 Service: 59 - 'glue.amazonaws.com' 60 Action: 61 - 'sts:AssumeRole' 62 RoleName: Logiq-Glue-Service-Role 1-Click deployment using CloudFormation

Overview

LOGIQ can be deployed on AWS in a single AMI instance in a 1-Click fashion using our CloudFormation template and the LOGIQ AMI from the Amazon Marketplace.

All the resources required to create and conﬁgure LOGIQ on AWS are taken care by the template. All you need to do is provide a few simple input parameters.

The CloudFormation template can be found below

https://logiqcf.s3.amazonaws.com/release/logiq-stack.json

Please note that using the Marketplace AMI is not free and you will be charged per the marketplace published rates for LOGIQ

Template parameters

The following parameters need to be provided for a successful deployment via the CloudFormation template

Stack name AWS Resource conﬁguration

The LOGIQ AMI is instantiated with an SSH key pair installed so a user can later connect to the AMI when required. In addition an S3 Bucket where the user data will be stored is required.

Optionally, the preﬁx for creating access control resources like a LOGIQ User and Role can be provided. For optional resources, we use defaults if none is provided. It is strongly advised that a unique preﬁx string be used if you plan on running multiple instances so as to keep the users and roles separate.

EC2 Instance conﬁguration

The LOGIQ services are deployed in an EC2 instance in a speciﬁc region. The template asks you to select a region for deployment. You also need to select an appropriate instance size. By default an m5.xlarge instance is created if user does not change it.

A larger instance results in better performance. In addition, to secure your environment, it is strongly recommended that you provide a CIDR range for the clients that can connect to the LOGIQ server. This will be programmed in the security group created during the deployment.

Supported Regions

The CloudFormation template supports all AWS regions including Amazon GovCloud regions.

Default user and password

Once the LOGIQ instance is created, you can login to the instance using the below credentials user: [email protected] password: - e.g. if CloudFormation stack is called Foo and bucket is called Bar the password is Foo-Bar

TLS Client certiﬁcates

LOGIQ server exposes several protocols via secure ports. The client public/private certiﬁcate to connect to the server is available via SSH into the AMI. The below command illustrates ssh into LOGIQ server instance and print the certiﬁcate details 1 $ ssh -i ~/Downloads/logiqai.pem [email protected] 2 $ sudo docker logs -f `sudo docker ps | grep quay.io/logiq/flash:rc.single