Graylog Documentation Release 3.1.0
Total Page:16
File Type:pdf, Size:1020Kb
Graylog Documentation Release 3.1.0 Graylog, Inc. Mar 12, 2021 GRAYLOG 1 Architectural considerations 3 1.1 Minimum setup..............................................3 1.2 Bigger production setup.........................................4 1.3 Graylog Architecture Deep Dive.....................................5 2 Getting Started 7 2.1 Planning Your Log Collection......................................7 2.2 Download & Install Graylog....................................... 11 2.3 Initial Configuration........................................... 13 2.4 Connect to the Web Console....................................... 16 2.5 Explore Graylog............................................. 18 2.6 Collect Messages............................................. 21 3 Installing Graylog 27 3.1 Virtual Machine Appliances....................................... 27 3.2 Operating System Packages....................................... 31 3.3 Chef, Puppet, Ansible.......................................... 45 3.4 Docker.................................................. 45 3.5 Amazon Web Services.......................................... 55 3.6 Microsoft Windows........................................... 56 3.7 Manual Setup............................................... 56 3.8 System requirements........................................... 61 4 Upgrading Graylog 63 4.1 Upgrading to Graylog 2.0.x....................................... 63 4.2 Upgrading to Graylog 2.1.x....................................... 68 4.3 Upgrading to Graylog 2.2.x....................................... 71 4.4 Upgrading to Graylog 2.3.x....................................... 73 4.5 Upgrading to Graylog 2.4.x....................................... 74 4.6 Upgrading to Graylog 2.5.x....................................... 75 4.7 Upgrading to Graylog 3.0.x....................................... 75 4.8 Upgrading to Graylog 3.1.x....................................... 82 4.9 Upgrading Graylog Originally Installed from Image.......................... 85 4.10 Upgrading Graylog Originally Installed from Package......................... 85 4.11 Upgrading Elasticsearch......................................... 85 5 Configuring Graylog 93 5.1 server.conf................................................ 93 5.2 Web interface............................................... 106 5.3 Load balancer integration........................................ 111 i 5.4 Using HTTPS.............................................. 113 5.5 Multi-node Setup............................................. 120 5.6 Elasticsearch............................................... 122 5.7 Index model............................................... 134 5.8 Backup.................................................. 141 5.9 Default file locations........................................... 141 5.10 Graylog REST API............................................ 143 6 Securing Graylog 151 6.1 Default ports............................................... 151 6.2 Configuring TLS ciphers......................................... 151 6.3 Security related topics.......................................... 152 7 Sending in log data 165 7.1 What are Graylog message inputs?................................... 165 7.2 Content packs.............................................. 165 7.3 Syslog.................................................. 165 7.4 GELF / Sending from applications.................................... 166 7.5 Using Apache Kafka as transport queue................................. 167 7.6 Using RabbitMQ (AMQP) as transport queue.............................. 167 7.7 Microsoft Windows........................................... 167 7.8 Ruby on Rails.............................................. 167 7.9 Raw/Plaintext inputs........................................... 168 7.10 JSON path from HTTP API input.................................... 168 7.11 Reading from files............................................ 169 7.12 Input Throttling............................................. 170 8 Graylog Sidecar 171 8.1 Installation................................................ 172 8.2 Sidecar Configuration.......................................... 174 8.3 Step-by-step guide............................................ 176 8.4 Creating a new Log Collector...................................... 180 8.5 Using Configuration Variables...................................... 182 8.6 Secure Sidecar Communication..................................... 184 8.7 Run Sidecar as non-root user....................................... 185 8.8 Upgrading from the Collector Sidecar.................................. 186 8.9 Sidecar Glossary............................................. 189 8.10 Debug................................................... 189 8.11 Uninstall................................................. 190 8.12 Known Problems............................................. 190 9 Searching 191 9.1 Search query language.......................................... 191 9.2 Time frame selector........................................... 193 9.3 Saved searches.............................................. 195 9.4 Histogram................................................ 196 9.5 Analysis................................................. 197 9.6 Decorators................................................ 199 9.7 Export results as CSV.......................................... 203 9.8 Search result highlighting........................................ 204 9.9 Search configuration........................................... 205 10 Extended Search 209 10.1 Views................................................... 209 10.2 Widgets.................................................. 209 ii 10.3 Aggregation............................................... 210 10.4 Message Table.............................................. 212 10.5 Value and Field Actions......................................... 212 11 Streams 215 11.1 What are streams?............................................ 215 11.2 How do I create a stream?........................................ 216 11.3 Index Sets................................................ 216 11.4 Outputs.................................................. 218 11.5 Use cases................................................. 218 11.6 How are streams processed internally?................................. 218 11.7 Stream Processing Runtime Limits................................... 219 11.8 Programmatic access via the REST API................................. 221 11.9 FAQs................................................... 222 12 Alerts 225 12.1 Alerts & Events............................................. 225 12.2 Defining an Event............................................ 225 12.3 All events stream............................................. 237 13 Dashboards 239 13.1 Why dashboards matter......................................... 239 13.2 How to use dashboards.......................................... 240 13.3 Examples................................................. 242 13.4 Widgets from streams.......................................... 243 13.5 Result................................................... 243 13.6 Widget types explained.......................................... 243 13.7 Modifying dashboards.......................................... 245 13.8 Dashboard permissions.......................................... 246 14 Extractors 249 14.1 The problem explained.......................................... 249 14.2 Graylog extractors explained....................................... 249 14.3 Import extractors............................................. 251 14.4 Using regular expressions to extract data................................ 253 14.5 Using Grok patterns to extract data................................... 253 14.6 Using the JSON extractor........................................ 255 14.7 Automatically extract all key=value pairs................................ 255 14.8 Normalization.............................................. 257 15 Processing Pipelines 261 15.1 Pipelines................................................. 261 15.2 Rules................................................... 262 15.3 Stream connections............................................ 265 15.4 Functions................................................. 265 15.5 Usage................................................... 287 16 Lookup Tables 293 16.1 Components............................................... 293 16.2 Setup................................................... 295 16.3 Usage................................................... 298 16.4 Built-in Data Adapters.......................................... 300 17 Geolocation 303 17.1 Setup................................................... 303 iii 17.2 Visualize geolocations in a map..................................... 304 17.3 FAQs................................................... 307 18 Indexer failures 309 18.1 Common indexer failure reasons..................................... 309 19 Users and Roles 311 19.1 Users................................................... 311 19.2 Roles................................................... 312 19.3 Permission system............................................ 317 19.4 External authentication.......................................... 319 19.5 Authentication providers......................................... 324 20 Integrations 327 20.1 Integrations Setup............................................ 327 20.2 Open Source............................................... 328 20.3 Enterprise................................................