Facing the challenge(s) of Windows logs collection to leverage valuable IOCs . Michel de Crevoisier Security Analyst, Radar Cyber Security 15.10.2019, Berne © RadarServices // Classification: Public The five challenges © RadarServices // Classification: Public #1 High diversity of log sources Server Microsoft 3rd party Built-in roles software software Advanced Threat ADFS Application Analytics (ATA) Ivanti software Certification authority Exchange PowerShell Kaspersky DHCP server Skype Security DNS server SQL Server Veeam Backup System IIS web server SYSMON […] […] NPS Radius Defender © RadarServices // Classification: Public 3 #2 Different log extensions EVTX ETL TXT (standard Windows logs (analytical logs, like DNS (IIS, NPS, DHCP, in XML format) Server or PowerShell) PowerShell Transcript, former DNS logs) © RadarServices // Classification: Public 4 #3 Multiple architectural approaches Access method / Protocol (MS-EVEN6, RPC, WMI,…) Push vs Pull Agent vs Agentless Intermediate collector VS Direct sending to receiver Central file store vs Shared folder Managed agent VS Unmanaged agent © RadarServices // Classification: Public 5 #4 Disabled and restrictive event logs • Protected users (if configured, on DCs only) Valuable event • LSA (Local Security Authority) logs disabled • IIS web server • DNS client Event logs with • SMB server restrictive • SMB client access • IIS web server © RadarServices // Classification: Public 6 6 #5 Operational constraints Security Data exchange Performance Configuration Environment • Avoid usage of • Data • High • Easy • Cloud high privileges encryption availability deployment • Domain VS • Isolation • Secured • Compression • Minimize Workgroup between authentication configuration • OT customer and method changes (Operational security • Low impact on Technology) provider operating system © RadarServices // Classification: Public 7 Collecting standard Windows logs © RadarServices // Classification: Public WEF/WEC introduction Unified & built-in solution to collect standard Windows logs WEF (Windows Event Forwarding) WEC (Windows Event Collector) Authentication and encryption Collects and stores all requested through Kerberos in a domain or events from WEF clients according TLS certificates in a Workgroup XML subscriptions Data exchange over WinRM (push High availability capacities where or pull) clients send events to each WEC collector XML-based language to control event IDs to collect or to suppress Certain 3rd party software can also: noisy events Emulate a WEC server by spoofing a Settings control over GPO WinRM listener (e.g.: SYSLOG-NG Premium, NXLog Enterprise, AlienVault USM > actually uses NXLog) EPS control rate Manage multiple WEC servers with a central management console (e.g.: SuperCharger from Logbinder) © RadarServices // Classification: Public 9 Who is publishing about WEF/WEC? HP/ArcSight, Australian Cyber Security, … 2017 2015 2013 2017 & 2019 © RadarServices // Classification: Public 10 WEF/WEC performance Scaling out Technical characteristics Limitations Up to 4.000 source clients per All collected events are saved in collector (source: Microsoft) Forwarded Events log file Average logging is 5.000 EPS, can All events are mixed without any go up to 10.000 EPS (source: Microsoft) tagging possibilities Maximum recommended size per Only standard event logs (EVTX) event log file: 4GB can be forwarded Maximum recommended size for all Windows logs files: 16GB Compression possible with event log size reduction © RadarServices // Classification: Public 11 WEF/WEC advanced approach The Palantir approach to the rescue Multiple event channels • Different size and rotation strategy • Channel can be tagged for SIEM ingestion • Channel can be placed on different storage for better performance Preconfigured subscriptions • XML query to specify the events to collect • Specify the event channel destination © RadarServices // Classification: Public 12 WEF/WEC advanced approach A look in production on a WEC server Deployment is Requires several not automatized manual actions Potential source of incorrect configuration Event channels Subscriptions © RadarServices // Classification: Public 13 WEF/WEC deployment enhancement PowerShell at the rescue Automated Automated WEC server Palantir toolset role setup deployment Covers event Adjusts log file channel and size and subscriptions location Fixes SDDL Available on permissions on GitHub WinRM service https://github.com/rs-dev/windows-event-collector_auto-deploy © RadarServices // Classification: Public 14 WEF/WEC Injecting data with agent from the WEC server to your SIEM ArcSight agent NXLog agent Community RSYSLOG agent JSON Snare agent Source clients WEC collector SIEM CEF Splunk UF agent WinCollect agent Chosen agent software solution Winlogbeat agent Other targetOther / Externaltarget / provider /External Archiving provider solution © RadarServices // Classification: Public 15 WEF/WEC Injecting data without agent from the WEC server to your SIEM NXLog agent Enterprise SYSLOG-NG Premium Source clients SIEM Certificates are required Certificates pushed on hosts on each source client ! Chosen software for WinRM server listener emulation © RadarServices // Classification: Public 16 Collecting Windows DNS transaction logs © RadarServices // Classification: Public Collecting DNS transaction logs Technical possibilities overview DNS transactions logs Linux/Unix Windows OS Passive DNS OS Bind, DNS server DNS client Firewall or 3rd NIDS solution Unbound, logs logs party solution Dnsmasq, … Mirrored traffic 1 2 3 DNS DNS Event SYSMON ETW ETL debugging log (ID 22) Server 2012 R2 Disabled © RadarServices // Classification: Public 18 1 Collecting DNS transaction logs Old school approach with Debugging DNS logs Very simple High impact on access performance Only for Not supported debugging by MS for purpose production Does not Timestamp include DNS structure may answer change Delay before data is written No event ID (>1min) © RadarServices // Classification: Public 19 2 About ETW Event Tracing for Windows Efficient kernel-level tracing facility that allows to save kernel or application-defined events Allows to dynamically enable or disable logging in real time without any restart of the system Great open source projects available: KrabsETW (Microsoft) Performant C++ library to interact with ETW (https://github.com/Microsoft/krabsetw) PowerKrabsEtw PowerShell module built around the KrabsETW APIs (https://github.com/zacbrown/PowerKrabsEtw) TA-DNSETW Splunk plugin to collect DNS events from ETW using "KrabsETW" (https://github.com/secops4thewin/TA-DNSETW) SilkETW (FireEye) Flexible C# ETW wrapper running as a service - Blackhat 19 (https://github.com/fireeye/SilkETW) NXLog Community Windows agent provided with a native ETW module (im_etw). Logs can be saved in a file and/or sent to a remote target © RadarServices // Classification: Public 20 2 Collecting DNS transaction logs Advanced approach with native ETW Solutions for production System tools: Low impact on Event ID •Built-in: Logman, Perfmon, Netsh •Installable: Xperf, Tracelog, NetMon, performance provided Microsoft MMA, Tracelogging DNS answer Not Splunk •App “TA-DNSETW”: read ETW using is provided compatible the KrabsETW library from Microsoft (but encoded) with WEC Requires NXLog Community agent or script No cache file •Built-in module to read and forward installation ETW logs © RadarServices // Classification: Public 21 3 About ETL Event Tracing Logs ETW trace session are saved into ETL log files ETL files can be placed on a shared folder on each DNS server to be read remotely Great open source tools available: ETL-to-EVTX PowerShell script that reads ETL logs and writes them into Windows Event Viewer (https://github.com/acalarch/ETL-to-EVTX) ETLParser (GCPartners) Executable which can decodes several types of ETL files (https://github.com/gcpartners/ETLParser) DNSplice Python script that parses DNS ETL files (https://github.com/nerdiosity/DNSplice) DNS Analytical App PowerShell script for Splunk UF that reads ETL logs (Splunk) (https://splunkbase.splunk.com/app/2937) NXLog Community Windows agent provided with a native ETL module. Logs can be saved in a file and/or sent to a remote target ETW2JSON (Microsoft) Read ETL file and convert it to JSON (https://github.com/microsoft/ETW2JSON) © RadarServices // Classification: Public 22 3 Collecting DNS transaction logs Advanced approach with ETL Solutions for production System tools: Low impact on Event ID •Built-in: Tracerpt performance provided •Installable: Microsoft Message Analyzer (MMA) ETL file can be DNS answer is Splunk placed in a provided (but •App “DNS analytical”: PowerShell shared folder encoded) script that extracts ETL logs and send it to a remote listener Not compatible NXLog Community with WEC per •Built-in module to read and forward default (*) ETL logs (**) *ETL-to-EVTX script can convert ETL logs to EVTX log file **Currently in preview. Will be fully released in NXLog agent v5 © RadarServices // Classification: Public according NXLog support 23 Steps and solutions overview © RadarServices // Classification: Public Overview of collecting methods 1: requires PowerShell script that extracts ETL content into EVTX log files 4: not recommended, requires to query SCCM SQL Server database 2: requires agent or plugin with ETL or ETW capacities 5: requires SQL Server advanced configuration 3: data in event log has no structure 6: pulling requires dealing with firewall, credentials and double NAT issues 7: only a limited set of logs are available.