Collecting logs from Windows Event Log
Windows Event Log captures system, security, and application logs on Windows operating systems. It serves as a repository of detailed events generated by the system and is the first resource IT administrators refer to when troubleshooting issues. Besides resolving problems, you can use Windows events to monitor, analyze, and satisfy compliance mandates.
NXLog is a versatile and efficient log collection solution to collect and aggregate logs from Windows Event log to any centralized log collection destination, be it a SIEM, database, or file server for log archival. NXLog can collect all Windows logs from most modern Windows systems, either natively via ETW, directly from Windows Event Log, local log files, or remotely from Windows systems that forward events over the network. NXLog’s advanced log collection, processing, and forwarding capabilities make it the ideal candidate for collecting Windows Event Logs.
About Windows Event Log
Unlike other event logs, such as the UNIX syslog, Windows Event Log is not stored as a plain text file but in a proprietary binary format. Therefore, viewing the logs in a text editor or sending them as syslog messages is impossible while retaining their original format. However, the raw event log data can be translated into XML using the Windows Event Log API.
The EVTX file format
Windows has stored Windows Event Log files in the EVTX file format since the release of Windows Vista and Windows Server 2008.
Before that, event log files were stored in the EVT file format.
Both are proprietary formats readable by the Microsoft Management Console (MMC) snap-in eventvwr.msc, commonly known as Windows Event Viewer.
The EVTX file format has many enhancements over its predecessor. It includes new event properties, channels to publish events, a new Event Viewer, a rewritten Windows Event Log service, and support for the Extensible Markup Language (XML) format. From a log collection perspective, the added support for XML is the most important feature since it allows sharing or processing event data in a structured format.
Windows saves EVTX files for built-in channels in the C:\Windows\System32\winevt\Logs\ directory.
You can export events manually from Event Viewer in four formats: EVTX, XML, TXT, and CSV.
NXLog can read EVTX and EVT files using the File directive of the im_msvistalog module. In addition, the CaptureEventXML directive of the same module allows you to store raw XML-formatted event data.
Viewing logs in the Windows Event Log
You can examine logs in Windows Event Log in the Event Viewer MMC snap-in included in Windows. Since the stored logs do not have the entire message (only the event properties are stored), Event Viewer inserts property values into a corresponding localized template to display the full event details.
Windows Event Viewer includes three views for displaying event data. When you double-click on an event, these are displayed on the preview pane or in the Event Properties window.
-
The General view is displayed by default. It includes the entire event message rendered from a template and standard event properties.
-
You can switch to the Friendly View by clicking on the Details tab. This view shows a hierarchical list of standard and event-specific properties. It does not show the descriptive message from the event template.
-
You can also switch to the XML View from the Details tab to see all the event properties in XML format.
A Windows event in XML format<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" /> <EventID>4624</EventID> [...] <Channel>Security</Channel> <Computer>USER-WORKSTATION</Computer> <Security /> </System> <EventData> <Data Name="SubjectUserSid">S-1-5-18</Data> [...] </EventData> </Event>
Consumers access events through the Event Log API. Refer to Windows Event Log Functions on Microsoft Learn for more information. In particular:
-
EvtQuery() to fetch events from a channel or log file that matches a given query. See Querying for Events.
-
EvtFormatMessage() generates the event message using the event properties and the localized template. See Formatting Event Messages.
Event log channels
The EVTX format introduces event channels. A channel is a stream of events collected from a publisher and written to a log file.
Channels are organized into two groups:
-
Windows Logs contains five channels for
Security,Setup,Forwarded Events,System, andApplicationevent logs. -
Applications and Services Logs have channels created for individual applications and components.
Two types of channels for different purposes are available:
-
Serviced channels are relatively low volume and offer reliable log delivery. Event collectors can subscribe to these channels, and you can forward events from them to another system.
-
Direct channels are for high-performance log collection and are disabled by default. It is not possible to subscribe to a direct channel. To see these channels in Windows Event Viewer, enable Show Analytic and Debug Logs in the View menu. To enable logging for a direct channel, right-click it and select Properties, then check the Enable logging option on the General tab.
The above channel types are divided further according to the intended audience:
-
Administrative channels are serviced channels that collect log events intended for end users, administrators, and support.
-
Operational channels are serviced channels collecting log events intended for troubleshooting.
-
Analytic channels are direct channels and contain log events that describe program operations. These channels often collect a high volume of events.
-
Debug channels are direct channels intended to be used by developers only for debug logging.
| Channel groups | Channels | Channel type |
|---|---|---|
Windows Logs |
Application |
Administrative (serviced) |
Security |
Administrative (serviced) |
|
Setup |
Operational (serviced) |
|
System |
Administrative (serviced) |
|
Forwarded Events |
Operational (serviced) |
|
Applications and Services Logs |
DHCP-Server/Admin |
Administrative (serviced) |
DHCP-Server/AuditLogs |
Analytic (direct) |
|
DHCP-Server/DebugLogs |
Debug (direct) |
|
(And many more publisher-defined channels) |
||
The im_msvistalog module can collect logs from a specific channel with the Channel directive.
For more information about event channels, see Event Logs and Event Logs and Channels in Windows Event Log on Microsoft Learn.
Event log providers
Providers write events to the event logs. An event provider can be a service, driver, or program that runs on the computer and has the necessary instrumentation to write the events.
Event providers are categorized into four main types:
-
Manage Object Format (MOF) providers (also referred to as "classic")
-
Windows Software Trace Preprocessor (WPP) providers
-
Manifest-based providers
-
TraceLogging providers
For more information, see Providers on Microsoft Learn.
Windows event collection
NXLog provides the following modules for capturing Windows events.
-
The im_msvistalog module is available on Windows only and captures event log data from Windows 2008/Vista and later. It can collect logs locally or from a remote system via MSRPC (NXLog Enterprise Edition only). See Local Windows log collection with im_msvistalog and Remote Windows log collection with im_msvistalog.
-
The im_wseventing module is available on Linux, macOS and Windows (NXLog Enterprise Edition only). It can receive event log data from remote Windows systems via Windows Event Forwarding. See Remote Windows log collection with im_wseventing. We recommend this module for remote log collection because of the ease of configuring WEF clients through GPO.
-
The im_mseventlog module is available only on Windows and captures local events from Windows XP, 2000, and Windows 2003. See Local Windows log collection with im_mseventlog.
Local Windows log collection with im_msvistalog
The im_msvistalog module can collect logs locally from Windows 2008/Vista and later.
| Because the Windows Event Log subsystem does not support subscribing to the Debug and Analytic channels, the im_msvistalog module cannot collect events from these channels. |
This configuration reads all event logs from the local Windows Event Log. It then converts the logs to JSON format using the to_json() procedure of the xm_json module and writes them to a file.
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
</Input>
<Output file>
Module om_file
File 'C:\logs\windows_events.log'
Exec to_json();
</Output>For information about filtering Windows logs, see Filtering Windows logs.
Remote Windows log collection with im_msvistalog
NXLog Enterprise Edition can collect event logs from remote Windows systems with the im_msvistalog module. In this mode, you do not need to install the NXLog agent on each Windows system; instead, a central agent collects the events from them via MSRPC.
| Because the Windows Event Log subsystem does not support subscribing to the Debug and Analytic channels, the im_msvistalog module cannot collect events from these channels. |
This configuration uses the im_msvistalog module to collect Windows logs from a remote server named MYSERVER.
To adapt this example to your environment, specify the relevant RemoteServer, RemoteUser, RemoteDomain, and RemotePassword for your target machine.
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='1'>
<Select Path='Application'>*</Select>
<Select Path='Security'>*[System/Level=4]</Select>
<Select Path='System'>*</Select>
</Query>
</QueryList>
</QueryXML>
RemoteServer MYSERVER
RemoteUser Administrator
RemoteDomain mydomain.local
RemotePassword secret
</Input>Local Windows log collection with im_mseventlog
The im_mseventlog module captures event logs from Windows XP, Windows 2000, and Windows 2003.
The module retrieves the available log sources from the SYSTEM\CurrentControlSet\Services\Eventlog registry key and polls logs from all the sources.
You can also specify the log sources you want to collect with the Sources directive.
This example shows the most basic configuration of the im_mseventlog module. It collects Windows logs from all registered sources and forwards them to a remote server via TCP.
<Input eventlog>
Module im_mseventlog
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.1:514
</Output>Remote Windows log collection with im_wseventing
NXLog Enterprise Edition provides the im_wseventing module for receiving Windows event logs from remote machines via Windows Event Forwarding (WEF). This module is available for all supported Operating systems.
This configuration listens for connections on port 5985 from all WEF clients. It expects to receive events over HTTPS.
<Input wseventing>
Module im_wseventing
ListenAddr 0.0.0.0
Port 5985
Address https://linux.example.com:5985/wsman
HTTPSCertFile %CERTDIR%/server-cert.pem
HTTPSCertKeyFile %CERTDIR%/server-key.pem
HTTPSCAFile %CERTDIR%/ca.pem
<QueryXML>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">*</Select>
<Select Path="Microsoft-Windows-Winsock-AFD/Operational">*</Select>
<Select Path="Microsoft-Windows-Wired-AutoConfig/Operational">
*
</Select>
<Select Path="Microsoft-Windows-Wordpad/Admin">*</Select>
<Select Path="Windows PowerShell">*</Select>
</Query>
</QueryList>
</QueryXML>
</Input>You can filter for specific hosts by adding the <Computer> tag to the QueryXML configuration block.
This tag expects a pattern that NXLog will match against the name of the connecting Windows client.
If the computer name does not match the specified pattern, NXLog will not collect its events.
The following QueryXML block, if added to the above configuration, would provide an alternative query for computer names matching the pattern foo*.
In contrast, other computers will use the QueryXML without the <Computer> tag.
<QueryXML>
<Computer>foo*</Computer>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">*</Select>
</Query>
</QueryList>
</QueryXML>Filtering Windows logs
Applications and services on Windows can generate a large volume of logs, and it is often necessary to collect a subset of the events. There are several ways to filter logs from the Windows Event Log using the im_msvistalog module.
-
You can collect all log events from a specific channel with the Channel directive.
-
You can specify an XPath query with the Query or QueryXML directives. An XPath query allows you to subscribe to multiple channels and filter logs by various attributes. However, XPath queries have a maximum length, limiting the possibilities for detailed event subscriptions. See the XPath filtering with XPath queries below.
-
You can process a log file with the File directive, in which case im_msvistalog will read all logs from the
.evtxfile. This method is intended primarily for forensics, for example, using the nxlog-processor to process historical data. -
After reading log events from the source, you can selectively discard events using the drop() procedure within an Exec block.
Subscribing to a restricted set of logs with an XPath query provides a performance advantage because NXLog will not collect unnecessary events in the first place. However, XPath queries have a maximum length and limited filtering capabilities, so you may need to combine XPath filtering with post-collection processing. For examples, see Windows event IDs to monitor section.
XPath filtering with XPath queries
Windows Event Log supports a subset of XPath 1.0. XPath queries can subscribe to events matching specific criteria with Windows Event Viewer and the im_msvistalog QueryXML directive. For more information, see Consuming Events on Microsoft Learn.
Windows Event Viewer offers the most practical way to write and test queries. For example, you can test an XPath query by filtering the current log or creating a custom view.
-
In Event Viewer, click on an event channel, then right-click it and choose Filter Current Log… from the context menu. A dialog box showing basic filtering options will be displayed.
-
Specify the desired criteria. Switch to the XML tab to see the corresponding XPath query. You can copy this query to the im_msvistalog QueryXML directive.
-
You can enable advanced filtering by selecting the Edit query manually checkbox on the XML tab. We recommend verifying that the query matches the correct events before copying it to the NXLog configuration.
Figure 1. A Custom View querying the Application Channel for events with EventID 1008
Sometimes in NXLog it is helpful to use a query with sources that may not be available.
In this case, set the TolerateQueryErrors directive to TRUE to ensure that the module will continue to collect logs.
This configuration only collects Sysmon operational events from the local Windows Event Log.
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
</Input>This configuration queries the System event log for events with levels below 4 (Critical, Error, and Warning).
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path='System'>*[System/Level<4]</Select>
</Query>
</QueryList>
</QueryXML>
</Input>Post-collection log filtering
You can filter logs using NXLog’s built-in log processing capabilities. For example, you can match events against any of the im_msvistalog_ fields and use the drop() procedure to delete unwanted events.
This example discards all Sysmon logs with event ID 3 for HTTP network connections to a particular server and all process creation and termination events (event IDs 1 and 5) for conhost.exe.
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if ($EventID in (1, 5) and
$Image == "C:\\Windows\\System32\\conhost.exe") or
($EventID == 3 and
$DestinationPort == 80 and
$DestinationIp == 10.0.0.1) {
drop();
}
</Exec>
</Input>Trimming Windows logs
Further to filtering for only necessary logs, trimming helps you to reduce their size. For example, starting from Windows Server 2008, some log events include a message describing the event’s purpose. While such messages might be helpful for manual troubleshooting, they are unnecessary for archiving and processing by SIEMs and log analytics platforms. Consider, for example, event ID 4624. Each event logged contains the following text in addition to the event data:
This event is generated when a logon session is created. It is generated on the computer that was accessed.
The subject fields indicate the account on the local system which requested the logon.
This is most commonly a server service or a local process such as Winlogon.exe or Services.exe.
The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).
The New Logon fields indicate the account for which the new logon was created, i.e., the account logged on.
The network fields indicate where a remote logon request originated.
The workstation name is not always available and may sometimes be left blank.
The impersonation level field indicates the extent to which a process in the logon session can impersonate.
The authentication information fields provide detailed information about this specific logon request.
- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key.
This would be 0 if no session key was requested.When dealing with thousands or millions of logs, processing and storing this log data for every event unnecessarily increases the network load and storage requirements. Removing descriptive log messages and other unnecessary information can reduce the amount of data by half, helping to drive down costs related to network bandwidth and disk space and making a substantial difference for SIEMs that charge by the amount of ingested data.
NXLog provides several options to trim event data depending on your use case and SIEM requirements.
Truncating the Message field of a Windows log
The im_msvistalog module populates the $Message field with the entire event log message, including any descriptive text.
If your SIEM accepts unstructured data, such as syslog or Snare format, NXLog uses this field to format the output data.
However, the event descriptions are usually not required by SIEMs and can be removed to reduce the log size significantly.
For example, the following table shows data for a sample event with event ID 4624 in syslog format.
| Log Size | Decrease | |
|---|---|---|
Original event |
2.17KB |
- |
Event with truncated Message field |
1006B |
57% |
There are two methods you can use to truncate text fields with NXLog:
-
You can define rules and apply actions to matching logs using the xm_pattern module. This is the preferred method since xm_pattern does not compare rules linearly, making it more efficient.
-
If you require linear comparison, you can use Regular Expressions.
Deleting fields from Windows logs
SIEMs capable of ingesting structured data are often pre-loaded with standard event information, such as the event type, category, and severity, mainly for security logs.
Therefore, fields like the $TaskValue, $Opcode, and $OpcodeValue can be removed before forwarding.
In addition, for SIEMs that do not require the original event message, the $Message field can also be removed since it is redundant.
The following table contains data for a sample log with event ID 4624 in JSON format.
| Log size | Fields | Decrease | |
|---|---|---|---|
Original event |
3.78KB |
50 |
- |
Event with truncated Message |
2.50KB |
50 |
34% |
Event with truncated Message + deleted standard fields |
2.18KB |
38 |
42% |
Deleted Message field |
1.45KB |
49 |
62% |
Deleted Message field + standard fields |
1.12KB |
37 |
70% |
There are two methods you can use to delete fields with NXLog:
-
You can define a field allowlist or blocklist using the xm_rewrite module. This is the preferred method when you need to delete several fields.
-
You can use the delete() procedure to delete individual fields.
Example configurations for trimming Windows logs
This xm_pattern pattern file compares Windows security logs based on their event IDs.
Each rule defines an Exec block so that the description is removed from the $Message field for each matching event.
You can also define regex patterns for a more generic configuration; however, regex patterns are less efficient than exact patterns and may delay log processing if used excessively.
See the xm_pattern documentation for an example.
<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
<created>2022-07-23 11:18:18</created>
<version>1</version>
<group>
<name>Security</name>
<id>1</id>
<matchfield>
<name>Channel</name>
<type>exact</type>
<value>Security</value>
</matchfield>
<!-- Remove Event Summary -->
<pattern>
<id>1</id>
<name>Security Event 4624</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4624</value>
</matchfield>
<exec>
$Message =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>2</id>
<name>Security Event 4625</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4625</value>
</matchfield>
<exec>
$Message =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>3</id>
<name>Security Event 4634</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4634</value>
</matchfield>
<exec>
$Message =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>4</id>
<name>Security Event 4648</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4648</value>
</matchfield>
<exec>
$Message =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>5</id>
<name>Security Event 4769</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4769</value>
</matchfield>
<exec>
$Message =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>6</id>
<name>Security Event 4616</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>4616</value>
</matchfield>
<exec>
$Messge =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<pattern>
<id>7</id>
<name>Security Event 4647</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>4647</value>
</matchfield>
<exec>
$Messge =~ s/\s*This event is generated.*$//s;
</exec>
</pattern>
<!-- Remove Certificate Summary -->
<pattern>
<id>8</id>
<name>Security Event 4769</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4768</value>
</matchfield>
<exec>
$Message =~ s/\s*Certificate information is only provided.*$//s;
</exec>
</pattern>
<pattern>
<id>9</id>
<name>Security Event 4769</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4771</value>
</matchfield>
<exec>
$Message =~ s/\s*Certificate information is only provided.*$//s;
</exec>
</pattern>
<!-- Remove Token Elevation Summary -->
<pattern>
<id>10</id>
<name>Security Event 4769</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>4688</value>
</matchfield>
<exec>
$Message =~ s/\s*Token Elevation Type indicates the type of.*$//s;
</exec>
</pattern>
<!-- Remove Stored Credentials Summary -->
<pattern>
<id>11</id>
<name>Security Event 5379</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>5379</value>
</matchfield>
<exec>
$Message =~ s/\s*This event occurs when a user performs a read operation on stored credentials in Credential Manager.*$//s;
</exec>
</pattern>
<!-- Remove Authentication Summary -->
<pattern>
<id>12</id>
<name>Security Event 4776</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>4776</value>
</matchfield>
<exec>
$Message =~ s/^The computer attempted to validate the credentials for an account.\n\n//s;
</exec>
</pattern>
<!-- Remove Network Share Summary -->
<pattern>
<id>13</id>
<name>Security Event 5140</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>5140</value>
</matchfield>
<exec>
$Message =~ s/^A network share object was accessed.\n\n//s;
</exec>
</pattern>
<pattern>
<id>14</id>
<name>Security Event 5144</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>5144</value>
</matchfield>
<exec>
$Message =~ s/^A network share object was deleted.\n\n//s;
</exec>
</pattern>
<pattern>
<id>15</id>
<name>Security Event 5145</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>5145</value>
</matchfield>
<exec>
$Message =~ s/^A network share object was checked to see whether client can be granted desired access.\n\n//s;
</exec>
</pattern>
<!-- Remove User/Device Claims Summary -->
<pattern>
<id>16</id>
<name>Security Event 4626</name>
<matchfield>
<name>EventID</name>
<type>Exact</type>
<value>4626</value>
</matchfield>
<exec>
$Message =~ s/\s*The subject fields indicate.*$//s;
</exec>
</pattern>
</group>
</patterndb>The NXLog configuration below collects security logs using the im_msvistalog input module. It processes each event according to the rules file above and converts the log record to syslog format.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension pattern>
Module xm_pattern
PatternFile 'C:\Program Files\nxlog\conf\patterndb.xml'
</Extension>
<Input SecurityEvents>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0' Path="Security">
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
match_pattern(); (1)
to_syslog_bsd(); (2)
</Exec>
</Input>| 1 | Processes the log event against the rules defined in the pattern file of the xm_pattern module instance. |
| 2 | Converts the log event to syslog using the to_syslog_bsd() procedure of the xm_syslog module. |
<14>Jul 26 10:12:02 WIN-F0RPR0BHTK0.domain.local Microsoft-Windows-Security-Auditing[0x0]: An account was successfully logged on. Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - Logon ID: 0x0 Logon Information: Logon Type: 3 Restricted Admin Mode: - Virtual Account: No Elevated Token: Yes Impersonation Level: Impersonation New Logon: Security ID: S-1-5-21-1241979678-2462307911-3370647597-1103 Account Name: WINAB-2JR3FR9RD$ Account Domain: DOMAIN.LOCAL Logon ID: 0x121355 Linked Logon ID: 0x0 Network Account Name: - Network Account Domain: - Logon GUID: {72e0f6ab-7332-5b0f-7ecb-96939bff476e} Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: - Source Network Address: 10.0.0.100 Source Port: 49748 Detailed Authentication Information: Logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0This configuration collects security logs using the im_msvistalog input module. It processes each event to remove unnecessary fields and then converts the log record to JSON format.
<Extension json>
Module xm_json
</Extension>
<Extension rewrite>
Module xm_rewrite
Delete SourceModuleName,SourceModuleType,Severity,SeverityValue, \
Opcode,OpcodeValue,TaskValue,ProviderGuid,EventType,Level, \
LevelValue,Version,Message
</Extension>
<Input SecurityEvents>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0' Path="Security">
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
rewrite->process(); (1)
to_json(); (2)
</Exec>
</Input>| 1 | Processes the event according to the xm_rewrite module instance configuration. |
| 2 | Converts the event to JSON format using the to_json() procedure of the xm_json module. |
{
"EventTime": "2022-07-26T10:12:02.577206+01:00",
"Hostname": "WIN-F0RPR0BHTK0.domain.local",
"Keywords": "9232379236109516800",
"EventID": 4624,
"SourceName": "Microsoft-Windows-Security-Auditing",
"RecordNumber": 2593,
"ActivityID": "{9F55F44E-A0CE-0003-C2F4-559FCEA0D801}",
"ExecutionProcessID": 636,
"ExecutionThreadID": 860,
"Channel": "Security",
"Category": "Logon",
"SubjectUserSid": "S-1-0-0",
"SubjectUserName": "-",
"SubjectDomainName": "-",
"SubjectLogonId": "0x0",
"TargetUserSid": "S-1-5-21-1241979678-2462307911-3370647597-1103",
"TargetUserName": "WINAB-2JR3FR9RD$",
"TargetDomainName": "DOMAIN.LOCAL",
"TargetLogonId": "0x121334",
"LogonType": "3",
"LogonProcessName": "Kerberos",
"AuthenticationPackageName": "Kerberos",
"WorkstationName": "-",
"LogonGuid": "{72e0f6ab-7332-5b0f-7ecb-96939bff476e}",
"TransmittedServices": "-",
"LmPackageName": "-",
"KeyLength": "0",
"ProcessId": "0x0",
"ProcessName": "-",
"IpAddress": "10.0.0.100",
"IpPort": "49732",
"ImpersonationLevel": "%%1840",
"RestrictedAdminMode": "-",
"TargetOutboundUserName": "-",
"TargetOutboundDomainName": "-",
"VirtualAccount": "%%1843",
"TargetLinkedLogonId": "0x0",
"ElevatedToken": "%%1842",
"EventReceivedTime": "2022-07-26T14:32:51.919931+01:00"
}Windows event IDs to monitor
One of the most challenging tasks regarding Windows log collection is deciding which event IDs to monitor. Due to the sheer number of event IDs, this can be daunting at first sight. Therefore, this section will guide you in selecting the event IDs to monitor and provide example configurations for collecting them.
| Event IDs are unique per source but are not globally unique. Different sources may use the same event ID to identify unrelated operations. |
Finding the right event IDs
An excellent general resource to start with is Windows 10 and Windows Server 2016 security auditing and monitoring reference. It provides detailed descriptions of event IDs used for security audit policies. Additional resources include:
-
The Microsoft Events and Errors page on Microsoft Learn provides a directory of events grouped by area. Start by navigating the areas listed in the Available Documentation section.
-
Palantir’s Windows Event Forwarding Guidance repository contains a comprehensive WEF Event Mappings table with categorized event IDs and details.
-
The JPCERT/CC Detecting Lateral Movements Tool Analysis provides a collection of event IDs that indicate lateral movements. See the configuration example below.
-
See the NXLog User Guide on Active Directory Domain Services for a list of security event IDs relevant to Active Directory and a configuration sample.
The table below displays a small sample of essential Security logs to monitor on Windows servers. In addition, see the Security-focused Event IDs to Monitor configuration examples below.
| Event ID | Description |
|---|---|
|
The audit log was cleared. |
|
An account failed to log on. |
|
A logon was attempted using explicit credentials. |
|
An operation was attempted on a privileged object. |
|
A service was installed in the system. |
|
A user right was assigned. |
|
A user right was removed. |
|
System security access was granted to an account. |
|
System audit policy was changed. |
|
An attempt was made to change an account’s password. |
|
A member was added to a security-enabled local group. |
|
A user account was changed. |
|
A user’s local group membership was enumerated. |
|
A change has been made to Windows Firewall exception list. A rule was added. |
|
A Windows Firewall setting has changed. |
|
A new external device was recognized by the system. |
|
The installation of this device was allowed, after having previously been forbidden by policy. |
Example configuration to collect Windows security logs
Once you have selected the event IDs you will monitor, you can configure the im_msvistalog module with the relevant query. Modify the configuration examples in this section to suit your deployment.
|
Due to a limitation in the Windows Event Log API, queries with more than 22 clauses will fail, producing the following error message:
The workaround for this limitation is grouping clauses and/or splitting the filter across multiple queries. In the example below, the filter consists of 6 EventIDs; however, these count as 2 in terms of the aforementioned limitation. Combining clauses in groups
|
| An XPath query that does not specify a provider will collect all matching event IDs globally from all providers. Make sure to configure your SIEM to associate events with the correct provider accordingly. |
This example provides a basic configuration for monitoring Windows security logs. It explicitly specifies a small number of event IDs to be collected.
<Input security_events>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0)
and (EventID=1102 or EventID=4719 or EventID=4704 or EventID=4717 or EventID=4738
or EventID=4798 or EventID=4705 or EventID=4674 or EventID=4697 or EventID=4648
or EventID=4723 or EventID=4946 or EventID=4950 or EventID=6416 or EventID=6424
or EventID=4732)]]
</Select>
</Query>
</QueryList>
</QueryXML>
</Input>This extended configuration defines groups of event IDs based on their source. Next, it uses the QueryXML directive to collect events from specific providers. Finally, it uses an Exec block to filter out event IDs that are not in the lists.
# define Account Usage Events
define AccountUsage 4740, 4648, 4781, 4733, 1518, 4776, 5376, 5377, \
4625, 300, 4634, 4672, 4720, 4722, 4782, 4793, \
4731, 4735, 4766, 4765, 4624, 1511, 4726, 4725, \
4767, 4728, 4732, 4756, 4704
# define Application Crash Events
define AppCrashes 1000, 1002, 1001
# define Application Whitelisting Events
define AppWhitelisting 8023, 8020, 8002, 8003, 8004, 8006, 8007, 4688, \
4689, 8005, 865, 866, 867, 868, 882
# define Boot Events
define BootEvents 13, 12
# define Certificate Services Events
define CertServices 95, 4886, 4890, 4874, 4873, 4870, 4887, 4885, \
4899, 4896, 1006, 1004, 1007, 1003, 1001, 1002
# define Clearing Event Logs Events
define ClearingLogs 1100, 104, 1102
# define DNS and Directory Services Events
define DNSDirectoryServ 5137, 5141, 5136, 5139, 5138, 3008, 3020
# define External Media Detection events
define ExtMedia 400, 410
# define Group Policy Error Events
define GroupPolicyError 112, 1001, 1125, 1126, 1127, 1129
# define Kernel Driver Signing Events
define KernelDriver 3001, 3002, 3003, 3004, 3010, 3023, 5038, \
6281, 219
# define Microsoft Cryptography API Events
define MSFTCryptoAPI 11, 70, 90
# define Mobile Device Activities
define MobileDeviceEvents 10000, 10001
# define Network Host Activities
define NetworkHost 4714, 4713, 4769, 6273, 6275, 6274, 6272, \
6278, 6277, 6279, 6276, 6280, 5140, 5145, \
5142, 5144, 4706, 1024, 4897, 4719, 4716, \
4779, 4778, 5632
# define PassTheHash Detection Events
define PassTheHash 4624, 4625
# define PowerShell Activities
define PowerShell 800, 169, 4103, 4104, 4105, 4106
# define Printing Services Events
define PrintingServices 307
# define Logon Events
define LogonEvents 4624, 4634
# define Software Service Installation Events
define Installation 903, 904, 6, 1022, 1033, 7045, 907, 908, 7000, \
800, 2, 905, 906, 19
# define System Integrity Events
define SystemIntegrity 4657, 1, 4616
# define System or Service Failure Events
define SystemServiceFail 7022, 7023, 7024, 7026, 7031, 7032, 7034
# define Task Scheduler Activities
define TaskScheduler 106, 141, 142, 200
# define Windows Defender Activities
define WinDefender 1008, 1006, 1116, 1010, 2003, 2001, 1009, 1118, \
1119, 1007, 1117, 3002, 2004, 1005, 5008
# define Windows Firewall Events
define WinFirewall 2009, 2004, 2005, 2006, 2033
# define Windows Update Error Events
define WinUpdateError 1009, 20, 24, 25, 31, 34, 35
<Input security_events>
Module im_msvistalog
TolerateQueryErrors TRUE
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Application">*</Select>
<Select Path="Security">*</Select>
<Select Path="System">*</Select>
<Select Path="Microsoft-Windows-Application-Experience/Program-Compatibility-Assistant">*</Select>
<Select Path="Microsoft-Windows-Application-Experience/Program-Compatibility-Troubleshooter">*</Select>
<Select Path="Microsoft-Windows-Application-Experience/Program-Inventory">*</Select>
<Select Path="Microsoft-Windows-Application-Experience/Program-Telemetry">*</Select>
<Select Path="Microsoft-Windows-Application-Experience/Steps-Recorder">*</Select>
<Select Path="Microsoft-Windows-AppLocker/EXE and DLL">*</Select>
<Select Path="Microsoft-Windows-AppLocker/MSI and Script">*</Select>
<Select Path="Microsoft-Windows-AppLocker/Packaged app-Deployment">*</Select>
<Select Path="Microsoft-Windows-AppLocker/Packaged app-Execution">*</Select>
<Select Path="Microsoft-Windows-CAPI2/Operational">*</Select>
<Select Path="Microsoft-Windows-CertificateServicesClient-Lifecycle-System/Operational">*</Select>
<Select Path="Microsoft-Windows-DNS-Client/Operational">*</Select>
<Select Path="Microsoft-Windows-GroupPolicy/Operational">*</Select>
<Select Path="Microsoft-Windows-Kernel-PnP/Configuration">*</Select>
<Select Path="Microsoft-Windows-NetworkProfile/Operational">*</Select>
<Select Path="Microsoft-Windows-NTLM/Operational">*</Select>
<Select Path="Microsoft-Windows-PowerShell/Admin">*</Select>
<Select Path="Microsoft-Windows-PowerShell/Operational">*</Select>
<Select Path="Microsoft-Windows-PrintService/Admin">*</Select>
<Select Path="Microsoft-Windows-PrintService/Operational">*</Select>
<Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>
<Select Path="Microsoft-Windows-TerminalServices-RDPClient/Operational">*</Select>
<Select Path="Microsoft-Windows-User Profile Service/Operational">*</Select>
<Select Path="Microsoft-Windows-Windows Defender/Operational">*</Select>
<Select Path="Microsoft-Windows-Windows Defender/WHC">*</Select>
<Select Path="Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurity">*</Select>
<Select Path="Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurityVerbose">*</Select>
<Select Path="Microsoft-Windows-Windows Firewall With Advanced Security/Firewall">*</Select>
<Select Path="Microsoft-Windows-Windows Firewall With Advanced Security/FirewallDiagnostics">*</Select>
<Select Path="Microsoft-Windows-Windows Firewall With Advanced Security/FirewallVerbose">*</Select>
<Select Path="Network Isolation Operational">*</Select>
<Select Path="Microsoft-Windows-WindowsUpdateClient/Operational">*</Select>
<Select Path="Windows PowerShell">*</Select>
<Select Path="Microsoft-Windows-CodeIntegrity/Operational">*[System[Provider[@Name='Microsoft-Windows-CodeIntegrity']]]</Select>
<Select Path="Microsoft-Windows-LSA/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if ($EventID NOT IN (%AccountUsage%)) and
($EventID NOT IN (%AppCrashes%)) and
($EventID NOT IN (%AppWhitelisting%)) and
($EventID NOT IN (%BootEvents%)) and
($EventID NOT IN (%CertServices%)) and
($EventID NOT IN (%ClearingLogs%)) and
($EventID NOT IN (%DNSDirectoryServ%)) and
($EventID NOT IN (%ExtMedia%)) and
($EventID NOT IN (%GroupPolicyError%)) and
($EventID NOT IN (%KernelDriver%)) and
($EventID NOT IN (%MSFTCryptoAPI%)) and
($EventID NOT IN (%MobileDeviceEvents%)) and
($EventID NOT IN (%NetworkHost%)) and
($EventID NOT IN (%PassTheHash%)) and
($EventID NOT IN (%PowerShell%)) and
($EventID NOT IN (%PrintingServices%)) and
($EventID NOT IN (%LogonEvents%)) and
($EventID NOT IN (%Installation%)) and
($EventID NOT IN (%SystemIntegrity%)) and
($EventID NOT IN (%SystemServiceFail%)) and
($EventID NOT IN (%TaskScheduler%)) and
($EventID NOT IN (%WinDefender%)) and
($EventID NOT IN (%WinFirewall%)) and
($EventID NOT IN (%WinUpdateError%)) {
drop();
}
</Exec>
</Input>Similar to the extended configuration above, this configuration defines groups of event IDs associated with detecting malicious lateral movements. It is based on the Computer Emergency Response Team (CERT) research on Detecting Lateral Movement through Tracking Event Logs.
# define Security Events
define SecurityEvents 4624, 4634, 4648, 4656, 4658, 4660, 4663, 4672, \
4673, 4688, 4689, 4698, 4720, 4768, 4769, 4946, \
5140, 5142, 5144, 5145, 5154, 5156, 5447, 8222
# define Sysmon Events
define SysmonEvents 1, 2, 5, 8, 9
# define Application Management event
define ApplicationMgmt 104
# define Windows Remote Management Events
define WRMEvents 80, 132, 143, 166, 81
# define Task Scheduler - Operational Events
define TaskSchedEvents 106, 129, 200, 201
# define Local Session Manager - Operational Events
define LocalSessionMgrEvents 21, 24
#define BitsClient Events
define BitsClientsEvents 60
<Input lateral_security_events>
Module im_msvistalog
TolerateQueryErrors TRUE
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*</Select>
<Select Path="System">*</Select>
<Select Path="Microsoft-Windows-Bits-Client/Operational">*</Select>
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>
<Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>
<Select Path="Microsoft-Windows-TerminalServices-LocalSessionManager/Admin">*</Select>
<Select Path="Microsoft-Windows-TerminalServices-LocalSessionManager/Operational">*</Select>
<Select Path="Microsoft-Windows-WinRM/Operational">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if ($EventID NOT IN (%SecurityEvents%)) and
($EventID NOT IN (%SysmonEvents%)) and
($EventID NOT IN (%WRMEvents%)) and
($EventID NOT IN (%TaskSchedEvents%)) and
($EventID NOT IN (%ApplicationMgmt%)) and
($EventID NOT IN (%LocalSessionMgrEvents%)) and
($EventID NOT IN (%BitsClientsEvents%)) {
drop();
}
</Exec>
</Input>Forwarding Windows event logs
After collecting logs from a Windows system with NXLog, you may need to send them to another host. This section provides details and configuration examples for forwarding log data.
Descriptions in Windows events may contain tabs and newlines, but some formats, like BSD syslog, do not support these characters. In this case, you can use a regular expression to remove them.
This configuration uses a regular expression to replace all tab characters and newline sequences in the event description with spaces.
<Input eventlog>
Module im_mseventlog
Exec $Message =~ s/(\t|\R)/ /g;
</Input>Forwarding Windows logs in BSD syslog format
You can convert Windows events to BSD syslog format with the to_syslog_bsd() procedure of the xm_syslog module. For more information, see Forward syslog to a remote logger via UDP, TCP, or TLS.
This configuration removes tab characters and newline sequences from the $Message field, converts the log record to BSD syslog format, and forwards the event via UDP.
<Extension syslog>
Module xm_syslog
</Extension>
<Input eventlog>
Module im_msvistalog
<Exec>
$Message =~ s/(\t|\R)/ /g;
to_syslog_bsd();
</Exec>
</Input>
<Output udp>
Module om_udp
Host 10.10.1.1:514
</Output>| The to_syslog_bsd() procedure will only use a subset of the Windows Event Log fields. |
<14>Sep 20 10:21:16 win7host Service_Control_Manager[448]: The Computer Browser service entered the running state.Forwarding Windows logs in JSON format
Most SIEM solutions accept logs in JSON format to preserve all event log fields. The xm_json module provides the to_json() procedure to convert data to JSON format. See the examples in the NXLog Enterprise Edition Reference Manual to better understand how NXLog parses and converts logs to JSON.
This configuration converts the event record to JSON format and forwards the event via TCP.
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
Exec to_json();
</Input>
<Output tcp>
Module om_tcp
Host 192.168.10.1:1514
</Output>{
"EventTime": "2022-09-20 10:21:16",
"Hostname": "win7host",
"Keywords": -9187343239835812000,
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 7036,
"SourceName": "Service Control Manager",
"ProviderGuid": "{525908D1-A6D5-5695-8E2E-26921D2011F3}",
"Version": 0,
"Task": 0,
"OpcodeValue": 0,
"RecordNumber": 2629,
"ProcessID": 448,
"ThreadID": 2872,
"Channel": "System",
"Message": "The Computer Browser service entered the running state.",
"param1": "Computer Browser",
"param2": "running",
"EventReceivedTime": "2022-09-20 10:21:17",
"SourceModuleName": "eventlog",
"SourceModuleType": "im_msvistalog"
}Some SIEMs require you to send data in syslog-encapsulated JSON.
This configuration converts the event record to JSON, adds a BSD syslog header, and forwards the event via UDP.
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input eventlog>
Module im_msvistalog
<Exec>
$Message = to_json();
to_syslog_bsd();
</Exec>
</Input>
<Output udp>
Module om_udp
Host 192.168.2.1:514
</Output><14>Sep 20 10:21:16 win7host Service_Control_Manager[448]: {"EventTime":"2022-09-20 10:21:16","Hostname":"win7host","Keywords":-9187343239835811840,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":7036,"SourceName":"Service Control Manager","ProviderGuid":"{525908D1-A6D5-5695-8E2E-26921D2011F3}","Version":0,"Task":0,"OpcodeValue":0,"RecordNumber":2629,"ProcessID":448,"ThreadID":2872,"Channel":"System","Message":"The Computer Browser service entered the running state.","param1":"Computer Browser","param2":"running","EventReceivedTime":"2022-09-20 10:21:17","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}Forwarding Windows logs in Snare format
Snare is another format commonly used with Windows logs. The xm_syslog module includes the to_syslog_snare() procedure to generate syslog-encapsulated Snare logs. See the Snare integration guide for more information.
This configuration removes tab characters and newline sequences from the $Message field, converts the event record to the Snare over syslog format, and forwards the event via UDP.
<Extension syslog>
Module xm_syslog
</Extension>
<Input eventlog>
Module im_msvistalog
<Exec>
$Message =~ s/(\t|\R)/ /g;
to_syslog_snare();
</Exec>
</Input>
<Output snare>
Module om_udp
Host 192.168.1.1:514
</Output><14>Sep 20 10:21:16 win7host MSWinEventLog 1 System 193 Mon Sep 20 10:21:16 2017 7036 Service Control Manager N/A N/A Information win7host N/A The Computer Browser service entered the running state. 2773Processing EVTX files on Linux
EVTX files collected from Windows systems can be processed on Linux with NXLog using an external Python script.
|
Because of the wide variety of Linux distributions available with their package requirements changing, we identified two methods for installing the python EVTX parser module. One may work for a particular Linux system, and the other may not. One of them is the python-evtx module, which you can install with The other is the Python 3 EVTX module, which you can install with |
This configuration uses the im_python module to execute a script and converts the logs to a JSON format. The Python script uses the Python EVTX module to read the events from the EVTX file.
<Extension xml>
Module xm_xml
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input evtx>
Module im_python
PythonCode /opt/nxlog/etc/process_evtx.py
<Exec>
parse_windows_eventlog_xml($Message);
delete($Message);
to_json();
</Exec>
</Input>import Evtx.Evtx as evtx
import Evtx.Views as e_views
import nxlog
def read_data(module):
nxlog.log_debug('Starting Processing EVTX')
with evtx.Evtx('/tmp/security.evtx') as log:
for record in log.records():
event = module.logdata_new()
event.set_field('Message', record.xml())
event.post(){
"EventReceivedTime": "2022-09-20T18:47:06.548281+01:00",
"SourceModuleName": "evtx",
"SourceModuleType": "im_python",
"SourceName": "Microsoft-Windows-Security-Auditing",
"ProviderGuid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
"EventID": 5379,
"Version": 0,
"LevelValue": 0,
"TaskValue": 13824,
"OpcodeValue": 0,
"Keywords": "0x8020000000000000",
"EventTime": "2022-09-16T17:36:15.751457+01:00",
"RecordNumber": 342859,
"ActivityID": "{d9a2c36d-bc3e-0003-83c3-a2d93ebcd601}",
"ExecutionProcessID": 872,
"ExecutionThreadID": 956,
"Channel": "Security",
"Hostname": "PC1",
"SubjectUserSid": "S-1-5-21-774634756-2905300177-2392840219-1003",
"SubjectUserName": "John",
"SubjectDomainName": "PC1",
"SubjectLogonId": "0x00000000000ccd46",
"TargetName": "MicrosoftOffice*",
"Type": "0",
"CountOfCredentialsReturned": "1",
"ReadOperation": "%%8100",
"ReturnCode": "0",
"ProcessCreationTime": "2022-09-16 17:36:12.644609",
"ClientProcessId": "12088",
"EventType": "AUDIT_SUCCESS",
"SeverityValue": 2,
"Severity": "INFO"
}| Sometimes, events from the Windows Event Log contain values that need to be resolved using an external reference. Processing EVTX files using the python-evtx library may result in some events containing unresolved values. |
