One security engineer’s trials and tribulations attempting to comprehend one of the least known but most powerful Windows services.
Before reading this post, please be sure to read @jepayneMSFT‘s excellent post on Windows Event Forwarding: Monitoring what matters — Windows Event Forwarding for everyone
Additionally, also check out Microsoft’s Use Windows Event Forwarding to help with intrusion detection
Introduction to Windows Event Forwarding
If you’re new to the concept of Windows Event Forwarding (WEF), the long story short is that a service exists in Windows where you can specify one or more servers to operate as Windows Event Log collectors. These collectors server as subscription managers and allow you to cherry pick which event logs you would like to collect from endpoints and the forwarded logs are then stored in buckets on the collectors. When using the Windows Event Forwarding service, the event logs are transferred natively over WinRM, which means you don’t have to worry about installing any sort of log forwarder software (Splunk/WinLogBeat/etc) on all of your endpoints to send logs to a centralized location.
Aside from the obvious benefit of not having to deploy any additional software, there are some great benefits to using WEF:
• Events are encrypted via Kerberos by default• Subscriptions can be created as XML files and backed by versioning software like git (just be sure to validate the XML!)• New machines are automatically enrolled into the logging infrastructure after joining the domain• WEF can be configured in either push or pull mode• The forwarding interval can be modified
Although there’s no shortage of documentation highlighting the benefits of using WEF, the primary focus of this writeup is to explain all of the components that need to exist for WEF to operate correctly and to document strategies for troubleshooting when things aren’t working as expected.
Core Components of a WEF Installation
There are countless guides on how to configure WEF available online of varying quality. The core components of the installation are:
- One or more servers to operate as the subscription manager and log collectors with the Windows Event Log Collector service running.
- All endpoints and subscription managers must have WinRM enabled.
- A GPO specifying the URL of the subscription manager(s). A tip I recently learned from Jessica’s post is that you can specify the check-in interval in this GPO by setting the refresh interval as shown in the screenshot below. Setting a short interval makes troubleshooting much easier in the long run but may be too noisy for production.
4. One or more event log subscriptions. A subscription is a collection of events based on Event ID’s or other criteria that tell the endpoints which event logs to forward. NSA’s Information Assurance group has an excellent set of subscriptions to use as a baseline:
https://github.com/iadgov/Event-Forwarding-Guidance/tree/master/Subscriptions/samples
5. A GPO to add the NETWORK SERVICE account to the Event Log Readers group.
6. A GPO to set ACLs on all relevant event log channels to allow read access by the Event Log Readers group. Many channels include that ACL by default, but the Security and other custom logs under the Microsoft/Windows service logs do not. More on creating that GPO here: https://support.microsoft.com/en-us/help/323076/how-to-set-event-log-security-locally-or-by-using-group-policy
7. Any organization who takes security seriously should have a GPO to enable enhanced auditing on windows enabled across their fleet: https://technet.microsoft.com/en-us/library/dn319056(v=ws.11).aspx
These seven items may sound relatively straightforward, but there are a surprising number of areas where it’s easy to make a mistake. Additionally, many aspects of WEF are unintuitive and make working with it less than user-friendly. In the next section, we’ll cover the tools that are available to gain better insights into your WEF configuration.
Gaining Insight Into Your WEF Infrastructure
Once you’ve completed setup on your WEF infrastructure, you may find that hosts are not checking in, or certain events are not being forwarded. It can be confusing to understand why things are happening, but this section aims to give you the tools you need to effectively troubleshoot an installation.
The Event Log Console
On your WEF subscription manager you can see realtime statistics about your subscriptions via the Event Viewer console by selecting a subscription and clicking “Runtime status”:
Subscriptions can also be edited by choosing “Properties > Select Events”. NOTE: It’s possible to accidentally load subscriptions with broken XML via the wecutil command (ask me how I know).
wecutil and wevtutil
wecutil (Windows Event Collector Utility) and wevtutil (Windows Event Utility) can also help you gain valuable insight into the current state of your subscriptions.
You can see the last time a host checked in to a specific subscription by running wecutil from a subscription manager:
PS C:\Windows> wecutil gr DNSSubscription: DNSRunTimeStatus: ActiveLastError: 0EventSources:dc.windomain.localRunTimeStatus: ActiveLastError: 0LastHeartbeatTime: 2017–07–22T07:54:27.296
TIP: To force a check in from a host, run gpupdate /force.
wevtutil is equally useful from the endpoint side to view things like event log channel ACLs:
PS C:\Windows> wevtutil get-log securityname: securityenabled: truetype: AdminowningPublisher:isolation: CustomchannelAccess: O:BAG:SYD:(A;;0xf0005;;;SY)(A;;0x5;;;BA)(A;;0x1;;;S-1–5–32–573)(A;;0x1;;;S-1–5–20)logging:logFileName: %SystemRoot%\System32\Winevt\Logs\security.evtxretention: falseautoBackup: falsemaxSize: 20971520publishing:fileMax: 1
NOTE: If a channel you are attempting to pull events from does not contain the SID for Event Log Viewers (A;;0x1;;;S-1–5–32–573), it will not be able to be read and an error message will be generated in the log channel documented below.
Microsoft-Windows-Eventlog-ForwardingPlugin Event Log Channel
This channel is a great place to look when trying to determine why events from specific subscriptions aren’t coming in.
Microsoft-Windows-Eventlog-ForwardingPlugin/Operational
This log exists deep within the event viewer on each enrolled endpoint that logs windows event forwarding runtime status information. Unfortunately for us, the error codes displayed in these messages don’t seem to be documented anywhere. Fortunately for you, I’ve anecdotally determined what some of them mean.
Here are the following errors I’ve come across in this logfile:
Error Message: The subscription $name can not be created. The error code is 15008.Issue: The subscription XML is malformed or the subscription is somehow invalidSolution: Update the XML to be valid.
Error Message: The subscription $name is created, but one or more channels in the query could not be read at this time.Issue: Event Log Reader group does not have read access to a channel specified in that subscriptionSolution: Use wevtutil to view the current ACL on the relevant channels and create/update a GPO to allow read access to that log channel
Error Message: The subscription $name can not be created. The error code is 5004.Issue: A channel specified in the description does not exist on the hostSolution: This is not necessarily a problem. If you have a subscription meant for DNS servers and only your Domain Controllers run DNS, you might see this error message on any server that doesn’t have the corresponding DNS server log channel.
Wrapping It Up
Properly administering WEF infrastructure requires a wide range of skills. The user should have a solid understanding of Group Policy, Event Log ACLs, Advanced Auditing, Powershell, and Active Directory. It’s tricky to properly configure WEF initially, but in my experience it has been rock solid and extremely performant. I highly recommend configuring WEF if you have a Windows domain in your environment and I hope this guide helps you work through any problems you may encounter along the way!