Reading Windows Logs efficiently and fast

You can give the EventLogReader class a try. See https://docs.microsoft.com/en-us/previous-versions/bb671200(v=vs.90).

It is better than the EventLog class because accessing the EventLog.Entries collection has the nasty property that its count can change while you are reading from it. What is even worse is that the reading happens on an IO threadpool thread which will let your application crash with an unhandled exception. At least that was the case some years ago.

The EventLogReader also gives you the ability to supply a query string to filter for the events you are interested in. That is the way to go if you write a new application.

Here is an application which shows how you can parallelize reading:

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.Eventing.Reader;
using System.Linq;
using System.Threading.Tasks;

namespace EventLogReading
{
    class Program
    {
        static volatile bool myHasStoppedReading = false;

        static void ParseEventsParallel()
        {
            var sw = Stopwatch.StartNew();
            var query = new EventLogQuery("Application", PathType.LogName, "*");

            const int BatchSize = 100;

            ConcurrentQueue<EventRecord> events = new ConcurrentQueue<EventRecord>();
            var readerTask = Task.Factory.StartNew(() =>
            {
                using (EventLogReader reader = new EventLogReader(query))
                {
                    EventRecord ev;
                    bool bFirst = true;
                    int count = 0;
                    while ((ev = reader.ReadEvent()) != null)
                    {
                        if ( count % BatchSize == 0)
                        {
                            events.Enqueue(ev);
                        }
                        count++;
                    }
                }
                myHasStoppedReading = true;
            });

            ConcurrentQueue<KeyValuePair<string, EventRecord>> eventsWithStrings = new ConcurrentQueue<KeyValuePair<string, EventRecord>>();

            Action conversion = () =>
            {
                EventRecord ev = null;
                using (var reader = new EventLogReader(query))
                {
                    while (!myHasStoppedReading || events.TryDequeue(out ev))
                    {
                        if (ev != null)
                        {
                            reader.Seek(ev.Bookmark);
                            for (int i = 0; i < BatchSize; i++)
                            {
                                ev = reader.ReadEvent();
                                if (ev == null)
                                {
                                    break;
                                }
                                eventsWithStrings.Enqueue(new KeyValuePair<string, EventRecord>(ev.FormatDescription(), ev));
                            }
                        }
                    }
                }
            };

            Parallel.Invoke(Enumerable.Repeat(conversion, 8).ToArray());

            sw.Stop();
            Console.WriteLine($"Got {eventsWithStrings.Count} events with strings in {sw.Elapsed.TotalMilliseconds:N3}ms");
        }

        static void ParseEvents()
        {
            var sw = Stopwatch.StartNew();
            List<KeyValuePair<string, EventRecord>> parsedEvents = new List<KeyValuePair<string, EventRecord>>();
                
            using (EventLogReader reader = new EventLogReader(new EventLogQuery("Application", PathType.LogName, "*")))
            {
                EventRecord ev;
                while ((ev = reader.ReadEvent()) != null)
                {
                    parsedEvents.Add(new KeyValuePair<string, EventRecord>(ev.FormatDescription(), ev));
                }
            }

            sw.Stop();
            Console.WriteLine($"Got {parsedEvents.Count} events with strings in {sw.Elapsed.TotalMilliseconds:N3}ms");
        }

        static void Main(string[] args)
        {
            ParseEvents();
            ParseEventsParallel();
        }
    }
}

Got 20322 events with strings in 19,320.047ms
Got 20323 events with strings in 5,327.064ms

This gives a decent speedup of a factor 4. I needed to use some tricks to get faster because for some strange reason the class ProviderMetadataCachedInformation is not thread safe and uses internally a lock(this) around the Format method which defeats paralell reading. The key trick is to open the event log in the conversion threads again and then read a bunch of events of the query there via the event bookmark Api. That way you can format the strings independently.

Update1

I have landed a change in .NET 5 which increases performance by a factor three up to 20. See https://github.com/dotnet/runtime/issues/34568. You can also copy the EventLogReader class from .NET Core and use this one instead which will give you the same speedup.

The full saga is described by my Blog Post: https://aloiskraus.wordpress.com/2020/07/20/ms-performance-hud-analyze-eventlog-reading-performance-in-realtime/

We discussed a bit about reading the existing logs in the comments, can access the Security-tagged logs by accessing:

 var eventLog = new EventLog("Security");
 for (int i = 0; i < eventLog.Entries.Count; i++)
 {
      Console.WriteLine($"{eventLog.Entries[i].Message}");
 }

This might not be the cleanest (performance-wise) way of doing it, but I doubt any other will be faster, as you yourself have already found out by trying out different techniques. A small edit duo to Alois post: EventLogReader is not faster out of the box than EventLog, especially when using the for-loop mechanism showed in the code block above, I think EventLog is faster -- it only accesses the entries inside the loop using their index, the Entries collection is just a reference, whereas while using the EventLogReader, it will perform a query first and loop through that result, which should be slower. As commented on Alois's post: if you don't need to use the query option, just use the EventLog variant. If you do need querying, use the EventLogReader as is can query on a lower level than you could while using EventLog (only LINQ queries, which is slower ofcourse than querying in while executing the look-up).

To prevent you from having this hassle again in the future, and because you said you are running a service, I'd use the EntryWritten event of the EventLog class:

    var eventLog = new EventLog("Security")
    {
        EnableRaisingEvents = true
    };
    eventLog.EntryWritten += EventLog_EntryWritten;

    // .. read existing logs or do other work ..

    private static void EventLog_EntryWritten(object sender, EntryWrittenEventArgs e)
    {
        Console.WriteLine($"received new entry: {e.Entry.Message}");
    }

Note that you must set the EnableRaisingEvents to true in order for the event to fire whenever a new entry is logged. It'll also be a good practice (also, performance-wise) to start a (for example) Task, so that the system won't lock itself while queuing up the calls to your event.

This approach works fine if you want to retrieve all newly created events, if you want to retrieve newly created events but use a query (filter) for these events, you can check out the EventLogWatcher class, but in your case, when there are no constraints, I'd just use the EntryWritten event because you don't need filters and for plain old simplicity.