Server-side event handling with Seq apps

Happy New Year! The festive season did put a bit of a dent in Seq progress, but here we are in 2014 and I think
we have the most interesting release yet to kick it off.

Structured log events are often nicer to work with than plain text, and Seq 0.7 takes full advantage of this to
provide a plug-in “app” model for processing events with C# code.

Events can be sent to an app manually via the Seq web UI, or, events that match a particular query or view can
be processed automatically as they arrive. This makes a huge range of interesting scenarios possible - here are some
that are implemented or being discussed:

• Sending the details of an event by email: we think that this is such a common use case that a simple version comes built-in

• Pushing events to a web service or API: errors might be posted to an issue tracker (or pager!), interesting data might be pushed to a dashboard

• Archiving or mirroring: as events come in they might be written off to text files or an alternative log archive

• Feeding BI tools: many businesses report off of relational databases; by writing event properties back to SQL Server, operational data can more easily be brought into the mix

• Looking for patterns: here things get fun, since apps are implemented in regular C#, tools like Rx or state machines can be applied in any way you please

So what does an app look like?

Sending email with the built-in SMTP app

Here’s one we prepared earlier

In this simple example our goal’s to motivate the Sales team with an email whenever a big cart is checked out.

1. Create a query to match events of interest

Early days - any cart with more than two items is worth celebrating:

Save the query with a good descriptive name; we’ll this one “Big Checkouts” :)

2. Settings -> Apps

Click Start new instance:

Each app can have a set of associated settings - in the case of the SMTP Email app these are pretty standard. Each running instance gets its own title and values for each setting.

Currently we’ll format the event as text containing the message and a list of properties. A more complete implementation accepting a template for the email body would be nice to do at some point (but given how easy it is to write your own apps, you might beat me to it!).

So that we can try out the app’s configuration before spamming everyone, we’ll leave the Manually send events to the app checkbox ticked; to start processing the results of the query as they arrive we can return here and untick the box to select a query.

3. Sending an event manually

Back at the query we set up in the events screen, hovering over the ID of a matched event (click the event first to expand) reveals the name of the app instance we just set up.

That’s it for the basics - our email should have been sent by now, or if not there will eventually be an error raised back to the event stream by the app. We’ll see how to pipe events from a query or view to an app when we build our own from the ground up.

Writing an app to run in Seq

Event handling code is fun, and there’s hardly enough of it! Seq makes it easy to write event handlers and upload them to run on the event stream.

This example is going to watch for timeouts errors - System.TimeouException and the like. From time to time these happen in healthy systems – connections are dropped, machines restarted, etc. An increase in the frequency of timeouts from a system is a good warning sign though that something is going south.

The app itself is going to be a little bit generic: if it sees more than a set number of events in a certain window, it will write a new event back to the stream. (Another app might be configured to pick this up and let someone know, e.g. by sending email.) We’ll use a query to choose timeouts specifically, and pipe these to the app.

1. Create a new class library

The app’s going to live in a regular .NET 4.5 assembly. If you’re feeling rigorous you might create a unit test project here too!

2. Install Seq.Apps from NuGet

PM> Install-Package Seq.Apps

3. Write the reactor class

Events come in, we tally them up in a sliding window, and if our threshold is met we write the warning. Simple to describe, should be simple to implement!

The class that will handle all of this follows the same basic pattern that all Seq apps use:

public class SlidingWindowReactor : Reactor, ISubscribeTo
{
    public void On(Event evt)
    {
    }
}

There’s a little bit of ceremony here – a base class, an interface and so-on, but nothing too scary.

The fun part is our sliding window check. First, where do we keep the tally? Simple – only a single instance of the reactor is used for the life of the app, so we can just keep this in a few private fields:

  int[] _buckets;
  int _currentBucket;
  int _eventsInWindowThreshold;
  …

Rather than clutter up this post with nitty-gritty I’ve posted the complete example in this Gist. You might find it worthwhile to go and skim through it before continuing on below.

The “resolution” of our check will be one second, and we’ll count the events observed each second in an array used as a circular buffer.

Reactors are single-threaded, so we don’t need any locks or other synchronisation.

When the first matching event arrives, the buffer is initialised (yes, an Init() method would be nice to support here).

  public void On(Event evt)
  {
    if (_buckets == null)
    {
        var window = App.GetSetting("WindowInSeconds");
        _buckets = new int[window];
        _eventsInWindowThreshold = App.GetSetting("EventsInWindowThreshold");
        _currentBucket = 0;
       …
    }

One thing to notice here is App.GetSetting() – just like the To address, Subject and so-on supported by the SMTP app, custom apps can be configured with their own settings.

The next thing we’ll do is check that the event (which might have come out-of-order) fits within our window, and slide the window if necessary:

    int eventBucket;
    if (!TrySlideWindow(evt, out eventBucket))
        return;

The implementation of TrySlideWindow() not so pretty – it is much nicer to view in long form so I’ll just show the method call here. (Fork the Gist if you can improve it ;).)

Next we increment the count for the second that the event being observed falls in:

    _buckets[eventBucket]++;

Is the sum greater than the threshold?

      if (_buckets.Sum() >= _eventsInWindowThreshold)
      {
        var message = App.GetSetting("ThresholdName");
        Log.Information("Threshold {ThresholdName} reached: {EventCount} " +
            "events observed within {WindowSize} sec.", _thresholdName, sum, _buckets.Length);
      }

The complete example is a bit more efficient, and suppresses the message for a set time after it has been triggered.

(The name Reactor is a literal description of ‘something that reacts’ by the way – not an implementation of the Reactor Pattern, which is entirely different.)

4. Upload the app

Hopefully the sample complies; when it does, pop over to Seq’s settings screen and Upload an app:

Seq.Apps.dll and Serilog.dll are already loaded into the container that runs the app, so only the single assembly containing the app itself needs to be selected in this screen.

Here at the moment there is a bit of manual effort that some future conventions should alleviate - the namespace-qualified name of the reactor type needs to be entered, as do each of the setting names supported by the app.

Once that’s done, the app will appear in the main Apps page, where instances can be started as we did in the first example.

5. Create a query to match timeouts

We’ll write and save a very simple query to find timeouts - any event that has an Exception property containing the word "timeout" will be matched.

6. Send the results of the query to a new app instance

This time when we start an instance of our new app, we’ll choose the query matching timeouts:

With our app chugging away in the background, we can query using the TimeoutName property that we set to see when the threshold was reached:

Hopefully this post has started to paint a picture of how apps fit in with Seq’s other features to make it an interesting platform not just for storing and searching events, but also for integrating them back into more sophisticated workflows.

If you build something fun I’d love to hear about it!