Last week I had some fun putting together several interesting technologies to build a better “live streaming” story for Seq. The result is a new server-side API, combined with a friendly C# client, that together make it very easy to receive events matching a signal or filter in real-time.

Seq has a paging API that can be used to retrieve the latest events from a stream in batches, but this isn’t the friendliest thing to program against for integrations that want to perform analytics or trigger actions based on events from the stream. seq-tail.exe, which is a sample showing how to tail events ‘live’ from a Seq server, needed a few dozen lines of code to retrieve events in a loop this way from Seq 3.3.

With Seq 3.4 and the latest API client (3.4.0-dev-*), this is all it takes to tail events matching a filter:

var connection = new SeqConnection("http://localhost:5341");

var filter = "@Level = 'Error'";

using (var stream = await connection.Events.StreamAsync<JObject>(filter: filter))
using (stream.Select(jObject => LogEventReader.ReadFromJObject(jObject))
             .Subscribe(evt => Log.Write(evt)))
{
    await stream;
}

The complete example is in this Gist.

The new API Events.StreamAsync() returns a hot IObservable<T> over a WebSocket. The observable will keep producing events until either it’s disposed, or the server is shut down.

Seq streams the events in compact JSON format, which the Seq API client library will deserialize into JSON.NET JObjects for us. If we wanted to, we could work directly with the JSON objects and access structured log event data as properties.

Serilog.Formatting.Compact.Reader provides the handy LogEventReader class that turns these documents back into Serilog LogEvents. Having the events represented in Serilog’s object model means they can be passed back into a logging pipeline. The Log class used in the snippet above could be configured to send the events to any of Serilog’s many sinks; the seq-tail.exe example makes use of this to output events back through literate console:

seq-tail.exe Screenshot

I like how this turns out; I’m always a little apprehensive about basing an API on IObservable<T> (a bit much conceptual baggage at times), but in this case it seems to fit the scenario well. Tailing a particular filter is easy, and scaling up to more complex stream processing with Rx should be straightforward.

Throughput appears to be pretty good, though it’s easy for network performance and latency at the client-side to slow things down. To keep resource usage under control, Seq maintains a queue of events for each connected client that is capped at 10,000 items. If the client falls behind, it won’t receive all the events that are bound for it (sending an error event, or disconnecting the client, could be better alternative strategies). Since filtering occurs server-side, and most connected clients are likely to be interested in only a small slice of the event stream, I think it will be unusual to see the limit hit in practice.

In the short term, the streaming API will simplify some typical integration scenarios. In the future these building blocks might contribute to an out-of-process host for Seq Apps, which may be important on platforms without AppDomain sandboxing (i.e. .NET Core).

You can try out this API today by installing the latest 3.4 preview build from the Seq downloads page and the pre-release version of the Seq.Api NuGet package. The code is hot-off-the-press, so feedback, suggestions and bug reports are appreciated.