Nicholas Blumhardt

Trace sampling in SerilogTracing

Tue, 15 Oct 2024 00:12:00 +0000

Sampling in SerilogTracing is taking shape. If you haven’t heard of SerilogTracing, that’s not surprising - it’s a brand new project launched in January 2024. But, we’ve reached 250k downloads of the core package, and we’re still picking up steam. 😎

This post takes a look at the sampling functionality that’s arrived in SerilogTracing 2.2.

Trace sampling

There are a few things we can do to reduce the volume of structured log and trace data an application generates. We can reduce the granularity of events, reduce event size, and conditionally switch events on and off using leveling.

Sampling is another popular option: if enough data is being generated, collecting only some portion of it can still provide reasonable visibility. Because it’s externally-controllable using standardized HTTP headers, sampling is known for being especially effective in situations where the source applications can’t otherwise be modified.

The first thing to understand about sampling in SerilogTracing is that it’s trace sampling: a trace is a collection of spans, and sampling aims to record either all spans in a trace, or none of them.

Why not (also) sample spans?

First, turning individual spans on and off using sampling would cause operations executing the same sequence of steps to have traces with different shapes. When you’re troubleshooting using tracing, comparing the overall shape of two traces should reveal whether each follwed the same sequence of steps - and sampling individual spans prevents this.

Second, a span on its own isn’t as interesting as a span in the context of its parent and child spans. By sampling individual spans, recorded spans would often be the only recorded span in their containing trace, and at that point a lot of their value would have been lost.

Sampling in a monolith

If we first look at tracing within a monolith or single service, we can see how this plays out. I prefer self-contained code samples over realistic ones, so to illustrate the basic sampling API we’ll run this program:

using Serilog;
using SerilogTracing;
using SerilogTracing.Expressions;

Log.Logger = new LoggerConfiguration()
    // The custom formatter here is provided by SerilogTracing.Expressions.
    .WriteTo.Console(Formatters.CreateConsoleTextFormatter())
    .CreateLogger();

// Set up SerilogTracing with the default options.
using var _ = new ActivityListenerConfiguration()
    .TraceToSharedLogger();

for (var i = 0; i < 10000; ++i)
{
    // The `outer` activity is the root of each trace the program generates.
    using var outer = Log.Logger.StartActivity("Outer {i}", i);

    // The `inner` activity is a child of `outer`.
    using var inner = Log.Logger.StartActivity("Inner {i}", i);
    
    await Task.Delay(100);
}

The program produces this output:

> dotnet run --project example/Sampling/Sampling.csproj
[09:41:44 INF] ├ Inner 0 (102.677 ms)
[09:41:44 INF] └─ Outer 0 (127.013 ms)
[09:41:44 INF] ├ Inner 1 (101.166 ms)
[09:41:44 INF] └─ Outer 1 (101.259 ms)
[09:41:44 INF] ├ Inner 2 (100.795 ms)
[09:41:44 INF] └─ Outer 2 (100.879 ms)
[09:41:44 INF] ├ Inner 3 (101.062 ms)
[09:41:44 INF] └─ Outer 3 (101.169 ms)
^C

Note that when tracing to the console, spans appear in the order that they complete, which is the opposite of what you’d see in a tracing tool. It’s not an ideal presentation, but it’s easier to get multiple example traces into a compact format this way.

The program isn’t currently using sampling: all of the traces are recorded.

Now, let’s add Sample.OneTraceIn(7) to the ActivityListenerConfiguration:

using var _ = new ActivityListenerConfiguration()
    .Sample.OneTraceIn(7)
    .TraceToSharedLogger();

Running again:

> dotnet run --project example/Sampling/Sampling.csproj
[09:42:56 INF] ├ Inner 0 (102.621 ms)
[09:42:56 INF] └─ Outer 0 (125.868 ms)
[09:42:56 INF] ├ Inner 7 (100.233 ms)
[09:42:56 INF] └─ Outer 7 (100.703 ms)
[09:42:57 INF] ├ Inner 14 (101.172 ms)
[09:42:57 INF] └─ Outer 14 (101.462 ms)
^C

Now, every seventh trace is now recorded. The child inner activity is always recorded if its outer parent is — so we see one in seven complete traces.

That’s just about all there is to say about Sample.OneTraceIn(); it’s intentionally the simplest possible strategy, serving as an example to build from, and an entrypoint to exploring the SerilogTracing API.

Sampling across services

Sampling gets more interesting when multiple services are involved. The first service involved in an operation might apply a sampler such as Sample.OneTraceIn(), but when it calls out to other services, how do they know that any spans they generate will be part of a sampled trace?

Propagating sampling flags

Along with the trace and span ids of the calling span, the W3C traceparent HTTP header carries sampling information from client to server in its final two hexadecimal digits:

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00

In this example from the spec, the trailing -00 means that the parent is not being recorded.

The alternative -01, indicates that the parent has been chosen for recording:

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

This is a curious system because headers often come from untrusted sources. Should an application or service receiving traceparent respect the caller’s sampling decision?

Public and internet-facing endpoints

If the service is public or internet-facing then the answer is an obvious “no”: a malicious actor could use the -00 sampling flags to turn off recording of their nefarious activities. Or, an external/third-party caller that always records all traces could consume excess storage by propagating its sampling decision with -01.

By default, SerilogTracing’s ASP.NET Core integration assumes that the caller is untrusted, and sampling decisions by external callers are ignored.

SerilogTracing will pick up the trace and parent span ids from callers, however, so you’ll still see that they’re present unless you disable them entirely using IncomingTraceParent.Ignore, which is the cleanest option for public or internet-facing services:

using var _ = new ActivityListenerConfiguration()
    .Instrument.AspNetCoreRequests(opts => 
        // Ignore all tracing information provided by callers - start a new trace for
        // each request, and apply local sampling policies (if any).
        opts.IncomingTraceParent = IncomingTraceParent.Ignore)
    .TraceToSharedLogger();

Internal endpoints

Conversely, if you’re “inside the firewall” and want sampling decisions to propagate between your services (the whole point, really), then you’ll need to trust the sampling decision made in the incoming traceparent:

using var _ = new ActivityListenerConfiguration()
    .Instrument.AspNetCoreRequests(opts =>
        // Accept trace and span ids, and sampling decisions provided by callers,
        // overriding local sampling decisions.
        opts.IncomingTraceParent = IncomingTraceParent.Trust)
    .TraceToSharedLogger();

If you choose to trust the incoming traceparent sampling decision, this will override any decisions made locally by sampling policies like Sample.AllTraces(), for activities with remote parents.

Sampling algorithms

Out of the box, SerilogTracing provides Sample.AllTraces() (the default), Sample.OneTraceIn(), and Sample.Using().

The argument accepted by Sample.Using() is a .NET ActivityListener sampling delegate, which you can learn a bit more about in this brief write-up, and use to construct custom sampling algorithms of your own.

Many more pre-built algorithms are possible, and we’re hoping to collect examples of these to share more widely in the community: please open a ticket if you have one to request or contribute.

.NET's ActivityListener sampling API

Fri, 04 Oct 2024 04:20:00 +0000

Distributed tracing can generate a lot of data, and sampling is the most established method to keep data volumes manageable. In .NET, the System.Diagnostics.ActivityListener class exposes two properties to control sampling: Sample, and SampleUsingParentId.

How do these work?

If you just switched browser tabs to check the documentation, I know why you’re back. There’s barely any useful information to be found about these, either in the .NET documentation or elsewhere online. Even the original API design proposal is a dead end. The canonical ActivityListener example, copied everywhere, includes a Sample implementation that’s something like this:

var listener = new ActivityListener();
listener.Sample = (ref ActivityCreationOptions<ActivityContext> _) =>
                                        ActivitySamplingResult.AllData;
// ...

You see, you need to specify a Sample function when creating an activity listener. The default sampling decision is ActivitySamplingResult.None, and if all registered ActivityListeners return this value, no activities will be created at all.

If you want to use the sampling function to do something more sophisticated than simply capture all traces, it’s assumed you’ll plug in the OpenTelemetry SDK and use its sampling APIs to achieve this, and there really isn’t any guidance out there otherwise. Depending on your circumstances, the OpenTelemetry SDK might be the right tool for the job, but it’s still deeply unsatisfying to rely on a core .NET diagnostics API that’s practically undocumented.

This year I’ve spent some time bridging System.Diagnostics.Activity and Serilog, and in the process had to dig deeper into how ActivityListener sampling works. Here are my conclusions, wrapped up in a tiny but non-trivial sampler. I’m fully aware that some of my conclusions and assumptions might be wrong; if you’re kind enough to send corrections I’ll make sure this article is updated.

`IntervalSampler`

The sampler presented here is called IntervalSampler. Its source code lives in a SerilogTracing example project on GitHub.

static class IntervalSampler
{
    public static SampleActivity<ActivityContext> Create(ulong interval)
    {
        ArgumentOutOfRangeException.ThrowIfZero(interval);
        var next = interval - 1;
        
        return (ref ActivityCreationOptions<ActivityContext> options) =>
        {
            if (options.Parent != default)
            {
                return (options.Parent.TraceFlags & ActivityTraceFlags.Recorded) ==
                                                    ActivityTraceFlags.Recorded ?
                    ActivitySamplingResult.AllDataAndRecorded :
                    options.Parent.IsRemote ?
                        ActivitySamplingResult.PropagationData :
                        ActivitySamplingResult.None;
            }

            var n = Interlocked.Increment(ref next) % interval;
            return n == 0
                ? ActivitySamplingResult.AllDataAndRecorded
                : ActivitySamplingResult.PropagationData;
        };
    }
}

IntervalSampler aims to collect one in every N possible traces (the “interval”), selected using modulo arithmetic. A more robust sampler might introduce some randomness into this process to avoid skewing the sample when an application produces the same types of traces in a very regular sequence, but those kinds of details would obscure the parts of the sampler that are important for our current purposes.

The sampler creates a sampling function that is wired up like so:

var listener = new ActivityListener();
listener.Sample = IntervalSampler.Create(7);
// ...

`Sample` vs `SampleUsingParentId`

The first thing you’ll encounter when setting up a sampler are the apparent duplication of the sampling function between

ActivityListener.Sample, which describes the parent of the sampled activity using ActivityContext, and
ActivityListener.SampleUsingParentId, which describes the parent using string.

public SampleActivity<string>? SampleUsingParentId { get; set; }
public SampleActivity<ActivityContext>? Sample { get; set; }

Both APIs were added in .NET 5, so one isn’t an obsolete alternative to the other. When should each be used?

It turns out that SampleUsingParentId supports both W3C and Microsoft’s legacy “hierarchical” tracing schemes. If a listener has both SampleUsingParentId and Sample configured, then SampleUsingParentId will be used. Otherwise, if the activity is using the W3C tracing scheme, Sample will be used.

So this suggests SampleUsingParentId is the best, most general thing to implement? No, not really. Non-W3C tracing is on its way to extinction, and within SampleUsingParentId you can’t directly access the modern, fundamental properties describing the parent activity, such as its trace id, span id, or trace flags.

IntervalSampler supports the Sample delegate signature:

return (ref ActivityCreationOptions<ActivityContext> options) =>
{
    // ...
};

TL;DR: unless you’re writing code that has to work in a legacy tracing scheme, Sample is the way to go, and you can safely ignore SampleUsingParentId.

Sampling traces, vs sampling activities

The next thing to confront is the subtle difference between the purpose of the Sample API — to determine whether or not to create an Activity — and the reason that you’re interested in it, which is to determine whether the trace to which the Activity belongs should be recorded.

An Activity is just one single span within a hierarchical trace. Sampling generally aims to either create all of the spans in a trace, or none of them. Once a decision has been made for the Activity corresponding to the root span in a trace, then all of its child activities should be included in the sample, too.

That’s what the first condition in our sampling delegate is concerned with:

if (options.Parent != default)
{
    return (options.Parent.TraceFlags & ActivityTraceFlags.Recorded) == 
                                        ActivityTraceFlags.Recorded ?
        ActivitySamplingResult.AllDataAndRecorded :
        options.Parent.IsRemote ?
            ActivitySamplingResult.PropagationData :
            ActivitySamplingResult.None;
}

If the activity we’re being asked to make a decision about would become the child of an existing activity, then we use the sampling decision already made for that activity.

If the parent activity is recorded (included in the sample), then the child is too, and ActivitySamplingResult.AllDataAndRecorded is the correct result.

Take care: the very similarly-named ActivitySamplingResult.AllData causes an Activity to be created, but it doesn’t mark the trace as being recorded. If you return ActivitySamplingResult.AllData from your sampler, activities likely won’t show up in your tracing system, and the sampling decision won’t be propagated downstream to other services and systems you call.

In the case that the parent isn’t included in the sample, we return ActivitySamplingResult.PropagationData to ensure a local activity is still created when the parent is remote, and otherwise return ActivitySamplingResult.None to save allocation of a new Activity instance.

At the root of the trace

The next, and final part of IntervalSampler, is concerned with root activities. These don’t have a parent, so when we make a sampling decision for them, we’re really making a decision about the whole trace: this Activity, and its (potential) future children.

var n = Interlocked.Increment(ref next) % interval;
return n == 0
    ? ActivitySamplingResult.AllDataAndRecorded
    : ActivitySamplingResult.PropagationData;

That’s why, when an activity isn’t included in the sample, we return ActivitySamplingResult.PropagationData instead of ActivitySamplingResult.None. If we returned ActivitySamplingResult.None, no activity would be created, and so later on we’d have no way to remember our decision when looking at more deeply-nested activities. The ActivitySamplingResult.PropagationData option does cause creation of an activity, but it’ll be marked in such a way that only minimal processing is performed on it, and it will ultimately be discarded.

So there you have it

Hopefully the information here helps you to skip some of the digging I’ve had to do, and sheds some light on what ActivityListener.Sample is all about. Corrections and errata welcome - and if you spot other examples or documentation surrounding ActivityListener.Sample that I’ve missed, I’d love to know about those, too.

Happy tracing! 👋

2024-10-05: added the IsRemote check when sampling by parent, to ensure an activity is always created for propagation purposes.

Visualizing the Serilog 4.1 batch retry algorithm

Thu, 03 Oct 2024 20:22:00 +0000

This is a quick post exploring how Serilog 4.1 (currently in preview) schedules retries when logging fails.

Logs tucked away in a file on a remote server aren’t very useful, so sending log events across a network to a centralized collector, log server, or service is very common. But, networks are unreliable and services have outages, so logging systems need to support retries when a network request fails.

Serilog 4.1 updates the algorithm used for scheduling retries, including the addition of BatchingOptions.RetryTimeLimit to control how long a failed request batch will be retried before giving up.

Since the algorithm uses exponential back-off and some internal thresholds to decide when a failed batch should be tried again, you might like to whip up a small test project to visualize this behavior under different conditions, with different options set.

To save you from needing to write this yourself, here it is:

using System.Diagnostics;
using Serilog;
using Serilog.Configuration;
using Serilog.Core;
using Serilog.Events;

var clock = Stopwatch.StartNew();

Log.Logger = new LoggerConfiguration()
    .WriteTo.FallbackChain(
        wt => wt.Sink(
            new NetworkSink(clock),
            new BatchingOptions
            {
                RetryTimeLimit = TimeSpan.FromSeconds(30),
            }),
        wt => wt.Sink(new LocalSink(clock))
    )
    .CreateLogger();

for (var i = 0; i < 1000; ++i)
{
    Log.Information("Event {I}", i);
    await Task.Delay(TimeSpan.FromSeconds(3));
}

The program logs 1000 events at three second intervals. It uses a batched network sink and a local fallback sink, both included below, to simulate some failures and reveal what’s going on behind the scenes.

The network sink accepts the first batch, and then fails for the next four:

class NetworkSink(Stopwatch clock) : IBatchedLogEventSink
{
    int _i;

    public async Task EmitBatchAsync(IReadOnlyCollection<LogEvent> batch)
    {
        var succeed = _i is 0 or > 4; 
        var symbol = succeed ? "\ud83d\udfe2" : "\ud83d\uded1";
        Console.WriteLine($"[{clock.Elapsed} Network] {symbol} Batch {_i} ({batch.Count} event(s))");

        ++_i;

        await Task.Delay(TimeSpan.FromMilliseconds(500));

        if (!succeed)
            throw new InvalidOperationException();
    }
}

Whether the batches succeed or fail, it takes 500 ms to make an attempt.

The local sink just prints the events it receives to the terminal:

class LocalSink(Stopwatch clock) : ILogEventSink
{
    public void Emit(LogEvent logEvent)
    {
        Console.WriteLine($"[{clock.Elapsed}   Local] \ud83d\udfe2 {logEvent.RenderMessage()}");
    }
}

Here’s the program’s output:

[00:00:00.0241652 Network] 🟢 Batch 0 (1 event(s))
[00:00:04.5297888 Network] 🛑 Batch 1 (1 event(s))
[00:00:07.0362428 Network] 🛑 Batch 2 (2 event(s))
[00:00:17.5382555 Network] 🛑 Batch 3 (5 event(s))
[00:00:18.0405741   Local] 🟢 Event 1
[00:00:18.0451773   Local] 🟢 Event 2
[00:00:18.0452174   Local] 🟢 Event 3
[00:00:18.0452246   Local] 🟢 Event 4
[00:00:18.0452297   Local] 🟢 Event 5
[00:00:38.0465922 Network] 🛑 Batch 4 (7 event(s))
[00:00:38.5476455   Local] 🟢 Event 6
[00:00:38.5477721   Local] 🟢 Event 7
[00:00:38.5478050   Local] 🟢 Event 8
[00:00:38.5478325   Local] 🟢 Event 9
[00:00:38.5478585   Local] 🟢 Event 10
[00:00:38.5478879   Local] 🟢 Event 11
[00:00:38.5479167   Local] 🟢 Event 12
[00:01:18.5473823 Network] 🟢 Batch 5 (14 event(s))
[00:01:21.0489512 Network] 🟢 Batch 6 (1 event(s))
[00:01:25.5509366 Network] 🟢 Batch 7 (1 event(s))
[00:01:28.0519726 Network] 🟢 Batch 8 (1 event(s))

It’s much more interesting to watch this in real-time: dotnet new console, dotnet add package serilog --prerelease, and pasting everything above into Program.cs will get you running, if you’re curious!

Breaking down the output:

[00:00:00.0241652 Network] 🟢 Batch 0 (1 event(s))

The first batch succeeds as expected.

[00:00:04.5297888 Network] 🛑 Batch 1 (1 event(s))
[00:00:07.0362428 Network] 🛑 Batch 2 (2 event(s))
[00:00:17.5382555 Network] 🛑 Batch 3 (5 event(s))

The next three batches fail. We start with just one event, but by the time we’re trying the third failed batch, four more events have accumulated.

The spacing between these is interesting:

Batch 1 comes ~3 seconds after its predecessor,
Batch 2 comes ~3 seconds later,
Batch 3 comes ~10 seconds after that.

The retry algorithm makes a couple of attempts at fairly short intervals, and then begins to exponentially back off. Exponential back-off helps conserve network bandwidth, and if the receiver is overloaded or restarting, gives it some breathing space to get back up and running. The gap between Batch 3 and Batch 4 will be even greater, as we’ll see in a moment.

[00:00:18.0405741   Local] 🟢 Event 1
[00:00:18.0451773   Local] 🟢 Event 2
[00:00:18.0452174   Local] 🟢 Event 3
[00:00:18.0452246   Local] 🟢 Event 4
[00:00:18.0452297   Local] 🟢 Event 5

Next, the five events that were queued up in Batch 3 are written to the local, fallback sink. Why does the system give up at roughly 17 seconds from the first failure, and not at 30 seconds, as we configured using RetryTimeLimit?

The answer is that because of exponential back-off, the next batch isn’t scheduled until around 37 seconds after the first failure, and this would exceed the time limit we’ve set, so the algorithm immediately triggers the failure handler and the batch is dropped.

Here’s that next retry attempt:

[00:00:38.0465922 Network] 🛑 Batch 4 (7 event(s))

The system has collected seven more events, the batch fails, and the events are immediately passed to the fallback sink:

[00:00:38.5476455   Local] 🟢 Event 6
[00:00:38.5477721   Local] 🟢 Event 7
[00:00:38.5478050   Local] 🟢 Event 8
[00:00:38.5478325   Local] 🟢 Event 9
[00:00:38.5478585   Local] 🟢 Event 10
[00:00:38.5478879   Local] 🟢 Event 11
[00:00:38.5479167   Local] 🟢 Event 12

Why aren’t these seven events retried in additional batches? After the retry delay has been reached, the algorithm switches modes, and all subsequent failing batches are immediately passed through to the fallback sink. This starts reducing the size of the internal buffer, while continuing to probe the remote service at reasonable intervals.

In this particular simulation, we’re now back online, so the next batch attempt succeeds and all subsequent ones do, too:

[00:01:18.5473823 Network] 🟢 Batch 5 (14 event(s))
[00:01:21.0489512 Network] 🟢 Batch 6 (1 event(s))
[00:01:25.5509366 Network] 🟢 Batch 7 (1 event(s))
[00:01:28.0519726 Network] 🟢 Batch 8 (1 event(s))

If we’d kept the network sink offline for longer, and simulated a higher logging rate or lower maximum batch size, we’d see the final failure state of the algorithm, which is to both drop batches and drain the entire queue at each subsequent failure. Ultimately, Serilog’s first priority is to avoid destabilizing the host application, and it’s not a good idea to keep on buffering events forever.

serilog/serilog is the place to find BatchingSink and FallbackAwareBatchFailure, which implement the above behavior. Bug reports and suggestions are most welcome over there, or here below.

I hope this post turns out to be useful when you need to better understand or tune batch retries in the future. We’ll have Serilog 4.1 out and ready for general production use soon. 😎

Serilog fallback sinks

Wed, 02 Oct 2024 22:50:00 +0000

Currently in preview, Serilog 4.1 adds support for falling back to a second sink when the first sink fails. This scenario shows up in a few variations, usually revolving around a network-based log collector, and a local, persistent destination such as a rolling log file that’s used when the network-based collector is unavailable. Some low-volume scenarios use email logging or a Slack channel as the fallback.

Here’s what falling back to the File() sink looks like, when a network sink like Seq() is unavailable:

Log.Logger = new LoggerConfiguration()
    .WriteTo.FallbackChain(
        wt => wt.Seq("https://seq.example"),
        wt => wt.File("log.txt", rollingInterval: RollingInterval.Day)
    )
    .CreateLogger();

Super simple! Grab the 4.1-dev-* versions of Serilog now and give it a try.

Keep in mind that most network sinks will retry failed requests for some time period before reporting a failure, so it usually takes ten minutes or so for logs to show up in the fallback sink. The 4.1 release also includes some control over the maximum retry time to tighten this up, which I’ll post about here tomorrow.

Fallback chains in Serilog 4.1 are built on another new feature, ILoggingFailureListener.

public interface ILoggingFailureListener
{
    void OnLoggingFailed(
        object sender,
        LoggingFailureKind kind,
        string message,
        IReadOnlyCollection<LogEvent>? events,
        Exception? exception);
}

public enum LoggingFailureKind
{
    Temporary,
    Permanent,
    Final
}

Failure listeners can try to record events elsewhere, or just report failures through metrics or some other mechanism. The WriteTo.FallbackChain() method uses failure listeners to move failed events from one sink to another.

You can implement an ILoggingFailureListener yourself, and hook it up directly, using WriteTo.Fallible():

// Implements `ILoggingFailureListener`
var listener = new MyFailureListener();

Log.Logger = new LoggerConfiguration()
    .WriteTo.Fallible(
        wt => wt.Seq("https://seq.example"),
        listener
    )
    .CreateLogger();

Sinks that throw exceptions from ILogEventSink.Emit() or IBatchedLogEventSink.EmitBatchAsync() get failure listener support out of the box, while other kinds of sinks can opt in by implementing ISetLoggingFailureListener.

We’ll have a stable release of Serilog 4.1 out soon. In the meantime, please share any feedback you have on fallback sinks or failure listeners here, or via the Serilog issue tracker. Thanks!

Serilog and .NET 8.0 minimal APIs

Thu, 18 Apr 2024 03:50:00 +0000

ASP.NET Core keeps evolving, and it’s been a while since I wrote one of these, so read on below for my take on an optimal Serilog configuration for ASP.NET Core 8 and the “minimal API” application model, a.k.a. WebApplicationBuilder.

I’ve arranged this into three parts: configuring Serilog, hooking up Microsoft.Extensions.Logging.ILogger<T>, and recording web requests. Each part builds on the one before it.

Configuring Serilog

First, it’s important to realize that ASP.NET Core apps are “just” regular console apps. You can set up Serilog and use it without anything ASP.NET Core-specific at all. Doing this first helps to validate that you have the foundations working, before moving onto anything more involved.

Assuming you have a blank “minimal API” project (dotnet new web), your Program.cs file will look like this:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Install Serilog and its console sink from NuGet:

dotnet add package Serilog
dotnet add package Serilog.Sinks.Console

At the top of Program.cs, create a logging pipeline and assign it to the static Log.Logger property:

using Serilog;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateLogger();

Log.Information("Starting up");

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Running your program should now produce the “starting up” message from Serilog, then some default output from the framework.

There’s one more thing to do. If the application fails to start, we want to capture any thrown exceptions, and also make sure any buffered log events are flushed before the process exits. Adding some try/catch/finally blocks will sort this out:

using Serilog;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateLogger();

Log.Information("Starting up");

try
{
    var builder = WebApplication.CreateBuilder(args);
    var app = builder.Build();

    app.MapGet("/", () => "Hello World!");

    app.Run();
    return 0;
}
catch (Exception ex)
{
    Log.Fatal(ex, "Unhandled exception");
    return 1;
}
finally
{
    await Log.CloseAndFlushAsync();
}

At this point, you can use Serilog’s Log class to write your own structured log events. In the next section, we’ll route the framework’s logging through the same pipeline.

You might be tempted to add some more sinks now. Hold on for just a moment, there are some decisions to make, which we’ll discuss in the next section.

Hooking up ASP.NET Core and `ILogger<T>`

This turns out to be ultra-simple. Install Serilog.Extensions.Hosting:

dotnet add package Serilog.Extensions.Hosting

And call builder.Services.AddSerilog():

try
{
    var builder = WebApplication.CreateBuilder(args);
    builder.Services.AddSerilog(); // <-- add this
    var app = builder.Build();
    // ...

Now, running the application will produce markedly different, and I think much nicer, results:

This is the time to start thinking about where else you would like to send your logs. Chances are, if you want to record events somewhere other than STDOUT, you’ll need to configure a destination - perhaps setting a log file path, or log server URL.

I’ll go out on a limb here, and suggest avoiding appsettings.json. I don’t expect you’ll all agree with me, so we’ll examine two different approaches, and you can make up your mind what works best for your application.

I’ll use the Seq sink as an example, since I built the first versions of Seq, and it’s now what I work on day-to-day. The process is very similar for any other typical Serilog sink; if you have trouble with the one you choose, let me know in the comments 🙂.

dotnet add package Serilog.Sinks.Seq

Once you’ve installed your sink package, you’ll need to add that sink to the logging pipeline.

Option 1 is to add your sink in code, and configure it there. You can make up environment variable names for any settings that you need to modify at deployment time:

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.Seq( // <-- add these lines
        Environment.GetEnvironmentVariable("SEQ_URL") ?? "http://localhost:5341",
        apiKey: Environment.GetEnvironmentVariable("SEQ_API_KEY"))
    .CreateLogger();

To get a Seq container running on http://localhost:5341, use:

docker run --rm -it -e ACCEPT_EULA=y -p 5341:80 datalust/seq

Run the app, open http://localhost:5341 in a web browser, and you’ll see something resembling:

I like this approach to configuration because it’s obvious, the compiler checks that I’ve provided all of the required parameters, and anything that’s not modified at deployment time can be specified in strongly-typed C#. It also works reliably alongside single-file publishing.

Option 2 is to use Serilog.Settings.Configuration to load your sinks from appsettings.json. Skip this section if you’re happy configuring everything in code as above.

dotnet add package Serilog.Settings.Configuration

Configuration isn’t immediately available, which means giving up on capturing exceptions that occur early in start-up, unless you want to jump through some additional hoops. My advice is to keep things simple, stripping back your Program.cs to something like:

using Serilog;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSerilog(lc => lc
    .WriteTo.Console()
    .ReadFrom.Configuration(builder.Configuration));

await using var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Your appsettings.json file should look like this, with the "Seq" section replaced by your sink(s) of choice:

{
  "Serilog": {
    "Using": ["Serilog.Sinks.Seq"],
    "WriteTo": [
      {
        "Name":  "Seq", 
        "Args": {
          "serverUrl": "http://localhost:5341",
          "apiKey": "<API key here>"
        }
      }
    ]
  },
  "AllowedHosts": "*"
}

The "Args" names above are the parameter names of the WriteTo.Seq() method: Serilog’s configuration system is just a JSON-driven way of calling the same configuration methods we see in the C# version. Check out the Serilog.Extensions.Configuration project README for more information on the configuration syntax.

Running the application will produce similar output to what we saw the first time. If the app fails to start, you’ll get an exception message and stack trace dumped to STDERR by the framework itself.

JSON configuration can take a little more work to get right, is a bit more fragile, and needs a little more testing. It can still be very enjoyable to use if you have a scenario that requires it. The choice is in in your hands, but the remaining examples will be built on the configuration-as-code version.

Recording web requests

At this point, the application writes a handful of log events each time a web request is handled. These contain useful data, but they’re rather noisy, and the information we typically need about a web request is spread across multiple events. In this section, we look at how to collapse this information into a single, nicely-formatted event.

The classic way to do this is to install Serilog.AspNetCore, and add app.UseSerilogRequestLogging() to your web application’s startup code, right after builder.Build(). This is super simple and effective, but ASP.NET Core is moving in a different direction. If you’re setting things up from scratch, you’ll have a better experience in the long run by relying on ASP.NET Core’s built-in Activity support to record web requests.

If you’re not familiar with System.Diagnostics.Activity, for the purposes of this post it’s enough to know that an activity represents an operation with a beginning and an end. ASP.NET Core wraps an activity around each web request it handles. Because the activity is completed after all middleware runs, the activity carries a more accurate picture of the request’s duration and final status code than the UseSerilogRequestLogging() middleware can provide. As a bonus, activities participate in distributed tracing, so you can view requests in and out of your app in a hierarchy if your chosen sink supports it.

Step one is to install SerilogTracing, its console output formatting support, and ASP.NET Core integration:

dotnet add package SerilogTracing
dotnet add package SerilogTracing.Expressions
dotnet add package SerilogTracing.Instrumentation.AspNetCore

In Program.cs, add the trace-aware formatter to the console sink, and reduce the verbosity of some Microsoft.AspNetCore.* log sources:

Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Override("Microsoft.AspNetCore.Hosting", LogEventLevel.Warning)
    .MinimumLevel.Override("Microsoft.AspNetCore.Routing", LogEventLevel.Warning)
    .Enrich.WithProperty("Application", "Example")
    .WriteTo.Console(Formatters.CreateConsoleTextFormatter(theme: TemplateTheme.Code))
    .WriteTo.Seq(
        Environment.GetEnvironmentVariable("SEQ_URL") ?? "http://localhost:5341",
        apiKey: Environment.GetEnvironmentVariable("SEQ_API_KEY"))
    .CreateLogger();

I’ve used Enrich to add the Application property to all events, since this helps with tracking traced operations between systems. This is a purely optional addition, though.

And then, configure the listener that will write ASP.NET Core’s activities to the Serilog pipeline:

using var listener = new ActivityListenerConfiguration()
    .Instrument.AspNetCoreRequests()
    .TraceToSharedLogger();

Here’s the final, complete example:

using Serilog;
using Serilog.Events;
using SerilogTracing;
using SerilogTracing.Expressions;

Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Override("Microsoft.AspNetCore.Hosting", LogEventLevel.Warning)
    .MinimumLevel.Override("Microsoft.AspNetCore.Routing", LogEventLevel.Warning)
    .Enrich.WithProperty("Application", "Example")
    .WriteTo.Console(Formatters.CreateConsoleTextFormatter(theme: TemplateTheme.Code))
    .WriteTo.Seq(
        Environment.GetEnvironmentVariable("SEQ_URL") ?? "http://localhost:5341",
        apiKey: Environment.GetEnvironmentVariable("SEQ_API_KEY"))
    .CreateLogger();

using var listener = new ActivityListenerConfiguration()
    .Instrument.AspNetCoreRequests()
    .TraceToSharedLogger();

Log.Information("Starting up");

try
{
    var builder = WebApplication.CreateBuilder(args);
    builder.Services.AddSerilog();
    var app = builder.Build();

    app.MapGet("/", () => "Hello World!");

    await app.RunAsync();
    return 0;
}
catch (Exception ex)
{
    Log.Fatal(ex, "Unhandled exception");
    return 1;
}
finally
{
    await Log.CloseAndFlushAsync();
}

And the output at the terminal:

If you’re following along using Seq, you’ll see the same events there:

SerilogTracing works with all Serilog sinks, but at the time of writing, the Seq sink, SerilogTracing.Sinks.OpenTelemetry, and SerilogTracing.Sinks.Zipkin have been written or modified to support back-ends with hiearchical tracing features. In other sinks, spans will show up as regular log events.

Tip: if you’re using a sink without hierarchical tracing support, add Enrich.WithSpanTimingMilliseconds() to add a convenient Elapsed property to request completion events.

Where to next?

So this is how I’d do it; you might find a different approach that works for you! Let me know how you go, or say “hi”, on ~~Twitter~~X and Mastodon.

The serilog tag on StackOverflow is the place for help with Serilog and how it integrates with other tech.

Thanks for stopping by! 👋

From SerilogTimings to SerilogTracing

Thu, 07 Mar 2024 05:19:00 +0000

SerilogTimings is a handy little library that wraps blocks of code in “operations”:

using Serilog;
using SerilogTimings;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateLogger();
    
using (Operation.Time("Say hello to {Name}", Environment.UserName))
{
    Log.Information("Hello!");

    using (var op = Operation.Begin("Measure temperature"))
    {
        // Attached to the log event but not shown in text output
        op.Complete("Temperature", 35);
    }
}

When an operation completes, either explicitly or by being disposed, SerilogTimings writes an event with the duration of the operation through Serilog:

[13:53:20 INF] Hello!
[13:53:20 INF] Measure temperature completed in 0.6 ms
[13:53:20 INF] Say hello to nblumhardt completed in 26.5 ms

SerilogTimings is simple, transparent, and gets its job done with a minimum of fuss. It hasn’t changed significantly in many years, now. So why this post? Timed operations show up absoultely everywhere, and so tooling has evolved to make working with them much more efficient.

Operations might be nothing more than an event with a duration attached, but by adding a handful of other correlation ids behind the scenes, collections of timed operations in a hierarchy become traces, and a whole range of tooling now exists that can visualize and manipulate them.

A new Serilog extension called SerilogTracing updates the SerilogTimings API to generate traces that interoperate with more tools and .NET libraries.

Here’s the equivalent of the above example, using SerilogTracing instead:

using Serilog;
using SerilogTracing;
using SerilogTracing.Expressions;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console(Formatters.CreateConsoleTextFormatter())
    .CreateLogger();
    
using (Log.Logger.StartActivity("Say hello to {Name}", Environment.UserName))
{
    Log.Information("Hello!");

    using (var op = Log.Logger.StartActivity("Measure temperature"))
    {
        op.AddProperty("Temperature", 35);
    }
}

The output at the console is similar:

[14:43:49 INF] │ Hello!
[14:43:49 INF] ├ Measure temperature (0.487 ms)
[14:43:49 INF] └─ Say hello to nblumhardt (19.015 ms)can

But by writing to a tracing-enabled sink, the result can now also be visualized in Seq, Zipkin, and many other tools:

The image above uses WriteTo.Seq() and Seq 2024.1. Wider sink support is listed in the SerilogTracing README.

Along with trace visualization, SerilogTracing integrates with System.Diagnostics so that HTTP requests, database operations, and pretty much any instrumented .NET activity will be automatically traced.

API conversion

SerilogTracing’s ILogger.BeginOperation() and ILogger.TimeOperation() are both covered by SerilogTracing’s ILogger.StartActivity().

Unlike SerilogTimings, SerilogTracing doesn’t provide a static wrapper around the Serilog Log class, so Operation.Begin() and Operation.Time() map to the same ILogger.StartActivity() extension method, which needs to be called on Log.Logger directly.

The SerilogTracing API for completing operations is separate from adding properties: instead of passing properties to Complete(), you explicitly complete a SerilogTracing activity using Complete(level, exception), or add properties using AddProperty().

Formatting conversion

The example above uses Formatters.CreateConsoleTextFormatter() from the SerilogTracing.Expressions NuGet package to add timings to console output. If you want to customize how spans are output, or match SerilogTimings more closely, there’s a template in the SerilogTracing README that you can modify to achieve this.

The future

SerilogTimings is not dead, but it’s done. If you’re interested in improving how your application exposes timing information, SerilogTracing is the natural successor to SerilogTimings, and opens up a range of interesting new possibilities.

The case for an application-level tracing API in .NET

Thu, 22 Feb 2024 22:58:00 +0000

If you want to record a log event from your application in .NET, you can do that today without a lot of noise or ceremony using Microsoft.Extensions.Logging:

_log.LogInformation("Completing order {OrderId} for {Customer}", order.Id, order.Customer);

Admittedly it’s a long line, but from this statement, packed in alongside the timestamp, level, and whatever correlation identifiers are set behind the scenes, you get:

A human-friendly message — "Completing order 15 for nblumhardt"
A low-cardinality event type — "Completing order {OrderId} for {Customer}"
A fully-structured payload — {"OrderId": 15, "Customer": "nblumhardt"}

Packing all of this into a single statement is important: it’s easy for diagnostics to start obscuring the real application logic, otherwise.

If you want to record a trace span (timed operation) from your application code in .NET today, you’ll need to use System.Diagnostics:

using var complete = MyActivitySource.StartActivity("Complete order");
complete?.AddTag("OrderId", order.Id);
complete?.AddTag("Customer", order.Custome);

Three lines in this case, but also, while we’ve maintained the structured payload and low-cardinality/machine-readable event type ("Complete order"), there’s no pleasant human-readable description of the span that includes the vital details of this span itself. If you want that you’ll need to add at least one more line.

There’s also the distracting ? language machinery needed to handle null activities, or danger of forgetting about them in projects that don’t enable strict null checks.

But what do we actually get in return, that the logging API doesn’t provide? Just a duration and a parent span id, if there is one. The activity ends up producing output that isn’t all that different from a log event.

So far, the major consumers of System.Diagnostics.ActivitySource are building frameworks and libraries, and using the current API it’s possible to generate spans precisely and efficiently. Frameworks like ASP.NET Core have different needs from regular applications, and seeing a bit more plumbing exposed in those codebases isn’t a big problem.

Noise and complexity is a bigger concern in application code, though.

Couldn’t we combine the ideas from these APIs and have a tracing experience that’s the best of both worlds?

using var complete = MyActivitySource.StartActivity(
    "Complete order {OrderId} for {Customer}", order.Id, order.Customer);

This is the API that SerilogTracing provides:

using var complete = _log.StartActivity(
    "Complete order {OrderId} for {Customer}", order.Id, order.Customer);

In framework code it’s just sugar; but in application code — this is how I want to record traces! As SerilogTracing picks up steam ahead of its 1.0, it will be interesting to see whether someone can bring this UX back to ILogger<T> and/or ActivitySource, too.

SerilogTracing

Mon, 22 Jan 2024 21:58:00 +0000

TL:DR: Check out SerilogTracing, a simple, minimal extension for Serilog that integrates with System.Diagnostics.Activity to provide hierarchical, distributed traces and compatibility with the Serilog sink ecosystem.

Traces are amazing for analyzing performance and for describing complex operations that flow across multiple systems. Modern .NET has tracing support built-in, so it’s now surprisingly easy to generate and consume traces with all of the hierarchical, distributed goodness provided for free by the framework.

SerilogTracing is a new project that handles the mechanics of:

Creating beautiful, clean, fully-structured traces using the message template syntax that Serilog users will know and love, and
Getting traces out into tracing systems, log files, the terminal, and any other place a Serilog sink can reach.

And because it’s built on the same tracing APIs as the rest of .NET, it can hope to interoperate with whatever other tracing or diagnostic infrastructure you might wish to use, now or in the future.

This post is a summary of what you can expect from SerilogTracing, but the best examples to get started with are in the SerilogTracing README, which includes detailed instructions.

Creating beautiful traces

A trace is made up of one or more spans, which are generally represented using activities in .NET.

You wrap an activity around some meaninful piece of work using Serilog.ILogger.StartActivity() and a using statement:

using var activity = _log.StartActivity("Fulfill order {OrderId}", order.Id);
// ... some application logic ...

When the activity is disposed or Complete() is called, a span will be written through the logger.

The first thing you’ll notice is that StartActivity() names the span using a message template. Any property values captured by the message template will be attached to the activity, so in this case, the span will carry an OrderId property.

If you’ve used Serilog previously, this will be familiar. Message templates are a powerful way to pack a low-cardinality event type (the template itself), a high-cardinality formatted message, and multiple fully-structured properties, all into a single, clean, readable line of code.

Locally, you get nicely-presented terminal output:

And, you can choose to send any combination of the template, the formatted message, and the structured properties to your tracing system, depending on what works best for you there.

Getting traces out

To get started, any Serilog sink will do! Traces are just collections of spans, and SerilogTracing represents spans as log events. This means that you can start instrumenting your code with tracing while logging to nothing more complex than a text file or the terminal.

// A very minimal configuration ...
Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateLogger();

// ... is still enough to start tracing.
using var activity = Log.Logger.StartActivity(...);

But, there are two things you’ll need to do if you want to view your spans in a hierarchy, or participate in distributed tracing.

Consume external activities

The first is to subscribe to activities created by other .NET components. This is important because the code generating these activities is also responsible for distributed trace propagation, and if no subscribers are listening, they won’t bother propagating important identifiers such as the incoming trace and parent span ids.

using var _ = new ActivityListenerConfiguration().TraceToSharedLogger();

If you’re working on the server-side, check out the instructions for adding ASP.NET Core instrumentation here, too.

Install some tracing-enabled sinks

Spans are just LogEvents, but a handful of conventions make it possible to render them as traces if you have a suitable back-end available.

In particular, the SpanStartTimestamp and ParenSpanId properties are enough to drive hierarchical trace visualization. The included Zipkin sink handles spans only (it ignores other events), and manages this in just a few dozen lines of interesting code.

Right now, unless you’re using Seq, Zipkin, or a back-end that supports OTLP, chances are you’ll need to modify an existing Serilog sink to recognize these properties: if you do, please reach out, I’m interested to hear how you go, and keen to help where I can.

The pitch!

So to wrap this whole thing up, the pitch for SerilogTracing comes down to:

Keep your existing Serilog logging unchanged
Generate rich spans with a single line of code each
Subscribe to spans from other .NET components with a single line of code at startup
Participate in distributed tracing and traceparent propagation thanks to .NET ❤️
Send everything through Serilog’s enrichers, filters, and sinks
See spans as structured log events in sinks that don’t support tracing
See spans as traces in compatible sinks

To balance out this very rosy picture 😊 please remember that SerilogTracing is brand new, pre-1.0, and still under very active development.

And, it would be remiss of me not to mention that much of SerilogTracing has been designed and built by my colleagues Ashley and Liam at Datalust: hat tip to you both! 🎩

Instructions for getting started are in the README. I hope this is useful for you!

Edited 2024-02-23: updated the example to match the current type and method names.

Serilog project update, May 2023

Mon, 08 May 2023 02:45:00 +0000

👋 Howdy! I hope you’re enjoying the, frankly, pretty fantastic state of structured logging in .NET these days. If you’re using Serilog then I hope you’ve also been enjoying the long period of stability we’ve maintained on the 2.x release series (I don’t want to rewrite my application logging every eighteen months, either 🙂).

While we’ve been keeping Serilog fresh, it’s time to make some minor breaking changes, particularly in the target frameworks and library versions that the mainline Serilog packages will support. A few interesting things are coming in the next month or so, and so here’s a quick run through what to expect from the Serilog project going into mid-2023.

Serilog 3.0 progress

Serilog 3.0 is coming together and should be finalized soon. It drops support for several ancient .NET versions and some long-unsupported minor releases. The directly-targeted frameworks are now:

netstandard2.0 and netstandard2.1
net462, net47, and net471
net5.0, net6.0, and net7.0

This leaves .NET Framework versions prior to 4.6.2, and .NET Core versions prior to 2.0, on the 2.x series of Serilog releases. The vast majority of sinks will remain compatible with both Serilog 2.x and 3.x so please don’t feel unloved if you’re maintaining an application that can’t move forward for one reason or another.

3.0 also includes a large number of performance improvements, bug fixes, and full support for as much of modern .NET as we can reach. You can try it out now! Grab the latest pre-release version from NuGet with:

dotnet add package Serilog --prerelease

All feedback and bug reports — especially any compatibility problems! — are welcome over on the project issue tracker.

Updated versioning for Microsoft.Extensions.* dependent packages

The following packages are about to ship 7.0.0 versions, and will match both the version (major and minor) and target framwork support of their matching Microsoft.Extensions.* packages:

Serilog.Extensions.Logging (tracks Microsoft.Extensions.Logging)
Serilog.Extensions.Hosting (tracks Microsoft.Extensions.Hosting)
Serilog.AspNetCore (also tracks Microsoft.Extensions.Hosting, via its Serilog.Extensions.Hosting dependency)
Serilog.Settings.Configuration (tracks Microsoft.Extensions.Configuration)

This means that, while you can use Serilog.Extensions.* v7 packages on a wide range of .NET versions, they’ll always depend on the same (v7) underlying versions of the corresponding Microsoft.* libraries. When Microsoft.Extensions.* v8 packages ship, we’ll simultaneously release v8 packages, too.

All up, this should make using the Serilog platform extensions packages more predictable, and result in fewer dependency resolution problems or security scanner false-positives.

Serilog.Sinks.OpenTelemetry 1.0.0 RC

OpenTelemetry includes a standardized protocol for transmitting structured log events. Our aim for Serilog.Sinks.OpenTelemetry v1 is to provide first-class support for connecting Serilog to back-ends that might not have dedicated or well-maintained Serilog integrations today.

The 1.0.0-dev-* packages on NuGet have reached “release candidate” status; if you have applications instrumented with Serilog, and a collector that supports the OpenTelemetry Logs protocol, your feedback and questions over in the project repository would be most welcome.

Who is to thank for all this?

Only the tiniest slice of all this is my own work; listing everyone’s contributions will have to wait for the release notes, but I should call out that some very talented and generous programmers are behind this latest push forwards. If you have the time and inclination to get involved in the Serilog project, you can find repositories for all of these packages under https://github.com/serilog.

Hot-reload any Serilog sink

Wed, 01 Feb 2023 00:30:00 +0000

Hi! 👋

Chances are you’ve found your way here because Serilog.Settings.Configuration’s runtime reconfiguration logic only supports minimum levels, and not any other sink parameters. This stems from a strong preference for immutability (and hence clean concurrency) in the Serilog design, but the question remains: if I need to update a sink parameter at runtime, how can I do that?

Sinks and configuration

Serilog sinks often need to be configured with some information about where a log event should be sent. The Seq sink, for example, accepts a URL and optional API key for the target Seq server:

Log.Logger = new LoggerConfiguration()
    .WriteTo.Seq("https://seq.example.com")
    .CreateLogger();

Whether it’s a URL like the one above, a connection string, SMTP server, or some other detail, if it needs to change at runtime without reloading the app, this post has you covered.

Introducing Serilog.Sinks.Map

This great little package is normally used to route log events to different destinations based on event properties (custom ones, and typical ones like the level). To do that, Serilog.Sinks.Map tracks a collection of sinks, initializing, flushing, and disposing of them as required.

We’ll configure it to keep only a single Seq sink open, but replace it whenever the value of the server URL changes:

// dotnet add package serilog.sinks.map

Log.Logger = new LoggerConfiguration()
    .WriteTo.Map(
        _ => GetServerUrl(),
        (url, writeTo) => writeTo.Seq(url),
        sinkMapCountLimit: 1)
    .CreateLogger();

GetServerUrl() in this example is your code, a function string GetServerUrl() that returns the current address to configure the sink with. Whenever the return value of the function changes, Serilog.Sinks.Map will invoke the callback in the second argument to reconfigure the sink.

Via JSON configuration

If the configuration you need is coming from appsettings.json, you’ll need to do a little more work to hook into the change detection behavior implemented by the Microsoft.Extensions.Configuration system.

This example assumes that you’re using ASP.NET Core 7:

// In Program.cs

var builder = WebApplication.CreateBuilder(args);

// <snip>

var configVersion = 0L;

builder.Host.UseSerilog((host, lc) => lc
    .WriteTo.Map(
        _ => Interlocked.Read(ref configVersion),
        (_, writeTo) => writeTo
            .Logger(sub => sub.ReadFrom.Configuration(host.Configuration)),
        sinkMapCountLimit: 1));

var app = builder.Build();

ChangeToken.OnChange<object?>(
    () => app.Configuration.GetReloadToken(),
    _ => Interlocked.Increment(ref configVersion),
    null);

And remarkably, that’s enough to have your entire Serilog pipeline reloaded whenever a configuration change occurs. 🙌

Update: this does not work for MinimumLevel, because levels have already been processed by the time they reach the sink. So… almost your entire Serilog pipeline 😉.

Thanks to Brad Lindberg for raising this question recently on GitHub.