🚀 The reasons behind v9 - a ground-up rewrite

[mishamyte]

Hello everyone! 💜

v9 of Serilog.Sinks.Grafana.Loki has landed, and it is the biggest change in the history of this sink - a ground-up rewrite. Releases like this deserve an explanation, so in the spirit of the v8 post, here are the reasons behind every major decision.

💡 Why a rewrite at all?

Over the years we collected a family of bugs that all pointed at the same root cause: the sink mutated LogEvent instances while preparing batches. Renamed properties leaked between retries (#103), properties promoted to labels corrupted message rendering (#138), the level property kept getting re-added when Loki was down (#273), and the whole thing was not thread-safe (#224). Each of these got a workaround, but workarounds for a structural problem only multiply.

At the same time, the sink carried its own delivery infrastructure - queue, timer, backoff schedule - written years before Serilog had native batching. Serilog 4.x now ships all of it, battle-tested, in the core package.

So v9 does the honest thing: the pipeline is immutable (labels are derived from a read-only view of the event; nothing is ever renamed or removed), and delivery is delegated to Serilog's native batching. The whole bug class above is gone by construction, not by patching.

💡 Reliability: what native batching buys you

• A bounded queue by default (50 000 events). The unbounded queue could take your app down with an OutOfMemoryException during a long Loki outage (#141) - one of the oldest open requests.
• Retry with exponential backoff. A failed batch is retried for up to retryTimeLimit (default 10 minutes) instead of being dropped on the first error (#305).
• Fully async emission - and no more deadlocks on dispose.
• Fail fast on misconfiguration: the uri (#256) and the tenant are now validated when you call CreateLogger, with a clear ArgumentException. The tenant follows Loki's documented tenant ID rules - which also fixes the v8 regex that rejected perfectly valid tenant IDs (#202).

💡 Performance: ~55× less allocation, ~4.5× faster

v8 built an intermediate object graph and rendered it to a JSON string for every batch. v9 streams JSON bytes in a single forward pass through a reused, pooled Utf8JsonWriter - no intermediate object graph, no intermediate strings, and a focused optimization pass on top of that.

On the common workload (10 000 templated events, end to end through the public API):

	v8 8.3.2	v9
Allocated	74.95 MB	1.36 MB (~55× less)
Time	79.1 ms	17.7 ms (~4.5× faster)

Exception-heavy logging improves too (~3.3× less allocation, ~2.2× faster) - the remaining cost is .NET stack-trace materialisation, which every sink pays. v8 also pushed its large intermediate strings onto the Gen2/LOH heap; v9 stays in Gen0 on the common path.

We also benchmarked against Serilog.Sinks.Loki.YetAnother - a well-regarded low-allocation Loki sink - through the identical harness, and v9 comes out ahead on both metrics. All numbers, methodology and the change-by-change story are public: Benchmarks · V9 Optimization Log · benchmarks/ (reproducible with one command, and CI now runs the comparison on every PR).

💡 The features you asked for

• Structured metadata (#255): propertiesAsStructuredMetadata attaches per-line, non-indexed key/value pairs - filterable like a label, with zero cardinality cost. The right home for RequestId, UserId and friends (Loki 3.0+). It is opt-in, so the default payload stays byte-identical and works with any Loki version.
• TraceId / SpanId (#233): Serilog 4.x carries the ambient Activity trace context on every event; traceIdMode / spanIdMode route them to the JSON body or - recommended - to structured metadata, ready for Grafana ↔ Tempo correlation.
• Bring-your-own HttpClient / HttpMessageHandler: the ILokiHttpClient hierarchy is gone. Gzip, retries, mTLS, bearer auth - all the things you previously needed a custom ILokiHttpClient for - are now standard DelegatingHandler concerns, with copy-paste examples in the wiki.
• Pluggable exception serialization (ILokiExceptionFormatter): scrub PII, drop stack traces in production, or reshape the output entirely.

💡 Yes, it is written in F# now

The internals are F# - immutable by default, which is exactly the property the old pipeline lacked. From the consumer side nothing changes: the public API is designed to be idiomatic C# (WriteTo.GrafanaLoki(...) with the same kind of parameters as before, full appsettings.json support). The only visible difference is FSharp.Core appearing as a transitive dependency. The performance numbers above should settle any doubts about overhead. 🙂

💡 Upgrading

v9 is a major release with deliberate breaking changes. Two pages cover everything:

• Upgrading from v8 to v9 - the practical checklist
• Breaking changes - the exhaustive list

The headline items: .NET 8+ and Serilog 4.3.1+ are required, batchPostingLimit is now batchSizeLimit, Fatal renders as fatal (was critical - update your dashboards), promoted properties stay in the log body, and custom HTTP clients move to DelegatingHandler.

---

As always - feedback, questions and issues are very welcome. If something behaves unexpectedly after the upgrade, enable SelfLog and open an issue; we appreciate it a lot. 🙏

Glory to Ukraine! 🇺🇦