-
Notifications
You must be signed in to change notification settings - Fork 40
Common problems
It's common practice for library creators to be mindful of the expected behavior of their code. Well designed libraries are intuitive and introduces no surprises to its consumers.
Issues created in this repository indicate that there are some aspects of this sink that aren't intuitive. The intention with this page is to describe these unintuitive behaviors and why they exist.
Lets back up somewhat and start with Serilog's take on reliability.
Serilog takes the view that, all things considered, logging is a lower priority than other application code and should never avoidably impact the operation of a running application.
This means that Serilog itself, as well as its sinks, shouldn't throw exceptions if it can be avoided. Well, if the sink isn't behaving as expected, how should I debug and diagnose it? SelfLog to the rescue. All sinks log exceptions to SelfLog. Its a bit like the Russian nesting doll, a logger within a logger 😀
Read Debugging and Diagnostics and Reliability on Serilog wiki for more information.
Sending log events over the network is an asynchronous operation. Mainly because I/O is an extremely slow operation, but also since the sink is batching multiple log events into a single HTTP request.
Before terminating the console application, make sure to flush Serilog.
ILogger log = new LoggerConfiguration()
.WriteTo.GrafanaLoki("http://localhost:3100")
.CreateLogger();
log.Information("Hello!");
Log.CloseAndFlush();Read Lifecycle of Loggers on Serilog wiki for more information.
This could happens due to log rejection from Loki side or network troubles.
v9 delegates delivery to Serilog's native batching. On an unsuccessful POST - a Loki rejection or a transient network error - the batch is retried with exponential backoff up to retryTimeLimit (default 10 minutes), after which it is dropped so the pipeline can keep making progress.
A few situations are possible:
-
Rejection by Loki - for example a misconfiguration such as a label value exceeding Loki's allowed length, or out-of-order entries. The batch is retried until
retryTimeLimit, then dropped. The Loki configuration guide can help here. -
Transient failure / Loki unavailable - the batch is retried with backoff and delivered once Loki recovers, provided that happens within
retryTimeLimit. -
Queue overflow - events wait in a bounded in-memory queue (
queueLimit, default 50 000). If it fills faster than batches drain (for example during a long outage), the newest events are dropped to cap memory use.
In every case SelfLog is the best way to find the source of the problem - it reports the HTTP status code and response body returned by Loki.
Please check that you follow label best practices. Also a Configuration pages from documentation could be useful