Using Serilog with MongoDb running in Docker-Compose
I’m working on a side project at the moment, it’s based on a microservice architecture. To understand what is going on inside my application logging will be extremely important.
For no particular reason other than I hadn’t used it before and that it is supposed to be good for structured logging I chose to use Serilog.
To store the logs I wanted to be able to access them at a later stage. I felt that a NoSQL database would be good for this because depending on what I’m logging there will be different requirements so the extra flexibility with no defined schemas compared to SQL will be good. I went with MongoDb because it is well supported, pretty fast and I want to get more experience with it.
To get this setup I needed to
- Add MongoDb to Docker-Compose
- Add a connection string for the MongoDb instance to the config files
- Install the Serilog NuGet packages for .Net core
- Setup the Mongo client
- Setup the logger in Program.cs
- Add the Serilog request logger to Startup.cs
Adding MongoDb to Docker-Compose
This was fairly straight forward and I found the information I needed in other blogs1.
I added the following configuration into the docker-compose file
mongo.db:
image: "mongo"
environment:
MONGO_INITDB_ROOT_USERNAME: "root"
MONGO_INITDB_ROOT_PASSWORD: "myBigDevPassword!"
volumes:
- ./mongo-volume:/data/db
ports:
- "27017:27017"
networks:
- backend
networks:
backend: {}
volumes:
mssql-server-linux-data:
mongo-volume:The things to note here is the volume, if you’re like me and don’t know a huge amount about Docker, this is useful because it gives the container somewhere to store records and data belonging to the database. This needs to be defined in the “volumes” section of the compose file and then it can be used in the volumes section of the Mongo image where you specify the location that the data will be stored2.
MongoDb Connection String
This is simple in the ConnectionString section of appsettings add the MongoDb connection string. It is the standard MongoDb format.
"ConnectionStrings": {
"LogConnection": "mongodb://root:[email protected]:27017"
},Note: The database isn’t specified at the end of the connection string. This will be explained later.
Install Serilog packages
The packages needed for Serilog in .Net Core are
- Serilog
- Serilog.AspNetCore
- Serilog.Sinks.MongoDb
Install them either with the CLI or your package manager or any other way you like.
Setup the Mongo client
This is where this blog post differs from others. I stumbled across a GitHub issue on the Serilog MongoDb Sink project3. From this I saw that you could create and pass in the MongoDb client instead of the Serilog sink doing it for you
When I was following the Serilog documentation and other blog posts about using Serilog with .Net Core 2/3 I was unable to connect to my MongoDb instance running in Docker. This was because I was using a root server user in the connection string and specifying the Database where the log collection was to be created. This user doesn’t have access to all databases running on the server using the docker image as specified at the start.
I tried numerous ways of getting this to work by using the connection string when creating the logger as it is in the documentation. The issue I was running into was that Serilog under the hood was using the connection string with the Mongo Client package to both authenticate the user and to create the collection and store the logs4.
The problem with this is that if you specify the database name in the connection string the MongoClient will try and authenticate the user against that database, but the user I created in the Mongo image is created in the admin database, where it doesn’t have access to the database that is specified in the connection string.
To resolve this I tried to create the database and add the user to it when the docker image was being created, but this didn’t work because docker won’t create a database or collection until you are doing an insert.
What worked was creating the MongoClient manually and then passing it to the Serilog MongoDb sink. By doing this I was able to authenticate the root user against the admin database by default. This is done by not specifying the Database in the connection string.
var connStr = Configuration.GetConnectionString("LogConnection");
var client = new MongoClient(connStr);
Then I got the Database which creates it if it doesn’t exist.
var db = client.GetDatabase("Logging");Setup the Serilog logger
The next step was to create and configure the logger.
_logger = new LoggerConfiguration()
.MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
.Enrich.FromLogContext()
.WriteTo.Debug()
.WriteTo.MongoDB(db,
LogEventLevel.Debug,
"BackendLog",
1,
TimeSpan.Zero)
.CreateLogger();This is standard to what you will see in most blogs except that for WriteTo.MongoDb I am passing in the parameters to the MongoDb sink instead of using the defaults. The reason for this is that I am passing in the database created from the Mongo client, this form of the constructor requires you to pass all the parameters for the sink.
This has an added advantage of that I can specify the batch size and time span that the logs are written to the database. The default is a batch size of 50, this means that the logs won’t be written to the database until there are 50 of them to write. Timespan is set to X by setting it to 0 it means that the logs will be written immediately5. For development and debugging I want to be able to see the logs immediately so setting the batch to 1 and the timespan to 0, I will have real-time logging.
Next in add the Logger to Log.Logger.
Log.Logger = _logger;Note: In this code example I have the batch size and time span hardcoded. In development, I want real-time logging, but this could cause performance issues in different environments where there is a higher load. My advice would be to have these parameters read from the appsettings file and passed into the MongoDb Sink.
Serilog does allow for configuration from the appsettings files but I haven’t tried to see if that will work because I wanted to try configure Serilog using code. If you want to try it it is documented in the Serilog docs and other blogs6–8.
The final step is to add Serilog into the WebHost pipeline. This is done by adding
.UseSerilog()I create the Serilog logger as a class variable so that I can use it in the Main and in CreateWebHostBuilder so that I can configure it once.
In the end my Program.cs looks like
public class Program
{
private static IConfiguration Configuration { get; } = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
.AddEnvironmentVariables()
.Build();
private static ILogger _logger;
public static void Main(string[] args)
{
var connStr = Configuration.GetConnectionString("LogConnection");
var client = new MongoClient(connStr);
var db = client.GetDatabase("Logging");
_logger = new LoggerConfiguration()
.MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
.Enrich.FromLogContext()
.WriteTo.Debug()
.WriteTo.MongoDB(db,
LogEventLevel.Debug,
"BackendLog",
1,
TimeSpan.Zero)
.CreateLogger();
Log.Logger = _logger;
Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));
try
{
Log.Information("Starting up");
CreateWebHostBuilder(args).Build().Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Application start-up failed");
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureServices(services => services.AddAutofac())
.UseStartup<Startup>()
.UseSerilog();
}Add the Serilog Request Logger
The last thing to do is to add the request logger in Startup.cs. In Configure add it to the pipeline using
app.UseSerilogRequestLogging();Where you place it in the pipeline affects what gets logged. Only features added after the request logging will get logged9.
Optional: Add the Serilog Debugger
To help debug your Serilog configuration, there is a debugger available. You can use this by adding SelfLog to the Startup constructor as follows.
public Startup(IConfiguration configuration)
{
Configuration = configuration;
Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));
}This will output any issues with Serilog to the debugger, this can be removed when you are happy that it is all set up and working.
References
- 1.Zgadzaj M. MongoDB container with Docker Compose. Change(b)log.dev. https://zgadzaj.com/development/docker/docker-compose/containers/mongodb.
- 2.Docs D. Compose file version 3 reference. Docker.com. https://docs.docker.com/compose/compose-file/#volumes.
- 3.Goetzmann T. MongoDb Sink with X.509 Cert (MongoDB with SSL) #12. GitHub Serilog/serilog-sinks-mongodb . https://github.com/serilog/serilog-sinks-mongodb/issues/12#issuecomment-245657074.
- 4.Singh D. Unable to connect to MongoDb (using authentication) using mongocsharpdriver 2.7.0. Stack Overflow. https://stackoverflow.com/questions/53756773/unable-to-connect-to-mongodb-using-authentication-using-mongocsharpdriver-2-7.
- 5.DL M. Serilog.Sinks.MongoDB not logging into local MongoDB. Stack overflow. https://stackoverflow.com/questions/53742353/serilog-sinks-mongodb-not-logging-into-local-mongodb.
- 6.Schroeder Daniel. Examples of setting up Serilog in .Net Core 3 Console apps and ASP .Net Core 3. Daniel Schroeder’s Programming Blog. https://blog.danskingdom.com/Examples-of-setting-up-Serilog-in-Console-apps-and-ASP-Net-Core-3/.
- 7.Blumhardt N. Setting up Serilog in ASP.NET Core 3. NBlumhardt.com. https://nblumhardt.com/2019/10/serilog-in-aspnetcore-3/.
- 8.Wiki S. Serilog Wiki. GitHub. https://github.com/serilog/serilog/wiki.
- 9.Lock A. Adding Serilog to the ASP.NET Core Generic Host . AndrewLock.net. https://andrewlock.net/adding-serilog-to-the-asp-net-core-generic-host/.