Getting application logging right is crucial for every software project. When done correctly it makes the maintainers work a lot simpler. There’s no worst feeling than getting an error report and searching for logs to try to understand what happened and not finding a single entry. This is especially bad with difficult to reproduce production errors.
Like everything else, having good logs requires a little bit of thought and consistency. But is not that hard.
I’m not an expert on this subject. This post contains my notes on what has worked for me in the past in the context of backend web applications.
Use a real logger
Every language has a very simple way to send strings to stdout. That’s not a “real” logger.
In Node you have console.log() and in Python print(). These work great for simple scripts and print debugging. On a real production application, you want to use a “real” logger.
For Node, I prefer using pino. It’s fast, has great support, and is very easy to use.
What to log
- Request/Response – You must log every request and response. For requests, I like to include all headers, except the
Authorizationheader. For responses, I always include the HTTP status code and the response time. Both request and response include a unique request ID to correlate them. - App Errors – every
catchin your code must include an error log describing what went wrong. Also whenever your app gets into a state that will result in an HTTP status code greater than 299 you should have some sort of application-level error log, describing why your request ended in an error. - Security events – Always log when a login fails, or when a user tries to access a page or an object is not supposed to. Always log password changes, email changes or any other security credentials change. Make sure you don’t log the actual password or sensitive information.
- Business events – This is optional but it’s a good idea. If your app is a shopping cart, create a log entry every time you close a sale. The idea here is that you might want to create alerts based on these log events. If you usually see 100 sales per hour and suddenly see zero, it might be a good idea to take a look and understand what it’s going on.
A good log line
Having an unsearchable pile of logs is not useful. All my logs are in JSON format. This is very useful if you use a service that allows you to query JSON logs. I use AWS CloudWatch logs which is not perfect, but it does a decent job of helping me find what I’m looking for if I created the logs correctly.
Every log line must have the following.
- Log level – Usually a number to indicate what kind of log it is. Info, error, and debug are the most common.
- Message – This is the most important part of the log entry. Try to make this explicit and easy to understand for your future self or your fellow team members. Try to make messages unique so they are easy to search on your codebase. Having multiple error logs with the same message makes things harder than they have to be.
- Group – All of my logs have a
groupfield that use to indicate what part of the application is producing the log entry. Most of the time this group is used to indicate which specific file is creating the log entry. This facilitates searching. - Context – Application error logs usually require some context that will help you determine what is the problem without logging all of your application state. It is very important to include the minimum amount of data possible on every log entry but enough to determine what is going on. For example if your input validation failed it might be a good idea to log what exactly went wrong. For example if you only allow names of a certain length, it might be a good idea to log the length of the submitted name that produced the error.
- Timestamp – Almost every logger adds a timestamp. Just make sure that the format used works for you and make sure you include the time zone or that your are very certain of what zone it’s being used.
Security concerns
If your system deals with PII or CHD, make sure you are not logging sensitive information. Pino and every decent logger have a feature to redact logs.
Also, make sure that you periodically check your logs for sensitive data. This is something not obvious and I’ve seen it go wrong multiple times.
Don’t log sensitive or secret information.
Final thoughts
I know logs are boring but they are an essential part of every app. A lot of thought is put into many aspects of software development but I rarely find people writing or discussing this subject. You should put some thought on your logging practices. You future self will thank you.
In summary:
- Having decent logs will greatly improve the maintenance of your software.
- Error logs are the most important kind of log.
- Always include context on error logs.
- Log every request and response and include status codes.
- Make sure you don’t log sensitive information.