Defensive Programming in Python: Part 1: Golden Rules for Logging

Published in

Python in Plain English

4 min readFeb 14, 2024

Introduction

Welcome to the first post in the series of Defensive Programming. In this series, I am going to cover how to write code in Python that helps you sleep well at night. Code that you can rely on to work in production and not fail you unexpectedly.

Python is a wonderful language — it gives you a lot of flexibility — but with flexibility comes a lot of responsibility.

So be responsible! Be defensive! Be awesome.

Golden Rules

Rule 1: Say No to Print Statements

Why? Using print statements for monitoring what's happening in your code are quick and dirty. It works for a quick look but falls short for ongoing debugging or when you need to understand the application's behavior over time.

How? Replace print statements with logging.

# instead of this
print("Starting the process...")


# do this 
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Starting the process...")

# or this 
from loguru import logger
logger.info("Simple info log")

Rule 2: Embrace Loguru

Loguru provides a user-friendly approach to logging, with minimal setup required. It automatically handles file rotation, serialization, and asynchronous logging, among other features.

With Loguru, logging can be as simple as this

# Log to console 
from loguru import logger
logger.info("Simple info log")

# log to file with log rotation 
from loguru import logger 
logger.add("file_1.log", rotation="500 MB")
logger.info("This will be logged in file")

The inbuilt logger of Python is too complex to use and adds friction for newcomers. With loguru it’s too simple.

Rule 3: Stick to the Built-in Logging Library When Necessary

If you are using a framework that already supports logging in its framework — for instance Django — then you don’t have to reinvent the wheel and forcefully use Loguru.

Use Loguru only when you’re not using a framework or when the framework that you’re using doesn’t provide good support for logging.

When you absolutely have to use the built-in library — ensure you use config files. Refer to this gist: https://gist.github.com/kingspp/9451566a5555fb022215ca2b7b802f19

Rule 4: Use Log Levels Appropriately

Why? Log levels help differentiate the severity and type of the logged information, making it easier to filter and search through logs.

How? Here’s an example demonstrating different log levels:

logger.debug("This is for tracing fine-grained information.")
logger.info("General info about program execution.")
logger.warning("Something unexpected happened, but it's not a big deal.")
logger.error("Something failed and needs attention.")
logger.critical("Serious error, indicating the program might be unable to continue running.")

Rule 5: Protect the ERROR Level

ERROR / CRITICAL level logs must indicate significant issues that could impact the application’s functionality or data integrity. These should always be addressed.

How? Only use the ERROR level for logging exceptions or significant issues:

try:
    risky_operation()
except Exception as e:
    logger.error(f"Risky operation failed: {e}")

Rule 6: Log Judiciously

Why? Over-logging can lead to bloated log files/systems and make it harder to find important information. Under-logging can miss critical insights into what the application is doing.

How? Log key events, errors, and state changes without logging every single action. Use log levels to filter importance.

Rule 7: Aggregate and Make Logs Searchable

Why? Centralizing logs from different sources (services, applications) makes it easier to correlate events and troubleshoot issues.

How? Use tools like ELK (Elasticsearch, Logstash, Kibana), Splunk, or Graylog to aggregate logs. Ensure logs are structured (e.g., JSON format) to make them easily searchable.

Rule 8: Use Trace IDs in Web Applications

Why? Trace IDs help track the flow of requests across services, which is invaluable in microservices architecture or complex systems.

How? For Django applications, the django-guid package can attach a unique ID to every request, making it easier to follow a request's path through logs.

Example with django-guid:

# settings.py
MIDDLEWARE = [
    ...
    'django_guid.middleware.GuidMiddleware',
    ...
]


# In your logging configuration
'formatters': {
    ...
    'detailed': {
        'format': '%(asctime)s %(levelname)s [%(id)s] %(message)s',
    },
    ...
},

For other frameworks, look for similar middleware or packages that can generate and attach a unique ID to each request.

Conclusion

In conclusion, effective logging is an art that balances between too much and too little, ensuring that every log entry serves a purpose. By adopting the golden rules we’ve discussed, you’ll transform your logging practices from a mere afterthought to a powerful tool in your development arsenal.

Remember, logging is not just about catching errors; it’s about creating a transparent, traceable, and understandable record of your application’s behavior and performance.

As we wrap up this discussion on logging in Python, I encourage you to revisit your current logging practices. Reflect on how you can integrate these golden rules into your projects. The effort you put into refining your logging strategy will pay dividends in the long run by saving you time during debugging, providing clearer insights into your application’s operation, and ultimately leading to higher-quality, more reliable software.