Logging in Python27 Jul 2024 / 22 minutes to read Elena Daehnhardt |
|
In this post, I cover everything from logging to configuring logging to output messages to different destinations.
I also included some examples of logging levels and how to log messages at different levels based on the severity of the issue.
I hope my post will help anyone understand how to use logging effectively in their Python programs. If you have any thoughts or suggestions, feel free to share them with me.
Introduction
Logging is essential for developers to track events, debug issues, and understand how their programs work. Python’s built-in logging module offers a flexible way to create log messages from Python programs.
Logging allows us to:
- Track the flow of your program
- Debug and diagnose issues
- Monitor applications in production
- Gain insights into user behaviour
Logging examples
Python’s logging module is simple and can be configured to suit different needs. Let’s start with a basic example.
Basic Logging Examples
We import Python’s built-in logging module with the ‘import logging` statement.
Next, the logging.basicConfig(level=logging.INFO) line configures the logging system to capture messages at the INFO level and higher. The logging.info('This is an informational message.') line logs an informational message, which will be displayed because the logging level is set to INFO.
import logging
# Configure the basic logger
logging.basicConfig(level=logging.INFO)
# Log a simple message
logging.info('This is an informational message.')
The output:
INFO:root:This is an informational message.
In this configuration, the format of the log messages includes the time, the name of the logger, the log level, and the actual log message. The \texttt{datefmt} parameter specifies the format of the date.
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
Configuring Logging
Logging can be configured to output messages to different destinations (console, files, etc.) and in various formats. The logging also supports different levels.
Logging Levels
Python’s logging module has several built-in levels:
- DEBUG: Detailed information, typically of interest only when diagnosing problems.
- INFO: Confirmation that things are working as expected.
- WARNING: An indication that something unexpected happened or indicative of some problem soon.
- ERROR: Due to a more severe problem, the software cannot perform some functions.
- CRITICAL: A grave error indicating that the program itself cannot continue running.
Each level is represented by an integer value:
DEBUG: 10INFO: 20WARNING: 30ERROR: 40CRITICAL: 50
Messages logged at the CRITICAL level are the most severe and indicate critical issues that require immediate attention.
A critical error might occur in situations such as:
- A failure in a critical system component that prevents the application from running.
- A security breach or data corruption event that requires immediate action.
-
An unhandled exception in a critical part of the application that causes it to crash.
- Here’s a specific example:
try:
# Simulate a critical operation that may fail
raise RuntimeError("A critical failure occurred!")
except RuntimeError as e:
logger.critical("Critical error: %s", e)
In this example, if the critical operation fails, a RuntimeError is raised, and the exception is logged at the CRITICAL level, indicating a serious issue that needs urgent attention.
Configuring Basic Logging to a File
I usually store logging output into text files, which I later review. This is useful for periodically running scripts such as cron jobs.
In the example below, the logging.basicConfig function configures the logging system to write log messages to a file named app.log. The file mode is set to ‘w’ (write), meaning it will overwrite the file if it already exists.
It also sets the logging level to DEBUG, ensuring all messages at the DEBUG level and above are logged. It specifies a log message format that includes the logger’s name, the log level, and the message itself.
The subsequent logging.debug, logging.info, and logging.warning lines log messages at different levels, which will all be written to the app.log file because the configured level is DEBUG, which is the lowest level and thus captures all higher-level messages.
import logging
# Configure logging to write to a file
logging.basicConfig(filename='app.log', filemode='w', level=logging.DEBUG, format='%(name)s - %(levelname)s - %(message)s')
logging.debug('This message will be logged to the file.')
logging.info('This is an informational message.')
logging.warning('This is a warning message.')
The output will appear in app.log:
root - DEBUG - This message will be logged to the file.
root - INFO - This is an informational message.
root - WARNING - This is a warning message.
Advanced Logging Configuration
You can define custom loggers, handlers, and formatters for more advanced use cases.
Creating Custom Loggers and Handlers
import logging
# Create a custom logger
logger = logging.getLogger('my_logger')
# Create handlers
c_handler = logging.StreamHandler()
f_handler = logging.FileHandler('file.log')
# Set level of handlers
c_handler.setLevel(logging.WARNING)
f_handler.setLevel(logging.ERROR)
# Create formatters and add them to handlers
c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
f_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
f_handler.setFormatter(f_format)
# Add handlers to the logger
logger.addHandler(c_handler)
logger.addHandler(f_handler)
# Log messages
logger.warning('This is a warning.')
logger.error('This is an error message.')
Output on Console:
my_logger - WARNING - This is a warning.
Output in file.log:
2024-06-25 10:00:00,000 - my_logger - ERROR - This is an error message.
Let’s explore this custom logging example in detail.
Create a Custom Logger
First, we create a custom logger named ‘my_logger’:
logger = logging.getLogger('my_logger')
Create Handlers
Secondly, we create two handlers:
- c_handler: A stream handler that sends log messages to the console (stdout).
- f_handler: A file handler that writes log messages to a file named ‘file.log’.
c_handler = logging.StreamHandler()
f_handler = logging.FileHandler('file.log')
Set Levels for Handlers
c_handler.setLevel(logging.WARNING)
f_handler.setLevel(logging.ERROR)
- Sets the logging level for each handler:
c_handleris set to log messages at the WARNING level and above.f_handleris set to log messages at the ERROR level and above.
Create Formatters and Add Them to Handlers
c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
f_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
f_handler.setFormatter(f_format)
- Creates formatters that define the format of the log messages:
c_format: Formatter for the console handler, including the logger’s name, log level, and message.f_format: Formatter for the file handler, including a timestamp, logger’s name, log level, and message.
- Assigns these formatters to their respective handlers.
Add Handlers to the Logger
logger.addHandler(c_handler)
logger.addHandler(f_handler)
- Adds the configured handlers to the custom logger
my_logger, enabling it to send log messages to both the console and the file.
Log Messages
logger.warning('This is a warning.')
logger.error('This is an error message.')
- Logs two messages:
- A warning message, which will be displayed on the console because its level is WARNING.
- An error message, which will be displayed on the console and written to the file because its level is ERROR.
Logging in Python Functions and Classes
Logging can be integrated into your functions and classes to provide more granular information about their behaviour.
Logging in Functions
Let’s explore logging in a function that divides two numbers. The try block tries to divide x by y. If y is zero, it raises a ZeroDivisionError, and the program logs an ERROR message saying, “Division by zero!”. If the division is successful (i.e., y is not zero), it logs an INFO message saying “Division successful!” and returns the result of the division.
import logging
logging.basicConfig(level=logging.DEBUG)
def divide(x, y):
try:
result = x / y
except ZeroDivisionError:
logging.error("Division by zero!")
else:
logging.info("Division successful!")
return result
divide(10, 2)
divide(10, 0)
When you call divide(10, 2), the function logs “Division successful!” because 10 divided by 2 is 5. When you call divide(10, 0), the function logs “Division by zero!” because dividing by zero is impossible.
Output:
INFO:root:Division successful!
ERROR:root:Division by zero!
The first line shows that the division of 10 by 2 was successful. The second line shows that dividing 10 by 0 resulted in an error because you cannot divide by zero.
Logging in Classes
Even though class usage is outside the topic of this post, you can read my previous post Python classes and pigeons, which introduces object-oriented programming concepts and class creation in detail.
However, the example is self-explanatory, let’s explore it in detail:
- Calculator class: Contains methods to add and subtract numbers and logs these operations.
- __init__ method: Sets up the logger for the class.
- add method: Adds two numbers and logs the operation.
- subtract method: Subtracts two numbers and logs the operation.
- Creating and using the class instance: Demonstrates how to use the
Calculatorclass to perform and log operations.
import logging
class Calculator:
def __init__(self):
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.DEBUG)
def add(self, a, b):
self.logger.debug(f"Adding {a} and {b}")
return a + b
def subtract(self, a, b):
self.logger.debug(f"Subtracting {b} from {a}")
return a - b
calc = Calculator()
print(calc.add(10, 5))
print(calc.subtract(10, 5))
The code above follows these steps:
- Define the Calculator Class
class Calculator:- This line defines a class named
Calculator. A class is a blueprint for creating objects containing data (attributes) and functions (methods).
- This line defines a class named
- Initialize the Calculator Object
def __init__(self): self.logger = logging.getLogger(__name__) self.logger.setLevel(logging.DEBUG)- This is the constructor method, called
__init__. It runs when you create a newCalculatorclass instance. self.logger = logging.getLogger(__name__)creates a logger object for this class. The logger will capture messages related to the class’s operations.self.logger.setLevel(logging.DEBUG)sets the logging level to DEBUG, meaning all messages at this level or higher will be captured.
- This is the constructor method, called
- Define the Add Method
def add(self, a, b): self.logger.debug(f"Adding {a} and {b}") return a + b- This method takes two numbers,
aandb, and returns their sum. self.logger.debug(f"Adding {a} and {b}")logs a debug message showing the numbers being added.
- This method takes two numbers,
- Define the Subtract Method
def subtract(self, a, b): self.logger.debug(f"Subtracting {b} from {a}") return a - b- This method takes two numbers,
aandb, and returns their difference. self.logger.debug(f"Subtracting {b} from {a}")logs a debug message showing the numbers involved in the subtraction.
- This method takes two numbers,
- Create an Instance of the Calculator and Use It
calc = Calculator() print(calc.add(10, 5)) print(calc.subtract(10, 5))calc = Calculator()creates a new instance of theCalculatorclass.print(calc.add(10, 5))calls theaddmethod with 10 and 5, prints the result (15), and logs the debug message “Adding 10 and 5”.print(calc.subtract(10, 5))calls thesubtractmethod with 10 and 5, prints the result (5), and logs the debug message “Subtracting 5 from 10”.
When we run the class methods to add and subtract numbers, logging returns the following:
calc.add(10, 5), it logs “Adding 10 and 5” and returns 15.calc.subtract(10, 5), it logs “Subtracting 5 from 10” and returns 5.
Module-Level Loggers
A module-level logger is a logger that is configured and used within a specific module in a Python application. It is typically named after the module in which it resides, making it easy to identify where log messages are coming from. Using module-level loggers helps to organize logging output, especially in larger applications with multiple modules, and allows for more granular control over logging configurations.
Benefits of Module-Level Loggers
- Clarity and Organization: Each module can have its own logger, making tracing the source of log messages easier.
- Granular Control: Different modules can have different logging levels and handlers, allowing for fine-tuned logging behaviour.
- Modularity: Loggers can be configured independently, promoting modularity and separation of concerns.
Example of a Module-Level Logger
Assume you have a Python project with two modules: module1.py and module2.py.
module1.py:
import logging
# Create a logger for this module
logger = logging.getLogger(__name__)
def function1():
logger.debug('This is a debug message from module1')
logger.info('This is an info message from module1')
logger.warning('This is a warning message from module1')
module2.py:
import logging
# Create a logger for this module
logger = logging.getLogger(__name__)
def function2():
logger.debug('This is a debug message from module2')
logger.info('This is an info message from module2')
logger.warning('This is a warning message from module2')
main.py:
import logging
import module1
import module2
# Configure the root logger
logging.basicConfig(level=logging.DEBUG, format='%(name)s - %(levelname)s - %(message)s')
# Call functions from the modules
module1.function1()
module2.function2()
Explanation:
- Create a Module-Level Logger:
logger = logging.getLogger(__name__)- Each module creates a logger named after the module (
__name__contains the module’s name).
- Each module creates a logger named after the module (
- Define Functions with Logging:
def function1(): logger.debug('This is a debug message from module1') logger.info('This is an info message from module1') logger.warning('This is a warning message from module1')- Functions within the module use the module-level logger to log messages.
- Configure the Root Logger in main.py:
logging.basicConfig(level=logging.DEBUG, format='%(name)s - %(levelname)s - %(message)s')- The root logger is configured to display DEBUG and higher-level messages with a specific format.
- Call Functions from Modules:
module1.function1() module2.function2()- Functions from each module are called generating log messages.
Output:
module1 - DEBUG - This is a debug message from module1
module1 - INFO - This is an info message from module1
module1 - WARNING - This is a warning message from module1
module2 - DEBUG - This is a debug message from module2
module2 - INFO - This is an info message from module2
module2 - WARNING - This is a warning message from module2
In this example, the log messages include the module name, making it easy to identify where each message originated. This organization is crucial for debugging and maintaining larger projects.
I suggest also reading the docs logging—Logging facility for Python for more details and great tutorials. However, this post provides everything you need to get started easily.
Conclusion
The logging module in Python is helpful for understanding and fixing code issues. It helps you see what’s happening in your application and makes debugging and maintenance easier.
Did you like this post? Please let me know if you have any comments or suggestions.
Python posts that might be interesting for youReferences
|
About Elena Elena, a PhD in Computer Science, simplifies AI concepts and helps you use machine learning.
|