Reference

Public API documentation.

Settings

class log_outgoing_requests.conf.LogOutgoingRequestsConf(**kwargs)

Settings for django-log-outgoing-requests.

To override a setting, prefix it with LOG_OUTGOING_REQUESTS_ in your own settings file.

CONTENT_TYPES = [ContentType(pattern='application/json', default_encoding='utf-8'), ContentType(pattern='application/soap+xml', default_encoding='utf-8'), ContentType(pattern='application/xml', default_encoding='utf-8'), ContentType(pattern='text/xml', default_encoding='iso-8859-1'), ContentType(pattern='text/*', default_encoding='utf-8')]

Allowlist of content types for which the body may be saved to the database.

Use log_outgoing_requests.datastructures.ContentType to configure this setting.

DB_SAVE = True

Whether request logs should be saved to the database.

This can be overridden at runtime via configuration in the admin.

DB_SAVE_BODY = True

Whether request/response bodies should be saved to the database.

This can be overridden at runtime via configuration in the admin.

EMIT_BODY = True

Whether request/response body may be emitted in the logs.

HANDLER_ON_ERROR: Callable[[Exception], None] | None = None

Callback function to invoke if an exception happens during the emit phase.

HANDLER_USE_QUEUE: bool = True

Set to False to write the log record in the main thread.

By default, django-log-outgoing-requests expects to spin up a background thread to write logs.

MAX_AGE = 1

The maximum age (in days) of request logs, after which they are deleted (via a Celery task, Django management command, or the like).

MAX_CONTENT_LENGTH = 524288

The maximum size of request/response bodies for saving to the database, in bytes.

If the body is larger than this treshold, the log record will still be saved to the database, but the body will be missing.

RESET_DB_SAVE_AFTER = 60

If the config has been updated, reset the database logging after the specified number of minutes.

To protect against unintended logging of potentially sensitive data after debugging, this helps in resetting the “save to DB” configuration option. It resets back to “use the default from settings”.

If the value is falsy (including zero), then no reset takes place at all.

Note

this requires Celery to be installed, an optional dependency.

Formatters

class log_outgoing_requests.formatters.HttpFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Display request and response (meta) details of python requests library log records.

Depending on the configuration, either only the metadata or metadata + body of requests and matching responses is emitted.

Metadata:

  • HTTP method

  • Request URL

  • Request headers (masking auth details)

  • Response status

  • Response reason

  • Response headers

Handlers

Provide robust and opinionated logging handlers.

Log handlers are responsible for sending the actual log records produced by loggers to their final destination.

django-log-outgoing-requests provides a handler that knows how to send log records from structlog to the database and save them into the log_outgoing_requests.models.OutgoingRequestsLog model.

Our handler implementations take care of packages/modules that rely on process-forking behaviour, notably:

  • uwsgi

  • celery workers with prefork pool

Much of this implementation was taken from django-timeline-logger.

class log_outgoing_requests.handlers.DatabaseOutgoingRequestsHandler(*, use_queue_mode: bool = False, buffer_size: int = 5, flush_interval: float = 3.0, **kwargs)

Save the log record to the database if conditions are met.

The handler checks if saving to the database is desired. If not, nothing happens. Next, request and response body are each checked if:

  • saving to database is desired

  • the content type is appropriate

  • the size of the body does not exceed the configured treshold

If any of the conditions don’t match, then the body is omitted.

close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

emit(record: LogRecord)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

class log_outgoing_requests.handlers.QueueHandler(queue)

Keep the raw log record and drop streaming responses.

The stdlib implementation by default formats the log record to a string and clears most attributes to make them pickleable.

filter(record: LogRecord | RequestLogRecord | ErrorRequestLogRecord) → bool | LogRecord

Prevent unconsumed response bodies from being passed to the actual handler.

The response body must be consumed in the main thread to avoid thread-safety issues when reading GZIP’ed responses in multiple threads, especially with chunked transfer encodings where there’s no content-length header available where we check the size of response.content to decide if we can save the body to the database or not.

prepare(record: LogRecord)

Prepare a record for queuing. The object returned by this method is enqueued.

The base implementation formats the record to merge the message and arguments, and removes unpickleable items from the record in-place. Specifically, it overwrites the record’s msg and message attributes with the merged message (obtained by calling the handler’s format method), and sets the args, exc_info and exc_text attributes to None.

You might want to override this method if you want to convert the record to a dict or JSON string, or send a modified copy of the record while leaving the original intact.

log_outgoing_requests.handlers.ensure_listener(*handlers: Handler, _defer: bool) → Queue

Ensure a listener thread is running for QueueHandler.

Creates a queue if it doesn’t exist yet, and starts the background thread to listen to the queue to actually process the log records.

We don’t bother with preventing a background thread in the main runserver process that reloads the code and restarts the server - we don’t expect any audit logs to be created there, and trying to detect these situations is too fragile compared to real uwsgi servers & management command situations. The idle background thread should not cause significant overhead.

Parameters:

_defer – Defer starting the background tread or not - by default on uwsgi we defer the startup and call te actual startup in te post fork hook.

log_outgoing_requests.handlers.outgoing_requests_handler_factory(*, buffer_size: int = 5, flush_interval: float = 3.0) → QueueHandler | DatabaseOutgoingRequestsHandler

Create a logging handler instance suitable for production or testing.

By default, a queue-based handler is configured so that the actual logging of outgoing requests can be offloaded to a worker thread. This improves performance by taking database queries out of the main thread, and improves log integrity because the log insertions run in a separate thread and associated database transaction.

However, in (unit) test setups, you want to force the logs to be written in the same test transaction so that at the end of the test, the log record creation is rolled back by the test transaction, otherwise you break test isolation substantially and/or slow down test suites considerably by forcing them to be django.test.TransactionTestCase.

The appropriate handler is selected based on the LOG_OUTGOING_REQUESTS_HANDLER_USE_QUEUE Django setting.

Note that you cannot change this setup at runtime in tests through django.test.override_settings(), as the logging config does not get reinitialized when settings change (as it should be!). You should configure CI/local test environments to apply conditional configurations.

Parameters:

uWSGI/Celery integration

When integrating the queue-based logging handler (via log_outgoing_requests.handlers.outgoing_requests_handler_factory()) in uWSGI and/or Celery environments, some additional care is required.

Celery workers

Celery workers that use the prefork (the default) concurrency model must take care to defer the background thread initialization by setting an environment variable to true:

export _LOG_OUTGOING_REQUESTS_LOGGER_DEFER_LISTENER=true

Failing to do so will lead to deadlocks.

Celery beat

Celery beat shouldn’t need the functionality of this library, but to be safe, the deferred initialization is automatically taken care of.

uWSGI

uWSGI also uses a forking process model with the same deadlock risks like the Celery workers. However, it’s automatically taken care of when a uWSGI environment is detected and the background thread initialization is deferred until after the worker processes have forked.