jira/apps/api/plane/middleware/logger.py

164 lines
5.4 KiB
Python
Raw Permalink Normal View History

# Copyright (c) 2023-present Plane Software, Inc. and contributors
# SPDX-License-Identifier: AGPL-3.0-only
# See the LICENSE file for details.
2025-04-11 12:29:19 +00:00
# Python imports
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
import hashlib
import hmac
2025-04-11 12:29:19 +00:00
import logging
import time
# Django imports
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
from django.conf import settings
2025-04-11 12:29:19 +00:00
from django.http import HttpRequest
# Third party imports
from rest_framework.request import Request
# Module imports
from plane.utils.ip_address import get_client_ip
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
from plane.utils.exception_logger import log_exception
from plane.bgtasks.logger_task import process_logs
2025-04-11 12:29:19 +00:00
api_logger = logging.getLogger("plane.api.request")
2025-04-11 12:29:19 +00:00
class RequestLoggerMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def _should_log_route(self, request: Request | HttpRequest) -> bool:
"""
Determines whether a route should be logged based on the request and status code.
"""
# Don't log health checks
if request.path == "/" and request.method == "GET":
return False
return True
def __call__(self, request):
# get the start time
start_time = time.time()
# Get the response
response = self.get_response(request)
# calculate the duration
duration = time.time() - start_time
# Check if logging is required
log_true = self._should_log_route(request=request)
# If logging is not required, return the response
if not log_true:
return response
user_id = (
request.user.id if getattr(request, "user") and getattr(request.user, "is_authenticated", False) else None
2025-04-11 12:29:19 +00:00
)
user_agent = request.META.get("HTTP_USER_AGENT", "")
# Log the request information
api_logger.info(
f"{request.method} {request.get_full_path()} {response.status_code}",
extra={
"path": request.path,
"method": request.method,
"status_code": response.status_code,
"duration_ms": int(duration * 1000),
"remote_addr": get_client_ip(request),
"user_agent": user_agent,
"user_id": user_id,
},
)
# return the response
return response
class APITokenLogMiddleware:
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
"""
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
Middleware to log External API requests to PostgreSQL.
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
request_body = request.body
response = self.get_response(request)
self.process_request(request, response, request_body)
return response
def _safe_decode_body(self, content):
"""
Safely decodes request/response body content, handling binary data.
Returns None if content is None, or a string representation of the content.
"""
# If the content is None, return None
if content is None:
return None
# If the content is an empty bytes object, return None
if content == b"":
return None
# Check if content is binary by looking for common binary file signatures
if content.startswith(b"\x89PNG") or content.startswith(b"\xff\xd8\xff") or content.startswith(b"%PDF"):
return "[Binary Content]"
try:
return content.decode("utf-8")
except UnicodeDecodeError:
return "[Could not decode content]"
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
# Headers whose values must never be persisted in plaintext logs
SENSITIVE_HEADERS = frozenset({"x-api-key", "authorization", "cookie"})
def _redacted_headers(self, request):
"""
Returns the request headers as a string with sensitive values redacted,
so that credentials such as the API key are never stored in plaintext.
"""
redacted = {
key: ("[REDACTED]" if key.lower() in self.SENSITIVE_HEADERS else value)
for key, value in request.headers.items()
}
return str(redacted)
def process_request(self, request, response, request_body):
api_key_header = "X-Api-Key"
api_key = request.headers.get(api_key_header)
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
# If the API key is not present, return
if not api_key:
return
try:
log_data = {
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
# Tokenize the (high-entropy) API key into a stable, non-reversible
# identifier so logs can be correlated to a token without ever
# persisting the raw key. A keyed HMAC is used rather than a bare
# hash so the digest cannot be precomputed from a known key value.
"token_identifier": hmac.new(
settings.SECRET_KEY.encode(), api_key.encode(), hashlib.sha256
).hexdigest(),
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
"path": request.path,
"method": request.method,
"query_params": request.META.get("QUERY_STRING", ""),
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
"headers": self._redacted_headers(request),
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
"body": self._safe_decode_body(request_body) if request_body else None,
"response_body": self._safe_decode_body(response.content) if response.content else None,
"response_code": response.status_code,
"ip_address": get_client_ip(request=request),
"user_agent": request.META.get("HTTP_USER_AGENT", None),
}
refactor: logging with retention + API token hardening (#9148) * fix: harden API token handling against rate-limit tampering and plaintext logging - Make `allowed_rate_limit` read-only on APITokenSerializer so users can no longer raise their own API token rate limit via PATCH (GHSA-xfgr-2x3f-g2cf). - Stop persisting API keys in plaintext in APITokenLogMiddleware: store a SHA-256 hash as the token identifier and redact sensitive request headers (X-Api-Key, Authorization, Cookie) before logging (GHSA-r5p8-cj3q-38cc). * refactor: remove MongoDB log sink and add per-log-type retention Logs are now written to and cleared from PostgreSQL only; MongoDB is no longer used as a log sink or archive. - Drop the MongoDB write/archival paths from the API request logger, the webhook log writer, and the cleanup tasks; Postgres is the sole sink. - Cleanup tasks now hard-delete expired rows in batches via `all_objects` (rows are removed immediately, not soft-deleted). - Add env-backed, per-log-type retention settings: API activity logs (API_ACTIVITY_LOG_RETENTION_DAYS, default 14), webhook logs (WEBHOOK_LOG_RETENTION_DAYS, default 14), email logs (EMAIL_LOG_RETENTION_DAYS, default 7). HARD_DELETE_AFTER_DAYS no longer drives any log cleanup. - Delete settings/mongo.py, remove MONGO_DB_* settings and the plane.mongo loggers, and drop the pymongo dependency. * chore: gitignore local advisories.md notes file * fix: use keyed HMAC-SHA256 for API token log identifier Address CodeQL "weak hashing of sensitive data" by hashing the API key with a SECRET_KEY-keyed HMAC instead of a bare SHA-256. The identifier is a non-reversible tokenization of a high-entropy key (not password storage); keying it also prevents precomputing the digest from a known key value. * chore: address review feedback on log cleanup and request logging - process_logs accepts extra kwargs so jobs enqueued by an older release (with a mongo_log arg) don't fail during a rolling deploy. - Log-cleanup batch delete failures are logged and skipped rather than aborting the run, so a single bad batch can't block the rest. - Extend logger middleware test to assert Authorization and Cookie headers are redacted; add a test that a failing cleanup batch is swallowed. * fix: fall back to default when a log retention env value is invalid Negative (or unparseable) retention values would compute a future cutoff and delete every log row. The retention settings now fall back to their defaults in that case via a shared `_retention_days` helper.
2026-05-27 10:30:05 +00:00
process_logs.delay(log_data=log_data)
[WEB-5575]feat: enhance APITokenLogMiddleware to support logging to MongoDB (#8241) * feat: enhance APITokenLogMiddleware to support logging to MongoDB - Added functionality to log external API requests to MongoDB, with a fallback to PostgreSQL if MongoDB is unavailable. - Implemented error handling for MongoDB connection and logging operations. - Introduced additional fields for MongoDB logs, including timestamps and user identifiers. - Refactored request logging logic to streamline the process and improve maintainability. * fix: improve MongoDB availability checks in APITokenLogMiddleware - Enhanced the logic for determining MongoDB availability by checking if the collection is not None. - Added a check for MongoDB configuration before attempting to retrieve the collection. - Updated error handling to ensure the middleware correctly reflects the state of MongoDB connectivity. * feat: implement logging functionality in logger_task for API activity - Added a new logger_task module to handle logging of API activity to MongoDB and PostgreSQL. - Introduced functions for safely decoding request/response bodies and processing logs based on MongoDB availability. - Refactored APITokenLogMiddleware to utilize the new logging functions, improving code organization and maintainability. * refactor: simplify MongoDB logging in logger_task and middleware - Removed direct dependency on MongoDB collection in log_to_mongo function, now retrieving it internally. - Updated process_logs to check MongoDB configuration before logging, enhancing error handling. - Cleaned up logger.py by removing unused imports related to MongoDB. * feat: add Celery task decorator to process_logs function in logger_task - Introduced the @shared_task decorator to the process_logs function, enabling asynchronous processing of log data. - Updated function signature to include a return type of None for clarity.
2025-12-09 19:31:10 +00:00
except Exception as e:
log_exception(e)
return None