Skip to content

Latest commit

 

History

History
87 lines (59 loc) · 5.25 KB

LOGGING.md

File metadata and controls

87 lines (59 loc) · 5.25 KB
Raw

Logging

Currently, this image supports two different approaches how to handle logs generated by different subsystems in this image:

  • Elastic Common Schema (ECS) - recommended way for collecting structured logs, currently disabled by default
  • Syslog - deprecated way collecting textual logs, currently enabled by default

It is also possible to enable sending PHP exceptions to Sentry.

HTTP headers

  • X-Request-Id - should contain unique value, value of this HTTP header is logged in Apache access logs, PHP-FPM logs, audit logs, ECS and Sentry exceptions, so you can use this value to correlate requests between logs. For better tracking between your infrastructure, it should be set by the first proxy in your environment.
  • X-Forwared-For - should contain IP address of originating user. It must be set by the first proxy in your environment.

Elastic Common Schema (ECS)

Elastic Common Schema is JSON schema for generating standardized structured logs from multiple sources. Because of its nature, it provides much more context and allows better correlating between multiple logs than standard textual log format. You can switch from standard logs to structured logs just by switching ECS_LOG_ENABLED env variable. Logs will be then collected by Vector.

Even if you don't use Elastic, as logs are just JSON and also contain original log message (in message or event.original field), you can easily convert them to any log format you prefer.

For collecting and processing logs, Vector is used. You can send logs to container standard output (enabled by default when ECS is enabled), save to a file or send them to another Vector instance. If you want to use another of one of 52 supported outputs, you have to put file into /etc/vector/ folder in required format.

ECS Datasets

event.dataset field can contain one of these values:

  • httpd.access - access logs from Apache
  • httpd.error - error logs from Apache
  • php-fpm.access - access logs from PHP-FPM
  • php-fpm.error - error logs from PHP-FPM
  • jobber.runs - periodic tasks status
  • supervisor.log - logs from process manager
  • system.logs - usually PHP error messages
  • application.logs - logs from MISP application

Debugging

  • For live preview of generated log by ECS, you can use misp_ecs_show.py command inside container.
  • To check if Vector runs properly, you can use vector top or supervisorctl tail vector stderr commands inside container.

File system log locations

  • /var/log/messages - all logs captured by rsyslog (if syslog is enabled, see rsyslog.conf for definition)
  • /var/log/httpd/ - Apache logs (if ecs is enabled, only access log is available)
  • /var/log/php-fpm/ - PHP-FPM logs
  • /var/www/MISP/app/tmp/logs/ - application logs (PHP)

Healthcheck

Health of container subsystems is periodically checked by misp_status.py app. In case of problems you can run this app from inside of container by su-exec apache misp_status.py to check if all subsystems are running properly.

Environment variables

ECS and Syslog can be enabled at the same time, but it is recommended to choose just one variant.

ECS (recommended)

  • ECS_LOG_ENABLED (optional, boolean, default false) - enable collecting logs by Vector in ECS format (recommended)
  • ECS_LOG_CONSOLE (optional, boolean, default true) - output logs to container stderr, can be viewed for example by docker logs command
  • ECS_LOG_CONSOLE_FORMAT (optional, string, default ecs) - format of console logs, can be ecs or text
  • ECS_LOG_FILE (optional, string) - log file location
  • ECS_LOG_FILE_FORMAT (optional, string, default ecs) - format of file logs, can be ecs or text
  • ECS_LOG_VECTOR_ADRESS (optional, string) - redirect logs in ECS format to another Vector source

Syslog (deprecated)

If enabled, rsyslog collects all logs from container (see rsyslog.conf) and save them to SYSLOG_FILE or optionally sends them to remote syslog server.

  • SYSLOG_ENABLED (optional, boolean, default true) - enable rsyslog demon and sending logs from MISP app to them
  • SYSLOG_TARGET (optional, string) - if defined, all logs from the container are forwarded to a defined syslog server. Should be hostname or IP address of the system that shall receive messages.
  • SYSLOG_PORT (optional, int, default 601)
  • SYSLOG_PROTOCOL (optional, string, default tcp)
  • SYSLOG_FILE (optional, string, default /var/log/messages) - path to file that will contain all logs collected by syslog
  • SYSLOG_FILE_FORMAT (optional, string, default text-traditional) - sets SYSLOG_FILE log file format, can be json, text or text-traditional

Sentry

Sentry is a tool for error tracking and support for this tool is integrated into this image. If configured, unhandled exceptions will be logged in Sentry.

  • SENTRY_DSN (optional, string) - Sentry DSN to catch exceptions
  • SENTRY_ENVIRONMENT (optional, string) - user defined string that can help you distinguish between multiple MISP instances