Skip to content

Latest commit

 

History

History
77 lines (48 loc) · 2.94 KB

README.md

File metadata and controls

77 lines (48 loc) · 2.94 KB
Raw

All Logs Data

A complete logs tracking service which works independendly of other services in the system. Supports features such as:

  • Log dump for specific days.
  • Realtime grepping of statements

The overall flow of the system is as follows.

Image

Real-Time analysis

This would enable specific log statements to be grepped and counted in real time (just like axiom). Before triggering a bot, the real-time analysis needs to be enabled or switched on, which would start monitoring incoming logs and start creating a dump of analyzed data in a file. This data would include the following information:

  • Overview data
  • Grepped statement count of all services

Image

The configuration of greppable statements would be in the following format:

services = [
    {
        'service_name': 'UCI API',
        'service_container': 'uci-api-service',
        'match_statements': {
            'BotServiceError': 'ERROR [BotService]',
            'DeviceAdded': 'Added device to registry',
            'ResolveUser': 'ServiceService::resolve: Resolving users'
        }
    },
    {
        'service_name': 'Broadcast Transformer',
        'service_container': 'docker-deploy-prod_broadcast-transformer_1',
        'match_statements': {
            'Kafka Push to outbound': 'Kafka::Send: Pushed to topic process-outbound',
            'Broadcast Processed': 'Notification processed by broadcast-transformer:',
        }
    },
]

Log Format Spec

This is the format that would be followed while creation of logs. This spec allows UCI-API to read data without communicating directly with the LogBase container.

Image

Endpoints

These endpoints work in correspondence with uci-apis service. It is not a dependency for LogBase but rather a REST API that retrieves the data generated by LogBase. Any other REST API implementation may be written as well on top of LogBase.

  • Fetch Service log GET /admin/monitoring/logs/[SERVICE_NAME]?lines=[NUMBER OF LINES]&date=dd_mm_yyyy
  • Trigger Real-Time data analysis POST /admin/monitoring/realTime/start
  • Fetch Real-Time data GET /admin/monitoring/realtime?date=ss_mi_hh_dd_mm_yyyy
  • Stop Real-Time data analysis POST /admin/monitoring/realtime/stop
  • Get available realtime data files GET /admin/monitoring/realtime/available
  • Fetch Overview data GET /admin/monitoring/overview?date=ss_mi_hh_dd_mm_yyyy
  • Get download link for log GET /admin/monitoring/logs/[SERVICE_NAME]/download/log?date=dd_mm_yyyy
  • Get download link for error logs GET /admin/monitoring/logs/[SERVICE_NAME]/download/error?date=dd_mm_yyyy