Logging of PCM internals for testing and operational debugging

[riverma]

Description

Logging, and the ability to view detailed logs on-demand, is essential for assessing the proper functioning of the PCM. I'd like to get a discussion started on a comprehensive logging solution for the PCM, ideally using a centralized logging service (e.g. AWS CloudWatch)

Infrastructure Needed

• CloudWatch
• Forwarders for EC2 nodes
• Forwarders for Docker instances
• Cleaners to clean up old logs

Log Types

• DEBUG - developer-level detailed logs (the most verbose possible)
• INFO - operator-level logs (for testing, operational validation)
• WARN - semi-production level logs (for understanding impacts / issues during operations)
• ERROR - production level logs (for operational / live usage)

Log Sources

• AWS Lambda logs
• Elasticsearch
• EC2 instances logs
  • serviced
  • supervisord
• Application logs
  • Tosca (Javascript)
  • Figaro (Javascript)
    ...