common question: There are two different templating features, one here and one in cloudwatch_logs plugin

[nikhilo]

This is regarding the cloudwatch_logs plugin.
We have been facing an issue where several ECS tasks are writing to the same log stream, and might be getting throttled by CloudWatch. I tried to set the log_stream_template to $(ecs_task_id) and $(uuid). But none of them seemed to work. I got following errors,

[2022/10/14 10:42:00] [ warn] [record accessor] translation failed, root key=ecs_task_id


[2022/10/15 02:12:37] [ warn] [record accessor] translation failed, root key=uuid

1. Is there a list of variables supported by the log_stream_template config ?
2. What is the best way to have a unique log stream name for each task ?

[ojacques]

@nikhilo - hopefully you found your way through. Like you, I was looking to implement additional log streams by using templates. I found out that the cloudwatch_logs documentation mentions log_stream_template. However, using $(ec2_instance_id) or other documented syntax for the previous plugin cloudwatch did not match the usage in cloudwatch_logs, which uses record-accessor.

Long story short, I got it to work with an OUTPUT like so:

[OUTPUT]
    Name cloudwatch_logs
    Match ${APPLICATION_NAME}-firelens*
    region ${AWS_REGION}
    log_group_name ${LOG_GROUP_BASE_NAME}/var/jenkins_home/stdout
    log_stream_name other
    log_stream_template $ec2_instance_id
    log_key log   # Only ship the log message, without log metadata
    auto_create_group false
    retry_limit 2

With ${XYZ} being environment variables.

Also found out that the variables I could use are (in the case of an ECS task running on EC2):

{
    "source": "stdout",
    "container_id": "---",
    "container_name": "---",
    "ec2_instance_id": "---",
    "ecs_cluster": "---",
    "ecs_task_arn": "---",
    "ecs_task_definition": "---"
}

Hope this helps.

[PettitWesley]

@nikhilo there are two templating features in two different plugins.

• cloudwatch_logs templating is documented here: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch#log-stream-and-group-name-templating-using-record_accessor-syntax

  • this one is a simple templating feature which just takes values in the log record and uses them to construct the names.

• cloudwatch templating is documented here: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names

  • this one supports the special ECS resource key names as well as templating based on values in the log record.

Each feature is different.

@ojacques Your example works because you are using the built-in metadata keys provided by FireLens:

An alternative to templating: using init image with env vars in the config

We have this new image build with an init script: https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md#how-init-process-works

It can set env vars with ECS Metadata. You can reference env vars in the config. This is an alternative to templating that is more efficient since templating has a CPU overhead.

For example:

[OUTPUT]
    Name cloudwatch_logs
    Match ${APPLICATION_NAME}-firelens*
    region ${AWS_REGION}
    log_group_name ${ECS_CLUSTER}
    log_stream_name ${ECS_FAMILY}.${ECS_TASK_ID}
    log_key log   # Only ship the log message, without log metadata
    auto_create_group true
    retry_limit 2

[nikhilo]

@PettitWesley The init image worked as expected. Thank you! 🎉

[PettitWesley]

Keeping this open as guidance/common question.

[Mattie112]

Possible I have the same issue (and I am also confused on where to report it). I did create an issue at aws/eks-charts#927 but possible here is the better place.

Small summary:

  set {
    name  = "cloudWatchLogs.logGroupTemplate"
    value = "/aws/eks/${local.cluster_name}/$kubernetes['labels']['app']"
  }

does not work (accessor error in the log)

but the following does work:

  set {
    name  = "cloudWatchLogs.logGroupTemplate"
    value = "/aws/eks/${local.cluster_name}/$kubernetes['namespace_name']"
  }

If anyone can point me in the right direction that would be great.

[PettitWesley]

I'm not sure what's going wrong @Mattie112, see my answer here: aws/eks-charts#927 (comment)