Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 object key by Fluent Tag #1

Open
so0k opened this issue Mar 11, 2019 · 2 comments
Open

S3 object key by Fluent Tag #1

so0k opened this issue Mar 11, 2019 · 2 comments

Comments

@so0k
Copy link

so0k commented Mar 11, 2019

I modified code a bit to store objects by Fluent TAG (using fluentbit 1.0.4)

See README for fluentbit config:
https://github.com/so0k/go-s3-logsink#fluent-bit-config

changes in http-server:

  • Read FLUENT-TAG HTTP header from request:

https://github.com/so0k/go-s3-logsink/blob/073d7e1043da72da2ad1c7d81411418d57dca186/server/http_server.go#L47

  • Update function signatures to pass tag for Objectkey

https://github.com/so0k/go-s3-logsink/blob/073d7e1043da72da2ad1c7d81411418d57dca186/server/http_server.go#L86

  • I dropped UUID from object key and used tag instead

https://github.com/so0k/go-s3-logsink/blob/073d7e1043da72da2ad1c7d81411418d57dca186/server/http_server.go#L97

If you like, I do PR with the changes to your repo, unless you don't think this would be useful for you

Also, is there a concern with dropping the UUID, for the same container I do not believe flb ships logs more than once the same milli second and this should be safe, no?

@StevenACoffman
Copy link
Owner

This is great! Please make a PR! I've been meaning to revisit this and update things for the newer version of fluent-bit. I added you as a collaborator to make that easier. I'm not sure about the implications of dropping the UUID, but that might be fine. One of my primary goals was to facilitate easily taking the S3 files, sending them back into fluent-bit and have them be delivered properly.

This allows disaster recovery for log pipelines that ship logs to kafka and elasticsearch.

@so0k
Copy link
Author

so0k commented Mar 19, 2019

I'll upstream as soon as I have a chance, sorry it's taking a while

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants