AI Observability Feedback Support

[MarkRx]

The ability to capture and rank user feedback is crucial to being able to fine tune model prompts.

I propose a feature to capture user feedback. At a high level users could respond to AI comments in some formatted way which would be feed into the agent in an abstracted out tool agnostic format. The PR agent would then pass the information off to a user configured backend tracing/monitoring tool. Different optional integrations could exist and be community driven as needed to handle the implementation of integrating with the tool.

I'm still in the process of researching AI monitoring and observability but it could roughly look like this:

@mybot feedback <score> <comment>

from foo import bar

> Consider removing this unused import

@mybot feedback 1 This suggestion is wrong. The import is still used.

MyObject a = null;
a.foo();

> This code will throw a NullPointerException

@mybot feedback 10 You saved me from a costly mistake!

I may have time to POC something like this in the coming weeks with a LangSmith implementation.