[Feature Request] Internal and external validation

[ash2k]

Feature description:

Allow to have "external" and "internal" (example names) validation rules on fields/messages/etc. External proto message consumers would use the "external" rules and internal consumers - both "external" and "internal".

Problem it solves or use case:

There is clearly demand for using custom functions in CEL (e.g. bufbuild/protovalidate-go#73, bufbuild/protovalidate-go#118, bufbuild/protovalidate-go#119). Reasons:

• Greater flexibility. Not everything can be written in CEL.
• Greater speed. Some non-CEL implementation may use assembly for faster validation.
• Desire to avoid re-implementing pre-existing validating code in CEL. I'd use an off-the-shelf function from a library for [Feature Request] Validation for bytes #264 if I could inject it.
• Work around bugs ([Feature Request] Validation for bytes #264 - I'm stuck!).

Currently protovalidate is not meeting the demand for this functionality.

Proposed implementation or solution:

From the explanation here, it seems that the concern is that there will be proto messages that cannot be used outside of a project because they require some custom validation function. I think this is a valid concern.

What if we have a separate CEL-based validation option (repeated?) that allowed the user to use custom functions in it? The user would configure the validator with custom functions and a list of messages for which such custom ("internal") validation is enabled.

This would avoid the problem as all the existing and custom rules would not use any custom functions - they are good, safe for external consumption. They cannot have any custom functions now or later and will not break importing code (as today). At the same time for in-application validation users can utilize the new mechanism for their messages specifically.

Contribution:

I haven't looked at the code, but it should be quite straightforward. I'm willing to contribute this.

Examples or references:

Additional context:

Please think about the current situation. If I cannot use protovalidate to run validation on my message, I need to make this part of the app logic. This validation might be on multiple messages, potentially nested in other messages, all over codebase. So, naturally, I would want to use a "choking point", like gRPC interceptors, to ensure all the messages are validated. I'll have to re-implement a lot of what protovalidate does to walk those messages and validate the ones I need. It feels very wasteful - a lot of people will have to rebuild this.

Another approach I considered is to have a separate validator, compose multiple validators into an aggregate validator. Each does it's validation. The problem is the overhead of walking each message using reflection more than once.

Yet another approach is to allow user to inject functions with predefined signature that protovalidate would match the messages against. E.g. funct (m *MyProtoMessage) error (this is for Go). It would be called for messages, assignable to the type of the argument of the function. This is like an interceptor for protovalidate. The benefit of this approach is that there is nothing in the proto file, there is no CEL evaluation either. Just direct code that validates the message.

Can we come up with a mechanism in protovalidate that ticks all the desired boxes for both maintainers and consumers?

[nicksnyder]

Our position is pretty well summed up in bufbuild/protovalidate-go#73 (comment) but will repeat the important bits here.

The goal of protovalidate is for constraints to be portable across any implementation as the protos where the rules are defined might be run in any context. This means that the only supported custom CEL functions must be part of the protovalidate spec or defined as a custom CEL expression on the message/field itself.

Just like protobuf ensures as long as you have the schema you can deserialize a message written by any other program, we want the same to be true for validating whether that data is valid. This is not possible if we allow validation rules that can be verified in one environment but might not be able to be verified in another environment.

If you have rules that can't be expressed in CEL, and don't make sense to be a standard rule in all protovalidate implementations, then Protovalidate is not a solution for enforcing those rules and you will need to do that validation of those rules in a different way.

Unfortunately, your proposed solution does not avoid the problem -- it still allows unportable rules to be defined in protobuf files. It doesn't really help to separate portable rules from unportable rules. If two different implementations can validate the same message and get a different result (which can happen if one implementation can validate the nonportable rules and another implementation cannot), then we haven't achieved portable validation.

If there are standard rules you think we should add to all implementations, please file a new issue.

Sorry that you are still stuck on validating bytes fields. This is the issue to follow: #268 and/or contribute to if you wish.

I am going to close this issue because I don't see a portable way to achieve what you want within Protovalidate.