Review Perspectives on RBS Quality

[kokuyouwind]

As I mentioned at the recent RubyKaigi 2024, I am looking for ways to contribute to the RBS review and testing process using LLMs such as ChatGPT.
For example, we are thinking that the following things may be possible.

• Check the quality of the RBS included in the diff and suggest specific changes if there are still untyped issues.
• Compare with the README and source code of the original gem and suggest addition of missing classes/methods
• Refer to the README of the original gem and propose test cases for the missing use cases.

In considering prompts to accomplish these things, we would like to verbalize our perspective on the review.
@pocke write “I do NOT review the quality of the RBS” at #266. If you were to review the quality of the RBS, how would you If I were to review the quality of the RBS, can you tell me what perspective I should review?
For example, we could consider the following.

• Are there any untyped items left?
• Are there any missing definitions of classes or methods that could be used publicly?
• Is the concreteness of type annotations appropriate?
• Are appropriate interfaces extracted?
• Are the generics appropriate?

I think this discussion can be reflected in the guidelines for writing RBSs, and can also be used as a guideline for Gem Reviewers to review.

[pocke]

The following list is an example of reviews.

• Too narrow types
  • (String) -> String v.s. (String | Symbol) -> String
  • (String) -> String v.s. (String) { -> String } -> String`
• Too wide types
  • (Symbol) -> String v.s. (:foo | :bar) -> String
• boolish v.s. bool
  • Allow some methods of ActiveRecord::Relation to accept optional block arguments #168 (comment)

I usually check the Ruby source code to confirm the RBS definition is correct. So it will be improved if the LLM can use both of RBS and Ruby code.

[kokuyouwind]

Thanks for the example, I had overlooked bool and boolish.
I will try to see if a prompt that generates review results based on ruby code and differences in _src/lib works well, while incorporating the above perspectives into the instructions.