CLI evaluation

[lppedd]

Hi everyone!
In the last week I've spent a lot of time evaluating frameworks with CLI capabilities.
I'd like to summarize here my current understanding of the JVM ecosystem and why I did not chose Spring Shell.

• Which frameworks did I evaluate?

  • Micronaut (Picocli)
  • Quarkus (Picocli)
  • Spring Shell

• What was I looking for?

  • easy development workflow.
    I'm developing a command, I want to press a button in the IDE and debug it (with an active context) ❌
  • easy testing, with an out-of-the-box testing framework for the CLI ❌
  • small native size (in GraalVM terms obviously) ❌

  Bonus:

  • out-of-the-box interactive shell capability ✔️
  • multi-step command flow ✔️🆘

• What did I pick in the end?
  Quarkus

• Why?

  • built-in UPX compression step
  • faster native image start-up
  • cool command testing tools (@Launch annotation, LaunchResult parameter)

  Spring Shell related:

  • the command flow is awesome, but it's mostly broken on Windows. I found myself getting mad at the prompt when typing.
    Also, it doesn't work when starting the application through an IntelliJ IDEA (Ultimate) run configuration.
  • the command as method is a nice idea, but it's impractical with a lot of parameters, each with its own annotation spawning multiple lines
  • the documentation is still a bit a lacking.
    For Quarkus and Micronaut I can just navigate to the Picocli guide and that's it, as the framework layer is small.
    For Spring Shell I found myself looking at random blog posts or StackOverflow questions

All in all, you are doing an amazing job here, and I think I'll switch to Spring Shell in the future.
I'd just like to see more love for Windows (like, I inited a project, built the JAR straight away, and got an exception running it) and testing.

I'll keep this issue updated as I progress with my CLI project capabilities.

[jvalkeal]

Thanks a lot for a constructive feedback as it helps to make things better. I'm moving this into discussions as it's easier there to have further threaded "chats".

[jvalkeal]

Testing a shell app is important and indeed there's a big gap in this area. Good news is that we're working on it. Issue #489 is tracking it.

We currently test our sample app on linux/macos/win using gh actions based on e2e node based framework. This tests both native binary and fatjar. Node was chosen for this to have a proper pty support so that it's possible to simulate shell app runs on CI systems. It also uses xterm.js which emulates ansi escape sequences making it easy to see what's actually visible in a real shell.

We're also trying to make all these to work with junit tests where similar emulation is done on a jvm without use of Node. e2e/Node is really meant for testing final app binary.

[lppedd]

e2e tests are probably needed for your side, but for us, the actual users of Spring Shell, simple JUnit-based tests are the best option.
I've read someone suggesting to test the ShellComponents, as they are beans after all, but it should not be like that. The flow should be the same as in production, the test command string should go through the parser too.

[jvalkeal]

Windows is probably a most difficult OS to work with but based on my experiments it works relatively well with correct settings. I've used only CMD and PowerShell.

Most of a shell support are coming directly from a JLine library and they have done good job to support windows. You should create a new issues for trouble you're having with windows. Did you remember to use jna or jansi libs on windows as it's required to hook into native windows terminals?

[jvalkeal]

Looks like windows is missing some ansi characters, thus the weird one before "Path ok". It's common trick to hide cursor when these type of interactive "console UI's" are done(i.e. github cli works same) but looks like JLine for some reason doesn't send "hide cursor" sequence or something else is wrong. Looks like wrapping cmd in conemu breaks something.

I never saw things getting broken like that when I tried cmder.

Also spring-tips sample is based on old experimental spring-native.

[lppedd]

@jvalkeal

It's common trick to hide cursor

Speaking in UX terms I think it's not a great choice (and I've been working on 5250 terminal sessions a lot). No feedback on where the cursor is means one has to guess or go by memory. The question is, let's say it doesn't hide the cursor anymore, can it be moved in the right place, or will it be always at the beginning?

Also spring-tips sample is based on old experimental spring-native.

I think a starter project that's aligned with the latest changes would be pretty cool to have.
As of now Spring Initializr doesn't generate anything useful.

[jvalkeal]

Ah there was some fixes made in cli/cli#1085. We could try to follow how it works there as these components are kinda just functional clones how things work on go-libs. Maybe make it implementation setting if cursor should be visible and follow selected lines(to make possible screen readers happy).

Regarding that cursor not getting hidden, might be just missing capability settings in JLine. Conemu is much more mature now than when support for it were added to JLine.

[lppedd]

We could try to follow how it works there

Sounds good for me! I've had a look at how the flow is implemented and I've seen there is JLine stuff under the hood.
I'll check some JLine examples just to get a grip on how it works.

Regarding that cursor not getting hidden, might be just missing capability settings in JLine.

I'd say forget about this 😆 I don't care at all if it's visible as of now, since at least I know where I am.
I care more about getting it positioned where I'm actually typing my chars.

[jvalkeal]

I should add this to technical docs but a way spring-shell components and jline work together is:

• Rendering loop starts in

  spring-shell/spring-shell-core/src/main/java/org/springframework/shell/component/support/AbstractComponent.java

  Line 287 in a1bd979

  display.update(render(getThisContext(context)), 0);

• Component really just output what it want's to show
• Heavy lifting is done in JLine where Display concept can either user full screeen or area below command-line. JLine then setups and restores screen when component exists.

[jvalkeal]

Not really sure why you mentioned start-up time as Spring Shell as graal binary is fast as it starts/runs instantaneously. Image is a bit fat currently but I expect that to get better when graalvm and Spring AOT gets more mature.

[lppedd]

I've run some benchmarks on my machine.
The first start of a Spring Shell native executable is very slow, almost 2 seconds. 1 second for Micronaut, and a consistent 500ms for Quarkus.
Obviously this is machine-dependent, and I picked the best option for my case.

[jvalkeal]

This doesn't look right. With graal binary regardless of spring/micronaut/quarkus we're talking about milliseconds for execution instead of second(s). Interested if you can show some stats or real runs.

[jvalkeal]

Compiling sample with ./gradlew clean build nativeCompile and then running graal binary and fatjar:

$ time ./spring-shell-samples/build/native/nativeCompile/spring-shell-samples help
AVAILABLE COMMANDS
...

real	0m0.126s
user	0m0.068s
sys	0m0.064s

vs.

$ time $JAVA_HOME/bin/java -jar spring-shell-samples/build/libs/spring-shell-samples-3.0.0-SNAPSHOT.jar help
AVAILABLE COMMANDS


real	0m1.921s
user	0m6.981s
sys	0m0.287s

[lppedd]

@jvalkeal two subsequent runs. First one is straight after building with mvn clean package -Pnative.

[jvalkeal]

Very little we can do here as IntelliJ has known issues when trying to debug shell apps. For example in vscode debugging a shell app works without problems. Not sure if jetbrains have interest to make this part to work for a very limited audience.

https://youtrack.jetbrains.com/issue/IDEA-183619
https://youtrack.jetbrains.com/issue/IDEA-132822#comment=27-2604317

[lppedd]

Seems we're out of luck here.
I have some knowledge of developing the IJ platform and plugins, but my free time is very limited lately.

I'll check if there is way to forward the Run Configuration command to the integrated Terminal, instead of the Run/Debug console.

[jvalkeal]

I don't use idea so I've never looked it further, but I've seen this mentioned by other people. If I remember it correctly their debug session connects directly to jvm streams, thus totally bypassing any modern shell to understand capabilities. If you run any shell app in idea internal shell it works, but not with debug session.

They have pty and terminal emulation working with their own jediterm. That's the code I've been experimenting to integrate with testing framework getting proper terminal emulation work with unit tests. Idea has a foundation to make things work but I think it'd involve using their own jediterm with debug session I think. Don't know how much work that involves.

[lppedd]

I would also like to add a couple of perplexities.

1. non type-safe default values: why should a default value go through the parsing phase?
   The answer is obvious, as methods are used to describe the command entry point there is no real way to specify a default value which is not a constant in an annotation. This goes straight to my the command as method is a nice idea, but... initial consideration.
   And that's why field-level options/arguments might be a better alternative.
2. annotations fields: the @Shell* annotations are not using uniformed names.
   To specify a name used to refer to the definition @ShellMethod uses key, while @ShellOption uses value.
   To specify a description for the definition @ShellMethod uses value, while @ShellOption uses help.
   This is a bit confusing when you start experimenting.