Add custom request to return LSP internal state (#3194)

### Motivation

Diagnosing concurrency issues in the LSP tends to be very difficult and without more detailed information about the state of the language server, it becomes extra challenging to understand what's going on.

Let's add a command to surface internal state information, which we can hook to a command in the extension. When users manage to reproduce the problem, they can invoke the command and provide richer details.

### Implementation

The idea is to return internal state information that may be relevant to diagnose a corrupt state or crash. For now, I included the state of the worker, backtrace, size of the incoming queue and all stored documents.

This should hopefully help us understand the following things:

1. Is the worker dead or just stuck?
2. If it's stuck, where is it stuck? Is the queue increasing causing the worker to get backlogged?
3. Why did we get stuck? Are the documents we stored in an out of sync state with the client?

### Automated Tests

Added a test.

lib/ruby_lsp/base_server.rb

@@ -90,7 +90,8 @@ def start
         # The following requests need to be executed in the main thread directly to avoid concurrency issues. Everything
         # else is pushed into the incoming queue
         case method
-        when "initialize", "initialized", "textDocument/didOpen", "textDocument/didClose", "textDocument/didChange"
+        when "initialize", "initialized", "textDocument/didOpen", "textDocument/didClose", "textDocument/didChange",
+          "rubyLsp/diagnoseState"
           process_message(message)
         when "shutdown"
           @global_state.synchronize do

lib/ruby_lsp/server.rb

@@ -108,6 +108,8 @@ def process_message(message)
         )
       when "rubyLsp/composeBundle"
         compose_bundle(message)
+      when "rubyLsp/diagnoseState"
+        diagnose_state(message)
       when "$/cancelRequest"
         @global_state.synchronize { @cancelled_requests << message[:params][:id] }
       when nil
@@ -1368,5 +1370,24 @@ def compose_bundle(message)
         end
       end
     end
+
+    # Returns internal state information for debugging purposes
+    sig { params(message: T::Hash[Symbol, T.untyped]).void }
+    def diagnose_state(message)
+      documents = {}
+      @store.each { |uri, document| documents[uri] = document.source }
+
+      send_message(
+        Result.new(
+          id: message[:id],
+          response: {
+            workerAlive: @worker.alive?,
+            backtrace: @worker.backtrace,
+            documents: documents,
+            incomingQueueSize: @incoming_queue.length,
+          },
+        ),
+      )
+    end
   end
 end

test/server_test.rb

@@ -1371,6 +1371,27 @@ class Foo
     end
   end
 
+  def test_diagnose_state
+    @server.process_message({
+      method: "textDocument/didOpen",
+      params: {
+        textDocument: {
+          uri: URI::Generic.from_path(path: "/foo.rb"),
+          text: "class Foo\nend",
+          version: 1,
+          languageId: "ruby",
+        },
+      },
+    })
+    @server.process_message({ id: 1, method: "rubyLsp/diagnoseState", params: {} })
+    result = find_message(RubyLsp::Result, id: 1)
+
+    assert(result.response[:workerAlive])
+    assert_equal({ "file:///foo.rb" => "class Foo\nend" }, result.response[:documents])
+    assert(result.response.key?(:backtrace))
+    assert_equal(0, result.response[:incomingQueueSize])
+  end
+
   private
 
   def with_uninstalled_rubocop(&block)