WIP: Use NIO.2 WatchService for JRuby

[floehopper]

This is a quick & hacky attempt at getting the basics of #462 working closely based on the WatchDir example mentioned in the documentation.

I found that I had to bump LISTEN_TESTS_DEFAULT_LAG up when running the acceptance specs on my development machine (MacOS v10.13.6):

$ uname -a
Darwin trooz 17.7.0 Darwin Kernel Version 17.7.0: Sun Jun  2 20:31:42 PDT 2019; root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64
$ ruby --version
jruby 9.2.7.0 (2.5.3) 2019-04-09 8a269e3 Java HotSpot(TM) 64-Bit Server VM 25.152-b16 on 1.8.0_152-b16 +jit [darwin-x86_64]

Although I didn't try to minimise the value of LISTEN_TESTS_DEFAULT_LAG, I was using ~5 seconds to get the acceptance specs to pass successfully. This doesn't seem very encouraging, but maybe I'm doing something wrong! I'm hoping that pushing this up will yield some better results from Travis CI.

To-do (not exhaustive)

• Thread safety
• Recursive watching of sub-directories...?
• Cope with multiple directories
• Use callback passed into #_configure as intended...?
• Use #_process_event as intended...?
• Error handling...?
• Check that Listen::Adapter::Jruby.usable? implementation is sensible
• Update Travis CI config if necessary...?

/cc @headius @bratish

[coveralls]

Coverage decreased (-97.9%) to 0.0% when pulling 27c157a on floehopper:use-eventing-api-in-jruby into dc1e798 on guard:master.

[floehopper]

Hmm. The acceptance specs don't seem to be passing on the JRuby builds on Travis CI. 😞

For some reason, I can only get them to pass if I set the OS to OSX.

[scrhartley]

Not a ruby dev, but:

• Should reset be called after this loop, not in it.
• Should reset be called regardless of the result of dir.nil?

[floehopper]

• Should reset be called after this loop, not in it.

You're quite right - I got lost in all the braces in the Java example. I've addressed this in a fix-up commit.

• Should reset be called regardless of the result of dir.nil?

It doesn't appear to be the case in the Java example, but what you say makes sense. I think in the Java example, they're probably assuming it can never happen, because every path that's registered will have an entry in the keys Map. The same thing is probably true of the current implementation, but I can see a case for doing as you suggest, so I've made the change in another fix-up commit.

[floehopper]

Hmm. I just read this at the end of the documentation:

Most file system implementations have native support for file change notification. The Watch Service API takes advantage of this support where available. However, when a file system does not support this mechanism, the Watch Service will poll the file system, waiting for events.

Maybe that's what I'm seeing on Mac OSX.

[headius]

Wow, cool that this got a quick impl so...quickly!

The lag times could be influenced by slow JRuby startup?

Also per-platform support...yes I think one thing we'll want to look into is whether there's a way to query whether it's going to use filesystem events or polling. In the cases where the JDK is using polling, we can explore other options like using FFI to call into MacOS inotify APIs.

I will have a look at this tonight or tomorrow. Thanks @floehopper!

[headius]

Ok so it looks like only BSD/MacOS do not have support for using a native filesystem eventing system. Windows uses ReadDirectoryAttributes, Linux uses inotify, Solaris uses something similar.

I think the simplest way to detect this would be to get the WatchService instance and then check if it's the polling one. Unfortunately there's no public API for this, but the following kinda-gross hack should work:

ws = ... get watch service
if ws.name =~ /Polling/
  # fall back on existing polling impl or try using FFI or whatever

[headius]

I found a MacOS FSE library for JDK that uses JNA, a library similar to the JNR we use to implement FFI. This code could probably be translated into Ruby FFI pretty easily:

https://github.com/gmethvin/directory-watcher/tree/master/core/src/main/java/io/methvin/watchservice/jna

[floehopper]

@headius:

The lag times could be influenced by slow JRuby startup?

I'm pretty sure this wasn't the issue I was seeing on my Mac OSX development machine. Running the following script and executing touch foo.txt from another terminal window generated the following output, suggesting that there's a 5-10 sec lag.

require 'java'
java_import 'java.nio.file.FileSystems'
java_import 'java.nio.file.Paths'
java_import 'java.nio.file.StandardWatchEventKinds'

require 'pathname'
require 'fileutils'

EVENT_KINDS = [
  StandardWatchEventKinds::ENTRY_CREATE,
  StandardWatchEventKinds::ENTRY_MODIFY,
  StandardWatchEventKinds::ENTRY_DELETE
]

directory = Pathname.pwd
watcher = FileSystems.getDefault.newWatchService
path = Paths.get(directory.to_s)
keys = {}
key = path.register(watcher, *EVENT_KINDS)
keys[key] = path

puts 'Monitoring started...'

loop do
  key = watcher.take
  dir = keys[key]
  unless dir.nil?
    key.pollEvents.each do |event|
      kind = event.kind
      next if kind == StandardWatchEventKinds::OVERFLOW
      name = event.context
      child = dir.resolve(name)
      lag = Time.now - File.mtime(child.to_s)
      p [child.to_s, lag]
    end
  end
  valid = key.reset
  unless valid
    keys.delete(key)
    break if keys.empty?
  end
end

Monitoring started...
["~/Code/floehopper/listen/foo.txt", 5.1431249999999995]
["~/Code/floehopper/listen/foo.txt", 5.853784]
["~/Code/floehopper/listen/foo.txt", 3.5495989999999997]
["~/Code/floehopper/listen/foo.txt", 6.942901]
["~/Code/floehopper/listen/foo.txt", 7.41488]
["~/Code/floehopper/listen/foo.txt", 7.5622180000000006]
["~/Code/floehopper/listen/foo.txt", 8.245715]
["~/Code/floehopper/listen/foo.txt", 6.022797]
["~/Code/floehopper/listen/foo.txt", 7.021637]

Ok so it looks like only BSD/MacOS do not have support for using a native filesystem eventing system.

My development machine is running MacOs, so I guess this explains the results above - presumably in this case it has fallen back to polling...?

This seems to be confirmed by adding the following line to the script above:

puts watcher.class.name # => "Java::SunNioFs::PollingWatchService"

[floehopper]

I've also just noticed that the JRuby builds are failing even on master, so I'm going to revert my changes to the Travis CI config and force-push to this branch.

[floehopper]

Doh! The Linux adapter was being picked up in preference to the JRuby adapter. I've moved the JRuby adapter to the beginning of the OPTIMIZED_ADAPTERS array. Continuing to debug...

[houndci-bot]

Layout/IndentationConsistency: Inconsistent indentation detected.

[houndci-bot]

Metrics/BlockLength: Block has too many lines. [26/25]

[floehopper]

I've realised I don't yet fully understand how the gem is architected and so I think I have quite a bit of work to do before this will be ready to merge, so I'm going to close the PR, but continue working on the branch.