truffleruby fails sinatra tests (liquid) NoMethodError: undefined method 'peek_byte' for #<StringScanner ...>

[dentarg]

Please let me know if it is useful that I report like this or not – but I this seemed like something you would like to investigate as it was passing just some time ago and nothing changed in Sinatra regards to this (well, maybe some dependencies, as I mention below)

Both truffleruby and truffleruby-head fails the Sinatra tests currently

• truffleruby 24.1.2, like ruby 3.2.4, Oracle GraalVM Native [x86_64-linux] https://github.com/sinatra/sinatra/actions/runs/13119142286/job/36914681384#step:5:41
• truffleruby 25.0.0-dev-1a3571eb, like ruby 3.3.5, GraalVM CE Native [x86_64-linux] https://github.com/sinatra/sinatra/actions/runs/13119142286/job/36914681310#step:5:41

The tests were passing ~3 months ago (I see tilt 2.4.0 was used here, in the more recent runs tilt 2.6.0 was used, may be more differences, not sure it that one is relevant)

• truffleruby 24.1.1, like ruby 3.2.4, Oracle GraalVM Native [x86_64-linux] https://github.com/sinatra/sinatra/actions/runs/11936860840/job/33271497492
• truffleruby 24.2.0-dev-49a44754, like ruby 3.3.5, GraalVM CE Native [x86_64-linux] https://github.com/sinatra/sinatra/actions/runs/11936860840/job/33271497045

Example fail

  1) Error:
LiquidTest#test_renders_with_inline_layouts_0:
NoMethodError: undefined method `peek_byte' for #<StringScanner 0/15 @ "<EM>S...">
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/tokenizer.rb:74:in `next_token'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/tokenizer.rb:65:in `shift_normal'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/tokenizer.rb:57:in `tokenize'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/tokenizer.rb:33:in `initialize'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/parse_context.rb:45:in `new_tokenizer'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/template.rb:105:in `parse'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/liquid-5.7.2/lib/liquid/template.rb:85:in `parse'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/tilt-2.6.0/lib/tilt/liquid.rb:19:in `prepare'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/tilt-2.6.0/lib/tilt/template.rb:126:in `initialize'
    lib/sinatra/base.rb:939:in `compile_block_template'
    lib/sinatra/base.rb:926:in `block in compile_template'
    lib/sinatra/base.rb:962:in `block in fetch'
    <internal:core> core/hash.rb:236:in `fetch'
    lib/sinatra/base.rb:961:in `fetch'
    lib/sinatra/base.rb:925:in `compile_template'
    lib/sinatra/base.rb:873:in `render'
    lib/sinatra/base.rb:783:in `liquid'
    test/liquid_test.rb:30:in `GET /'
    lib/sinatra/base.rb:1807:in `block in compile!'
    lib/sinatra/base.rb:1074:in `block (3 levels) in route!'
    lib/sinatra/base.rb:1092:in `route_eval'
    lib/sinatra/base.rb:1074:in `block (2 levels) in route!'
    lib/sinatra/base.rb:1123:in `block in process_route'
    <internal:core> core/throw_catch.rb:36:in `catch'
    lib/sinatra/base.rb:1121:in `process_route'
    lib/sinatra/base.rb:1072:in `block in route!'
    lib/sinatra/base.rb:1069:in `each'
    lib/sinatra/base.rb:1069:in `route!'
    lib/sinatra/base.rb:1193:in `block in dispatch!'
    <internal:core> core/throw_catch.rb:36:in `catch'
    lib/sinatra/base.rb:1164:in `invoke'
    lib/sinatra/base.rb:1188:in `dispatch!'
    lib/sinatra/base.rb:1004:in `block in call!'
    <internal:core> core/throw_catch.rb:36:in `catch'
    lib/sinatra/base.rb:1164:in `invoke'
    lib/sinatra/base.rb:1004:in `call!'
    lib/sinatra/base.rb:993:in `call'
    rack-protection/lib/rack/protection/base.rb:53:in `call'
    rack-protection/lib/rack/protection/xss_header.rb:20:in `call'
    rack-protection/lib/rack/protection/path_traversal.rb:18:in `call'
    rack-protection/lib/rack/protection/json_csrf.rb:28:in `call'
    rack-protection/lib/rack/protection/base.rb:53:in `call'
    rack-protection/lib/rack/protection/base.rb:53:in `call'
    rack-protection/lib/rack/protection/frame_options.rb:33:in `call'
    lib/sinatra/middleware/logger.rb:17:in `call'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-3.1.9/lib/rack/head.rb:15:in `call'
    lib/sinatra/base.rb:227:in `call'
    lib/sinatra/base.rb:2138:in `call'
    lib/sinatra/base.rb:1677:in `block in call'
    lib/sinatra/base.rb:1898:in `synchronize'
    lib/sinatra/base.rb:1677:in `call'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-3.1.9/lib/rack/lint.rb:66:in `response'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-3.1.9/lib/rack/lint.rb:41:in `call'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-test-2.2.0/lib/rack/test.rb:360:in `process_request'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-test-2.2.0/lib/rack/test.rb:163:in `custom_request'
    vendor/bundle/truffleruby/3.2.4.24.1.0.1/gems/rack-test-2.2.0/lib/rack/test.rb:112:in `get'
    /home/runner/.rubies/truffleruby-24.1.2/lib/mri/forwardable.rb:240:in `get'
    test/liquid_test.rb:32:in `test_renders_with_inline_layouts_0'

[andrykonchin]

Thank you for the report, we'll look into it.

It's indeed useful to have such issues reported!

[eregon]

peek_byte was added in ruby/strscan#89 i.e. it's a fairly recent addition of ruby/strscan (in fact the strscan version shipped with CRuby 3.3.5 doesn't have that method).
strscan is implemented in pure-Ruby on TruffleRuby because it's far more efficient that way.
TruffleRuby is tested in ruby/strscan's CI but the peek_byte test is omitted which makes sense.

TLDR: peek_byte needs to be implemented in https://github.com/oracle/truffleruby/blob/master/lib/truffle/strscan.rb and then that test can be unomitted.
We should also check if other methods are missing compared to upstream while at it.

At some point we should consider upstreaming https://github.com/oracle/truffleruby/blob/master/lib/truffle/strscan.rb to ruby/strscan, it's some work though and we need to define e.g. which Primitives are stable enough to be used outside this repo, etc.

FWIW the PR which removed truffleruby from liquid CI seems Shopify/liquid#1167