Sporadic SSL errors

[espen]

Description

I have been running v7 in production for about a month but the last week I started seeing OpenSSL errors. I see there has been a breaking change in v7 in regards to certs but I was running it for about 4 weeks without any issues and then the last week I have started seeing errors like the one below. This occurs only in Sidekiq jobs that make external requests. I do retry failed jobs and the jobs succeed upon retry. I run many similar jobs but the error is only raised occasionally.

The error is: SSL_connect returned=1 errno=0 state=error: wrong version number from OpenSSL::SSL::SSLError.

Troubleshooting

/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/protocol.rb:44→ connect_nonblock
/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/protocol.rb:44→ ssl_socket_connect
/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/http.rb:1009→ connect
/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/http.rb:943→ do_start
/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/http.rb:932→ start
/PROJECT_ROOT/vendor/ruby-2.7.3/lib/ruby/2.7.0/net/http.rb:1483→ request
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16→ block in request_with_newrelic_trace
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:26→ block (2 levels) in request_with_tracing
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/tracer.rb:371→ capture_segment_error
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:25→ block in request_with_tracing
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent.rb:501→ disable_all_tracing
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/instrumentation.rb:24→ request_with_tracing
/GEM_ROOT/gems/newrelic_rpm-7.0.0/lib/new_relic/agent/instrumentation/net_http/chain.rb:16→ request_with_newrelic_trace
/GEM_ROOT/gems/airbrake-11.0.3/lib/airbrake/rails/net_http.rb:11→ block in request
/GEM_ROOT/gems/airbrake-11.0.3/lib/airbrake/rack.rb:21→ capture_timing
/GEM_ROOT/gems/airbrake-11.0.3/lib/airbrake/rails/net_http.rb:10→ request
/PROJECT_ROOT/lib/sms/clx.rb:12→ send

The last line is a net/http post request.

Your Environment

Rails 6.1.3.2. Ruby 2.7.3. Heroku.

[tannalynn]

@espen, thanks for letting us know about this issue you're seeing.

I wouldn't expect the cert change in 7.0 to be the issue, since if you were lacking the certs that we removed, you wouldn't be able to connect to the new relic servers in the first place and no data would be coming through. Since you had 4 weeks of it working correctly, that seems unlikely to be the issue.

Trying to look up the error, most of the things I'm seeing about that type of error are related to the server response, which seems like it would be weird since you are only seeing in sidekiq jobs sometimes. Very odd.

Has anything changed at the time you first started seeing this error? Maybe anything related to ruby or openssl/OS/environment?

[wyhaines]

I just read this, and I also wonder if perhaps there might be more than one generation of sidekiq worker running at the same time? Maybe there are a few old ones running even though they should not be, and they just need to be killed?

[espen]

I'm no longer seeing this in my app, not sure what changed 🤷‍♂️ The only place I am still seeing this error is in a Heroku buildpack that doesn't support Redis SSL: feedforce/heroku-buildpack-sidekiq-metrics#2

@wyhaines no, it is on Heroku so workers will be restarted for each deploy.