Skip to content

Add support for Rails 5.2.3#48

Merged
mriddle merged 3 commits intomasterfrom
mriddle-rails-5.2.3-support
Aug 26, 2019
Merged

Add support for Rails 5.2.3#48
mriddle merged 3 commits intomasterfrom
mriddle-rails-5.2.3-support

Conversation

@mriddle
Copy link
Copy Markdown
Contributor

@mriddle mriddle commented Aug 22, 2019

What

Upon upgrading from Rails 5.1.6.2 to 5.2.3, deploys to staging started failing
with the following exception during the readiness probe (curl server:port/z/ping)

NoMethodError (undefined method `any?' for nil:NilClass

The backtrace for the error was originally obfuscated by Rails due to backtrace
silencers. To reveal the full backtrace we added the following lines to an
initializer

Rails.backtrace_cleaner.remove_silencers!
Rails.backtrace_cleaner.remove_filters!

The exception was coming from here:
https://github.com/rails/rails/blob/v5.2.3/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#L423

Rails has modified the way it handles database connections in

Full backtrace

NoMethodError (undefined method `any?' for nil:NilClass)
/usr/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `block in connected?'
/usr/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/usr/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `connected?'
/usr/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
/usr/bundle/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:1021:in `connected?'
/usr/bundle/gems/activerecord-5.2.3/lib/active_record/connection_handling.rb:123:in `connected?'
/usr/bundle/gems/rollbar-2.15.5/lib/rollbar/middleware/rails/rollbar.rb:63:in `person_data_proc'
/usr/bundle/gems/rollbar-2.15.5/lib/rollbar/middleware/rails/rollbar.rb:44:in `fetch_scope'
/usr/bundle/gems/rollbar-2.15.5/lib/rollbar/middleware/rails/rollbar.rb:20:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/bundle/gems/rollbar-2.15.5/lib/rollbar/middleware/rails/show_exceptions.rb:22:in `call_with_rollbar'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/ddtrace-0.19.1/lib/ddtrace/contrib/rails/middlewares.rb:16:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/app/lib/middleware/error_uuid_middleware.rb:9:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/usr/bundle/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'
/usr/bundle/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/app/lib/hc/json_formatter.rb:90:in `tagged'
/usr/bundle/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
/usr/bundle/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/app/lib/middleware/tagged_logging_middleware.rb:17:in `block in call'
/usr/bundle/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/app/lib/hc/json_formatter.rb:90:in `tagged'
/usr/bundle/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
/app/lib/middleware/tagged_logging_middleware.rb:16:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/rack-utf8_sanitizer-1.3.2/lib/rack/utf8_sanitizer.rb:19:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/activesupport-5.2.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/usr/bundle/gems/actionpack-5.2.3/lib/action_dispatch/middleware/static.rb:127:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:95:in `call'
/app/lib/middleware/middleware_tracing_middleware.rb:56:in `call'
/usr/bundle/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/usr/bundle/gems/raindrops-0.19.0/lib/raindrops/middleware.rb:116:in `call'
/app/lib/middleware/request_queue_time_middleware.rb:13:in `call'
/app/lib/middleware/http_method_not_allowed_middleware.rb:20:in `call'
/usr/bundle/gems/ddtrace-0.19.1/lib/ddtrace/contrib/rack/middlewares.rb:82:in `call'
/usr/bundle/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'

Replication:

I could reproduce the behaviour with config.cache_classes = config.eager_load = true set in my config/environment/development.rb and executing rails runner "Process.fork { ActiveRecord::Base.connected? }"

This patch adds supports to this gem for Rails 5.2.3 with the following

  • Don't auto-create connection if nil on methods 5916c96
  • Clear entry from PoolProxy when Rails discards a ConnectionPool 460b4d1

@mriddle mriddle requested a review from bquorning August 22, 2019 07:46
@mriddle mriddle self-assigned this Aug 22, 2019
Unless told otherwhise, when this proxy class intercepts methods, it will create
a connection if one does not already exist.

https://github.com/zendesk/active_record_host_pool/blob/v0.11.0/lib/active_record_host_pool/pool_proxy.rb#L107-L108

In Rails 5.2.0, they added the following functionality which caused issues with
code mentioned above
https://github.com/rails/rails/blob/v5.2.3/activerecord/lib/active_record/railtie.rb#L180-L195

Here's the associated PRs
rails/rails#28057
rails/rails#31221

This patch overrides the methods, retrieves the connection if it's active and
calls super without creating a new one.

This resolves an issue for applications upgrading to Rails 5.2.3 whereby
connections would be created and thrown away during Rails.application.initialize!
@mriddle mriddle force-pushed the mriddle-rails-5.2.3-support branch 4 times, most recently from f375bf0 to b255483 Compare August 23, 2019 15:04
New functionality was introduced in Rails 5.2 to ensure forked children don't
send quit/shutdown/goodbye messages to the server on connections that belonged
to their parent. See rails/rails#31173 for more information.

We need to ensure that when the following line is called we also discard the
pools we've cached.

ActiveRecord::ConnectionAdapters:::ConnectionHandler.discard_unowned_pools

Without this change, applications using this gem and Rails 5.2.3 will run into
`NoMethodError (undefined method 'any?' for nil:NilClass)` exceptions from
forked processes (e.g. Puma/Unicorn workers) since they'll be using the cached
and discarded connection.

The upshot of this change is we can do away with code like this:
https://github.com/rails/rails/blob/c39ed435eb578c79867552c66da7eeb035fa58ad/railties/lib/rails/generators/rails/app/templates/config/puma.rb.tt#L35-L53
@mriddle mriddle force-pushed the mriddle-rails-5.2.3-support branch from b255483 to 26b879d Compare August 23, 2019 15:09
@mriddle mriddle requested review from bquorning and removed request for bquorning August 23, 2019 15:12
@mriddle
Copy link
Copy Markdown
Contributor Author

mriddle commented Aug 23, 2019

These changes have also been verified against a Rails 5.1 application to ensure they're backwards compatible.

@mriddle mriddle force-pushed the mriddle-rails-5.2.3-support branch from 26b879d to 5547a83 Compare August 26, 2019 06:09
Comment thread test/test_arhp.rb
# Verify that when we fork, the process doesn't crash
pid = Process.fork do
if ActiveRecord::VERSION::MAJOR == 5 && ActiveRecord::VERSION::MINOR == 2
assert_equal(false, ActiveRecord::Base.connected?) # New to Rails 5.2
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this test fail before the code change was applied?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!

Copy link
Copy Markdown
Contributor Author

@mriddle mriddle Aug 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified the fix in 3 different ways

  • Stashing the changes to pool_proxy.rb and running this unit test
  • Booting a Rails 5.2 console, forking a process and making a request to the DB
  • Deploying a Rails 5.2 application with these changes and running smoke tests

@bquorning
Copy link
Copy Markdown
Member

I’d like to have an extra set of eyes on this PR – @grosser ?

@mriddle mriddle requested a review from grosser August 26, 2019 12:13
Comment thread Readme.md Outdated
end

def release_connection(owner_thread = Thread.current)
p = _connection_pool(false)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this work ?
_connection_pool(false)&.release_connection(owner_thread)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but I'd prefer to leave the 3 methods as-is. It's easier to grok at a glance IMO.

Add an example command on how to run tests
@mriddle mriddle force-pushed the mriddle-rails-5.2.3-support branch from 5547a83 to de9359a Compare August 26, 2019 14:56
@mriddle
Copy link
Copy Markdown
Contributor Author

mriddle commented Aug 26, 2019

Thanks for the feedback and quick reviews! 🙏

@mriddle mriddle merged commit c43d766 into master Aug 26, 2019
@mriddle mriddle deleted the mriddle-rails-5.2.3-support branch August 26, 2019 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants