slow export

[tsah]

Export is very slow when dealing with a big number of entries. It took me about 6 minutes to export 4000 entries. Is there any way to speed it up?

[bbenezech]

Export needs a few things:

• auto-include association (you may have n+1 issues)
• batch treatment (loading 4k objects in memory is probably a bad idea)
• AJAX processing (even if user waits, front servers will timeout anyway)

Any PR welcome.

[outworlder]

Indeed, I've just run into this problem. Heroku will terminate requests after 30 seconds.

[rmcastil]

I'm currently running into this problem and would be up for creating a PR for fixing it.

@bbenezech does the rails_admin team currently have a plan on how they wanted to solve this issue?

[tensiondriven]

@sferik - Any ideas on how to work around this timeout issue on Heroku? Either by optimizing queries, adding indexies, or (worst case) forking rails_admin to use something like sidekiq to generate requested CSV's asynchronously and send them by email? This is a major issue for companies using rails_admin to power their operations/customer service. (Using Mongoid fwiw)

[tensiondriven]

Could this be best and most easily fixed by changing the export code to use streaming?

From https://devcenter.heroku.com/articles/request-timeout : "Cedar supports HTTP 1.1 features such as long-polling and streaming responses. An application has an initial 30 second window to respond with a single byte back to the client. However, each byte transmitted thereafter (either received from the client or sent by your application) resets a rolling 55 second window. If no data is sent during the 55 second window, the connection will be terminated."

[sferik]

Request streaming is only supported by Rails 4. I'm not quite ready to remove Rails 3 support for this feature but feel free to put it in a branch. As for doing the work in the background, that adds a lot of complexity.

RailsAdmin isn't intended to be your app, it's intended to be an admin interface for your app. It was never designed to "power [your] operations/customer service". You might want to find or build another tool to handle your exports if that's essential to the business.

[natarius]

@sferik I assume all the export code is in https://github.com/sferik/rails_admin/blob/master/lib/rails_admin/config/actions/export.rb

right!?

[natarius]

I would try to build a solution that creates the file in a sidekiq task (if sidekiq is available) and the browser would just poll the server every X seconds until its ready.

Opinions?

[marks]

+1 .. has anyone implemented a streaming or background solution before I go ahead and try to tackle this?

[natarius]

Nope....didn't come up with a simple enough design to not add too many dependencies (sidekiq/resque, S3, etc)

But would still love to have a solution for this :)

[mshibuya]

#1783 might partially solve this problem.

[dalpo]

#1830

[robertjwhitney]

Interested in helping with eager loading. Right now rails admin only eager loads belongs_to on a collection by design (hard coded). Thoughts on the design for making this optional, maybe specifically for export?

 def get_collection(model_config, scope, pagination)
      associations = model_config.list.fields.select {|f| f.type == :belongs_to_association && !f.polymorphic? }.map {|f| f.association[:name] }
      options = {}
      options = options.merge(:page => (params[Kaminari.config.param_name] || 1).to_i, :per => (params[:per] || model_config.list.items_per_page)) if pagination
      options = options.merge(:include => associations) unless associations.blank?
      options = options.merge(get_sort_hash(model_config))
      options = options.merge(:query => params[:query]) if params[:query].present?
      options = options.merge(:filters => params[:f]) if params[:f].present?
      options = options.merge(:bulk_ids => params[:bulk_ids]) if params[:bulk_ids]

      objects = model_config.abstract_model.all(options, scope)
    end

[robertjwhitney]

Adding some sort of option to list_entries, maybe?