Background processing with Paperclip
Image resizing, or other post-processing of Paperclip attachments, can be a long running task in some scenarios. Luckily for us, it’s pretty simple to hook on to Paperclip and run the heavy parts as a background job. This example goes through a normal photo upload.
I’m going to assume that you are familiar with Paperclip by now. If you’re not, you should download and implement immediately. The background daemon I’m using is delayed_job but you are of course free to use your weapon of choice. Ryan Bates recently did a nice screencast about delayed_job which I highly recommend (check out this Ascii-cast if you don’t like videos).
Ok, so on to the cool stuff, we start by making a table and all that:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
class CreateImages < ActiveRecord::Migration def self.up create_table(:images) do |t| # default paperclip fields t.string :data_file_name t.string :data_content_type t.integer :data_file_size t.datetime :data_updated_at # processing t.boolean :processing, :default => true # timestamps t.timestamps end end def self.down drop_table :images end end |
Pretty straight forward migration. The processing boolean is used to check whether a particular photo has been processed yet. It defaults to true, since we will always intercept Paperclip’s processing when a new Photo is added.
So let’s create an ImageJob, the little delayed_job gnome that will be doing all the dirty work.
1 2 3 4 5 |
class ImageJob < Struct.new(:image_id) def perform Image.find(self.image_id).perform end end |
This is a regular pattern for creating jobs in delayed_job. What it does is basically to (on perform) find the image in question, and call the instance method perform() on it which will do all the processing stuff. I normally add these jobs into lib/ since they’re not models per say.
In our Image model, we add the following:
1 2 3 4 5 6 7 |
before_data_post_process do |image| false if image.processing? # do not process if just added end after_create do |image| Delayed::Job.enqueue ImageJob.new(image.id) # add to queue end |
This utilizes Paperclip’s before_:attachment_post_process callback to halt the processing and just go through with the create action (save the record in db). After create, we add this Image to our delayed job queue.
Since it might take some time processing our Image, we add this little convenience method which serves two purposes; normally when using Paperclip you have to do model.attachment.url(:style) to get the url of the attachment, this method allows you to to model.url(:style). We are also able to return a default URL while processing the Image in the background.
1 2 3 4 5 6 |
def url(style = :original) if self.data && processing? && style != :original return data.send(:interpolate, @@default_url, "#{style}") end data.url(style) end |
So now we are ready to try out the actual processing, with this insanely advanced method on our Image model:
1 2 3 4 5 |
def perform self.processing = false # unlock for processing data.reprocess! # do the processing save end |
And that’s it. You can now start up your delayed_job queue listener, go into console and try out with a few files.
10.times { Image.create(:data => File.open('/home/lars/test.jpg')) } |
Happy hacking :-)
Throlkim commented 3 months later (November 09, 2009 15:48)
Thank you very much for this – I’d been struggling for the last day to figure out how to do this – I kept running into trouble with the queue not finding the file. It’s been a huge help, thanks muchly :)
xinuc commented 3 months later (November 13, 2009 04:52)
Dammit man!! I just copy your code word by word..
Thanks a lot..
incd commented 7 months later (March 21, 2010 08:25)
I’ve been trying to get this working, but I keep getting error:
AntonNemtsev commented 7 months later (March 22, 2010 04:36)
incd, show me your Model code and I’ll try to help…
cristi commented 9 months later (May 08, 2010 02:13)
Perfect. Thank you very much for this.
Write a comment