Easy zip generating with Rails, Nginx & Paperclip
Tonight I decided to try making NGINX handle my zip generating, to save the server some work, and the user of some frustration waiting for background jobs to finish. Here’s what I came up with.
First of all, I re-compiled Nginx with two modules; mod_zip, and passenger module. You may drop the Passenger module if you are using some other server, this solution works with anything that can print out some text :)
Roughly, these are the required steps:
1 2 3 4 |
gem install passenger cd nginx-folder/ .configure --add-module=/path/to/passenger/gem/ext/nginx --add-module=/path/to/mod_zip-1.x sudo make && sudo make install |
If you wonder where your passenger is, you can easily find the root of passenger by running
passenger-config --root |
So on to the fun part! I started by adding a member method to my galleries route called zip:
1 2 3 |
ActionController::Routing::Routes.draw do |map| map.resources :galleries, :member => {:zip => :get} end |
And In my controller I added these two methods (it can probably be done cleaner, but hey, it’s working :) Suggestions for other ways of writing these are welcome!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
class GalleriesController < ApplicationController def zip @gallery = Gallery.find(params[:id]) paperzip(@gallery.images.collect(&:data), @gallery.name) end private def paperzip( attachments= [], name= 'file', style= :original ) # set headers response.headers["Content-Disposition"] = "attachment; filename=#{name}.zip" response.headers["X-Archive-Files"] = "zip" # Generate file list render :text => attachments.collect{|a| ['-', File.size(a.path(style)), a.url(style,false), a.original_filename.gsub(/ /,'_')].join(' ') }.join("\r\n")+"\r\n" end end |
Whoa! Let me break that down for you a little bit. Firstly the member method zip, which now has the named route “zip_gallery_path(@gallery)” in you views btw:
1 2 3 4 |
def zip @gallery = Gallery.find(params[:id]) paperzip(@gallery.images.collect(&:data), @gallery.name) end |
This basically finds our gallery from the :id param, and calls on this method we’re defining as private for this controller, called paperzip() (I know, catchy). paperzip() takes a collection of Images (which has_attached_file :data), a string containing a name for the zip file about to be made, and a symbol which you can use to zip down the style you want. I’m running with :original here.
Let’s move on to the next couple of lines:
1 2 |
response.headers["Content-Disposition"] = "attachment; filename=#{name}.zip" response.headers["X-Archive-Files"] = "zip" |
What we do here is setting some custom headers for our response back to the client, and Nginx. The content-disposition tells it we want the user to be prompted to download a file, called something.zip. The X-Archive-Files header is the one that makes mod_zip trigger, and eagerly await a list of files to zip down and send. Mod_zip requires a list of files that looks something like this:
md5 bytes /relative/path filename<linebreak> |
Note that the linebreak needs to be a \r\n, and be on every line, even the last one. So, in order to construct this list, I added the following lines to the method, returning it using “render :text”
1 2 3 |
render :text => attachments.collect{|a| ['-', File.size(a.path(style)), a.url(style,false), a.original_filename.gsub(/ /,'_')].join(' ') }.join("\r\n")+"\r\n" |
And voila! Calling this action with a get request, will now trigger mod_zip and create us a fully working zip file. The good thing about using this over some other method, is that Nginx only uses a couple of kilobytes of that precious memory of yours while doing this, and does the job way quicker than any other method I’ve tried.
Lars commented about 4 hours later (November 20, 2009 23:58)
Found a small bug in Unzippers that check for CRC32, and corrected it with this code. However, I can see this is getting ugly and should be moved somewhere else probably, but it’s a proof of concept atleast :)
Lars commented about 5 hours later (November 21, 2009 01:05)
Since I had a rather boring evening to start with, I even released it as a gem (my first) @ GitHub
Dave commented 3 days later (November 24, 2009 01:43)
Stumbled across this article from reddit… Good write up! I did something similar with a music sharing application.
You can use nginx to stream the files instead of your rails app. Notice the response.headers[‘X-Accel-Redirect’] header. Notice the “location /music” directive. When setting the X-Accel-Redirect header, nginx will ‘redirect’ to that location.
Write a comment