Easy zip generating with Rails, Nginx & Paperclip

Tonight I decided to try making NGINX handle my zip generating, to save the server some work, and the user of some frustration waiting for background jobs to finish. Here’s what I came up with.

First of all, I re-compiled Nginx with two modules; mod_zip, and passenger module. You may drop the Passenger module if you are using some other server, this solution works with anything that can print out some text :)

Roughly, these are the required steps:

1
2
3
4
gem install passenger
cd nginx-folder/
.configure --add-module=/path/to/passenger/gem/ext/nginx --add-module=/path/to/mod_zip-1.x
sudo make && sudo make install

If you wonder where your passenger is, you can easily find the root of passenger by running


passenger-config --root

So on to the fun part! I started by adding a member method to my galleries route called zip:

1
2
3
ActionController::Routing::Routes.draw do |map|
  map.resources :galleries, :member => {:zip => :get}
end

And In my controller I added these two methods (it can probably be done cleaner, but hey, it’s working :) Suggestions for other ways of writing these are welcome!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class GalleriesController < ApplicationController
  def zip
    @gallery = Gallery.find(params[:id])
    paperzip(@gallery.images.collect(&:data), @gallery.name)
  end
  
  private 
    def paperzip( attachments= [], name= 'file', style= :original )
      # set headers
      response.headers["Content-Disposition"] = "attachment; filename=#{name}.zip"
      response.headers["X-Archive-Files"] = "zip"
  
      # Generate file list
      render :text => attachments.collect{|a| 
        ['-', File.size(a.path(style)), a.url(style,false), a.original_filename.gsub(/ /,'_')].join(' ')
      }.join("\r\n")+"\r\n"
    end
end

Whoa! Let me break that down for you a little bit. Firstly the member method zip, which now has the named route “zip_gallery_path(@gallery)” in you views btw:

1
2
3
4
def zip
  @gallery = Gallery.find(params[:id])
  paperzip(@gallery.images.collect(&:data), @gallery.name)
end

This basically finds our gallery from the :id param, and calls on this method we’re defining as private for this controller, called paperzip() (I know, catchy). paperzip() takes a collection of Images (which has_attached_file :data), a string containing a name for the zip file about to be made, and a symbol which you can use to zip down the style you want. I’m running with :original here.

Let’s move on to the next couple of lines:

1
2
response.headers["Content-Disposition"] = "attachment; filename=#{name}.zip"
response.headers["X-Archive-Files"] = "zip"

What we do here is setting some custom headers for our response back to the client, and Nginx. The content-disposition tells it we want the user to be prompted to download a file, called something.zip. The X-Archive-Files header is the one that makes mod_zip trigger, and eagerly await a list of files to zip down and send. Mod_zip requires a list of files that looks something like this:


md5 bytes /relative/path filename<linebreak>

Note that the linebreak needs to be a \r\n, and be on every line, even the last one. So, in order to construct this list, I added the following lines to the method, returning it using “render :text”

1
2
3
render :text => attachments.collect{|a| 
    ['-', File.size(a.path(style)), a.url(style,false), a.original_filename.gsub(/ /,'_')].join(' ')
  }.join("\r\n")+"\r\n"

And voila! Calling this action with a get request, will now trigger mod_zip and create us a fully working zip file. The good thing about using this over some other method, is that Nginx only uses a couple of kilobytes of that precious memory of yours while doing this, and does the job way quicker than any other method I’ve tried.


7255ce34f45920257de1237b7ade2c1e

Lars commented about 4 hours later (November 20, 2009 23:58)

Found a small bug in Unzippers that check for CRC32, and corrected it with this code. However, I can see this is getting ugly and should be moved somewhere else probably, but it’s a proof of concept atleast :)

1
2
3
4
5
6
7
8
render :text => attachments.collect{|a|
  [
    Zlib.crc32(File.read(a.path(style)),0).to_s(16), 
    File.size(a.path(style)), 
    a.url(style,false), 
    a.original_filename.gsub(/ /,'_')
  ].join(' ')
}.join("\r\n")+"\r\n"
7255ce34f45920257de1237b7ade2c1e

Lars commented about 5 hours later (November 21, 2009 01:05)

Since I had a rather boring evening to start with, I even released it as a gem (my first) @ GitHub

12be02204a4bcaeec2c6c369fe53a45b

Dave commented 3 days later (November 24, 2009 01:43)

Stumbled across this article from reddit… Good write up! I did something similar with a music sharing application.

You can use nginx to stream the files instead of your rails app. Notice the response.headers[‘X-Accel-Redirect’] header. Notice the “location /music” directive. When setting the X-Accel-Redirect header, nginx will ‘redirect’ to that location.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# Nginx example...
server {
  location / {
    # There's stuff in here, but it's not relevant... 
  } 
  location /music {
    root /some/path/to/music;
    internal;
  }     
}


class Album < ActiveRecord::Base
  def self.download_songs(artist, album, songs)
    body = ""
    songs.each do |song|
      url = "/artists/#{artist.id}/albums/#{album.sha1}/songs/#{song.sha1}/download?zip=true"
      body << "#{song.crc_32} #{song.file_size} #{url} #{File.basename(song.file_path)}\r\n"
    end
    body
  end
end

class AlbumsController < ApplicationController
  def download
    # Takes care of all the download logic.
    # before_filter finds @artist, @album, and @album.songs...
    album_songs = Album.download_songs(@artist, @album, @album.songs)
    response.headers['X-Archive-Files'] = 'zip'
    response.headers['Content-Disposition'] = "attachment; filename=#{@artist.name.gsub(' ', '_')}-#{@album.name.gsub(' ','_')}.zip"
    render :layout => false, :text => album_songs
  end
end

class SongsController < ApplicationController
  def download
    @artist = Artist.find(params[:artist_id])
    @album = @artist.albums.find_by_sha1(params[:album_id])
    @song = @album.songs.find_by_sha1(params[:id])

    nginx_file_path = @song.file_path[DESTINATION_LIBRARY.length..-1]                                                                
    response.headers['X-Accel-Redirect'] = File.join("/music", nginx_file_path)
    response.headers['Content-Type'] = 'audio/mpeg' 
    response.headers['Content-Disposition'] = "attachment; filename=#{@artist.name.gsub(" ","_")}-#{File.basename(@song.file_path.gsub(" ", "_"))}"
    render :nothing => true
  end
end

Write a comment