Loggly middleware for Express.js (server visibility)

Hi!

Oh how long without posting, I’ve been missing it dearly, but in transition between jobs I was super busy, and now I can take the time to add some cool stuff =)

Use Case

I’ve been building an API, it is very cool, but the visibility for it is naturally low compared to websites. Loggly offer a great service for monitoring servers via a simple API available in node.js. While this API is cool, it is super broad and require some experimentation to implement an adequate visibility level.

Wouldn’t be amazing to be able to see every request that come in your server and how it respond? Utopic? Absolutely possible!

Loggly API

Implementation

This implementation has been heavily inspired by Connect logger middleware – which is great.

I deeply recommend you to tune it for your visibility needs, though. Make good use: https://gist.github.com/flockonus/5380753

// suggested use:
app.configure(function() {
  app.set("port", process.env.PORT || 3005);
  app.use(logglyMiddleware);
  // ..
  app.use(express.bodyParser());
  app.use(app.router);
  // ..
});

Notes worth taking

  • This snippet will show you information in your console when o development
  • The info sent might be too much in terms of size or frequency, make good use of customization for your needs
  • If you are dealing with a website, I advice you to not send every path hit, it is just too much

Cheers!

CarrierWave – limit file size (plus gif fix)

CarrierWave has an awesome abstraction API. It is simple, clear and extensible. But has some critical vulnerability specially when combined with image processing, such as, ImageMagick when resizing an image will consume exponencial memory size and any upload can easily make your process crash, when not processed safely. Also, it is not pretty good to combine .gif out of the box, because it makes a collection out of the file.

Friendly advice beforehand; Using http://filepicker.io/ may be a way better idea if you are hosting in Heroku, just make sure if fits your constraints before get hard work done.

Solution Spec

Hard limit file size of the request, so the process don’t block for too long, and don’t blow memory!

If you behind a server such as Apache or Nginx, you can impose a limit to the request size, and you should!

Unless you are in Heroku, and afaik, there is no way to do that, at least just yet. So yes, this can be a major security breach for Rails apps on Heroku.

Given a successful upload, pre-validate size.

The ‘official’ solution attempt to validate the size after the file have been processed. It doesn’t help, since when processing an image rather large (6Mb image consumed 2GB memory in my case) your process will be killed! Letting your website down for some time, and letting your users down as well.

For gifs, take only the first image (less memory consumption too)

When processing .gifs it seems to make a vertical frameset will all the images in the sequence, so it looks like a movie roll, which is not what most people want. Lets just extract the first frame.

Interestingly enough, I found that the processor is invoked for all frames in the .gif. (thanks debugger!)

Solution code

This code takes care the mentioned specs (except for the request size limit), and I think the great advantage is that it avoids opening a file as Image if it fails the size constraint. As well as being very efficient with gifs (only acting on the first frame).
It works on Heroku, with integration for S3, and should work on Amazon Cloud and other VPS.

The shortcome is about handling the exception which is a bit messy involving controller-side logic in a non-automated AR fashion.

Controller

  def create
    begin
    @post = Post.new(params[:post])
    rescue Exception => e
      if e.message == 'too large'
        redirect_to news_path(err: 'file')
      else
        raise e
      end
    end
   #...

uploader

# encoding: utf-8


class NewsUploader < CarrierWave::Uploader::Base

  include CarrierWave::RMagick

  include Sprockets::Helpers::RailsHelper
  include Sprockets::Helpers::IsolatedHelper


  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  def pre_limit file
    #require 'debugger'; debugger
    if file && file.size > 5.megabytes
      raise Exception.new("too large")
    end
    true
  end

  def only_first_frame
    manipulate! do |img|

      if img.mime_type.match /gif/
        if img.scene == 0
          img = img.cur_image #Magick::ImageList.new( img.base_filename )[0]
        else
          img = nil # avoid concat all frames
        end
      end
      img
    end
  end

  version :large, if: :pre_limit do
    process :only_first_frame
    process :convert => 'jpg'
    process :resize_to_limit => [1280, 1024]
  end

  # Create different versions of your uploaded files:
  version :small, if: :pre_limit do
    process :only_first_frame
    process :convert => 'jpg'
    process :resize_to_limit => [360, 360]
  end


  # For images you might use something like this:
  def extension_white_list
    %w(jpg jpeg gif png)
  end

end

Posting a photo (as a business) to a Facebook Page via JavaScript SDK

The Facebook Javascript SDK (API) is not so well documented, what makes not so logical to post to a page, using it. For that I am about to show, make sure your connect scope having at least: “manage_pages,publish_actions,publish_stream”.

I am assuming you already succeeded providing login to the user; if this is not the case, refer to the docs.

We rely heavily on the FB.api method, the docs make a poorly job documenting, but is a starting point anyway. Also, notice the response is async.

Fetch all user pages (actually the first 5k), but only store the first as var page (this is just a sample)

var page = null;
FB.api('/me/accounts', function(response) { console.log(response); page = response.data[0]; })

Now, we want to publish a post to the page timeline, the thing is, we are not posting as ‘me’ anymore, we must post as the page, and it requires both the id and access_token!
I will make a sample function for it, the response is the post id if it all went well, or an error.

function postToPage(page, msg){
  FB.api('/'+page.id+'/feed', 'post', { message: msg, access_token: page.access_token },
    function(res) { console.log(res) }
  )
}
postToPage(page, 'hello fb-page world')

For last, posting a picture requre the url param, like so:

function photoToPage(page, src, msg){
  FB.api('/'+page.id+'/photos', 'post', { url: src, message:msg, access_token: page.access_token },
    function(res) { console.log(res) }
  )
}
photoToPage(page, 'http://24.media.tumblr.com/tumblr_m1ttif5puW1qcrr0lo1_500.png', 'nyan art')

* Don’t try to run all functions in sequence, since the result comes asynchronously you wont have the var page set.

** kudos for Владимир Дворник answer on Stack Overflow for the url param not documented in the Page Graph API.

*** Security note. The JavaScript SDK can be very unsafe in the case a hacker is able to inject JS code in your page (and even happens in Twitter all the time even though they have millions in funding) they may take advantage of all of the user’s permissions.

Done!