JSONP with Express.js API – cross domain request

The problem:

So you have this awesome API endpoint, but of course it is another domain as the static site.

Web browsers are pretty moody with Cross Domain and there are actually very good security reasons for that, but even using good practices the old Internet Explorer don’t care.

Welcome ancient jsonp techniques!

Solution

For some Express.js API running on Node.js and client using jQuery (or something alike)

In your server:

app.get('/my-unique-endpoint', function(req, res, next) {
  var myAwesomeResource = ["hello", "mundo"];
  if (req.query && req.query.callback) {
    res.set({
      // nice to have, but Chrome dont seem to mind either way
      'Access-Control-Allow-Origin': '*',
      // right content type prevent warnings and errors
      'Content-Type': 'text/javascript; charset=UTF-8',
      // optional, this is in seconds, equivalent to 8h
      'Cache-Control': 'public, max-age=28800'
    });
    return res.send(200, "" + req.query.callback + "(" + JSON.stringify(myAwesomeResource) + ");");
  } else {
    return res.send(200, myAwesomeResource);
  }
});

In the client:

// see the data in your console
$.getJSON("http://myawesomewebsiteurl/my-unique-endpoint?callback=?", function(data){ console.log(data) } )

Caveats

jsonp is works for GET verb, because it effectively execute the code received as if it was downloading a script.
for the same as above, the source should also be fully trusted!

Advertisements

Loggly middleware for Express.js (server visibility)

Hi!

Oh how long without posting, I’ve been missing it dearly, but in transition between jobs I was super busy, and now I can take the time to add some cool stuff =)

Use Case

I’ve been building an API, it is very cool, but the visibility for it is naturally low compared to websites. Loggly offer a great service for monitoring servers via a simple API available in node.js. While this API is cool, it is super broad and require some experimentation to implement an adequate visibility level.

Wouldn’t be amazing to be able to see every request that come in your server and how it respond? Utopic? Absolutely possible!

Loggly API

Implementation

This implementation has been heavily inspired by Connect logger middleware – which is great.

I deeply recommend you to tune it for your visibility needs, though. Make good use: https://gist.github.com/flockonus/5380753

// suggested use:
app.configure(function() {
  app.set("port", process.env.PORT || 3005);
  app.use(logglyMiddleware);
  // ..
  app.use(express.bodyParser());
  app.use(app.router);
  // ..
});

Notes worth taking

  • This snippet will show you information in your console when o development
  • The info sent might be too much in terms of size or frequency, make good use of customization for your needs
  • If you are dealing with a website, I advice you to not send every path hit, it is just too much

Cheers!

ImageMagick on Joyent

It is really simple:


pkgin search ImageMagick # you will get some result like this:
ImageMagick-6.6.6.5 Package for display and interactive manipulation of images
p5-PerlMagick-6.6.6.5 Object-oriented Perl interface to ImageMagick
php53-imagick-5.3.5.2.2.2 Provides a wrapper to the ImageMagick library
ruby18-RMagick-2.13.1nb3 Ruby binding to ImageMagick
ruby18-mini-magick-2.1 Ruby wrapper for ImageMagick command line
ruby19-RMagick-2.13.1nb3 Ruby binding to ImageMagick
ruby19-mini-magick-2.1 Ruby wrapper for ImageMagick command line

Now, install the latest version, today it is 6.6.6.5, so run:

pkgin install ImageMagick-6.6.6.5

Done!

For the Node.js package “imagemagick” at “0.1.2” works for my use cases, which are just resizes 🙂
If you are using other languages you may need the other packages as well.

Mongoose validate unique field (insensitive)

Making a validation function for Mongoose (Node.js – MongoDB ODM) that checks upon validation field.

function uniqueFieldInsensitive ( modelName, field ){
	return function(val, cb){
		if( val && val.length ){ // if string not empty/null
			
			var query = mongoose.models[modelName]
				.where( field, new RegExp('^'+val+'$', 'i') ) // lookup the collection for somthing that looks like this field 
			
			if( !this.isNew ){ // if update, make sure we are not colliding with itself
				query = query.where('_id').ne(this._id)
			}
			
			query.count(function(err,n){
				// false when validation fails
				cb( n < 1 )
			})
		} else { // raise error of unique if empty // may be confusing, but is rightful
			cb( false )
		}
	}
}

Then call it to a field (aka path):

UserSchema.path('nick').validate( uniqueFieldInsensitive('User', 'nick' ), 'unique' )

Recommend you organize all validations in a file for itself, or even a folder if grows too big.

If you have a unique index set for the path you may just save it and check for the error raised; the thing is the format of the error.

Basic Authentication on Node.js – Express and Mongoose

via Axiom Zen

Hello!

This post is about the most important parts I can think of implementing signup and login to Node.js powered by Mongoose and Express.js.

This post have a huge audience, but guys, please keep in mind that this post was done a while ago. It works pretty well as a cook-book, but there probably are better options for achieving this nowadays, new npm packages and such.

Implementing last night, I was impressed on how fast had the functional part done (around 3 hours), pretty much the same time I’d had it done in Rails (if not faster).

Goal:

  • 3 urls: GET login & signup, POST login, POST signup.
  • User fields: email, nick, password(encrypted).
  • Validate (and show) messages for all fields.
I’ve setup a small Github project that may be used as example (assuming you have mongoDB installed and running in localhost) – DOWNLOAD IT FROM GITHUB  

Model: user.js

var Schema = mongoose.Schema
  , ObjectId = Schema.ObjectId
  , Validations = require('./validations.js')
  , salt = 'mySaltyString'
  , SHA2 = new (require('jshashes').SHA512)()

function encodePassword( pass ){
	if( typeof pass === 'string' && pass.length < 6 ) return ''

	return SHA2.b64_hmac(pass, salt )
}

var UserSchema = new Schema({
    nick        : {type: String, required: true, unique: true, trim: true }
  , email       : {type: String, required: true, unique: true, trim: true, lowercase: true }
  , password    : {type: String, set: encodePassword, required: true }
});

UserSchema.statics.classicLogin = function(login, pass, cb) {
	if( login && pass ){
		mongoose.models.User
			.where( 'email', login )
			.where( 'password', encodePassword(pass) )
	  	.findOne( cb )
	} else {
		// just to launch the standard error
		var o = new this({nick: 'VeryUniquejerewelA', password: '', email: login+'aaa'})
		o.save(cb)
	}
}
UserSchema.path('nick').validate( Validations.uniqueFieldInsensitive('User', 'nick' ), 'unique' )
UserSchema.path('email').validate( Validations.uniqueFieldInsensitive('User', 'email' ), 'unique' )
UserSchema.path('email').validate( Validations.emailFormat, 'format' )
UserSchema.path('password').validate( Validations.cannotBeEmpty, 'password' )
UserSchema.plugin( mongoose.availablePlugins.timestamper )

mongoose.model('User', UserSchema)

.Highlights of this code: We are using the package jshashes, which supplies many convenient encryption methods, among those SHA512 –strong enough
It is a good practice to use a salt along, represented by the var salt. In practice it makes way difficult for a cracker that acquired access to the database do decipher the passwords stored.
The method encodePassword is used at two occasions, when setting the User password, and when retrieving it from database.
UserSchema.statics is a object that stores additional static methods our User model will offer.
The function classicLogin requires both login and pass to search the db for existence, otherwise, it will launch an error (kinda of a smelly workaround to make it work dry )
UserSchema.path(…).validate offers us validations, in our case, we do not allow repeated email or nick, and password should be bigger at least 6 characters long. Also email should at least look like a email.
All those validations work along with the own Schema definition: required, unique, trim, lowercase

Route: auth.js

// app.get( '/auth/popover', auth.popover);
exports.popover = function(req, res){
	//req.session.popover = new Date()
	console.log('My session:', req.session)
  res.render('auth/index_pop', req.viewVars);
};

// CLASSIC LOGIN / SIGNUP       --because everyauth seems too messy for login+pass
// app.post('/auth/classic-signup', auth.classicSignup)
exports.classicSignup = function(req,res,next) {
	if( !req.body ){
		console.log('why u signup nobody?')
		return res.redirect('/?nobodySignup')
	}

	var user = new app.models.User()

	user.set('nick', req.body.nick)
	user.set('email', req.body.email)
	user.set('password', req.body.pass)
	user.set('providers', ['signup:'+user.get('email')])
	user.set('profiles', [{ _name: 'signup'}])

	user.save( function(err) {
		if( err ){ // validation failed

			req.viewVars.u = user
			return classicYieldErr( req, res, 'signUp', err)

		} else { // signup successful

			req.session.user = {
				provider: 'signup',
				id: user.get('id'),
				nick: user.get('nick'),
			}

			req.flash('notice', 'Welcome!')
			req.viewVars.welcome_login = "Welcome, "+user.nick

  		res.render('auth/win_pop', req.viewVars )
		}
	})
};

// app.post('/auth/classic-login',  auth.classicLogin)
exports.classicLogin = function(req,res,next) {
	if( !req.body ){
		console.log('why u login nobody?')
		return res.redirect('/?nobodyLogin')
	}

	app.models.User.classicLogin( req.body.email, req.body.pass, function(err, user) {
		if( err ){ // validation failed

			return classicYieldErr( req, res, 'signIn', err)

		} else {

			if( user ){ // login

				req.session.user = {
					provider: 'signup',
					id: user.get('id'),
					nick: user.get('nick'),
				}

				req.flash('notice', 'Welcome!')
				req.viewVars.welcome_login = "Welcome, "+user.nick

	  		res.render('auth/win_pop', req.viewVars )

			} else { // not found
				return classicYieldErr( req, res, 'signIn', {errors:
					{'loginpass': {
						name: 'V',
						path: 'login+password',
						type: 'loginpass'
					}
				}})
			}

		}
	})
};

// display form error
function classicYieldErr( req, res, mode, err ){
	req.viewVars.erroredForm = mode
	if( mode === 'signIn' ){
		req.viewVars.signin_errors = app.helpers.displayErrors( err )
	} else {
		req.viewVars.signup_errors = app.helpers.displayErrors( err )
	}
	req.viewVars.email = req.body.email

	res.render('auth/index_pop', req.viewVars);
}

Here we define the 3 routes;
– one route(GET) that defines one page for both login or signup
– one route(POST) to submit login
– one route(POST) to submit signup
About the last 2, their only role is to allow access for valid data. That is, valid signup data, or login+password existing in our collection from MongoDB.
The function classicYieldErr is there just to serve the errors in sort of an uniform way. We will see about this function on the next file

Model Helper: validations.js

exports.uniqueFieldInsensitive =  function ( modelName, field ){
	return function(val, cb){
		if( val && val.length ){ // if string not empty/null
			// only for new docs
			if( this.isNew ){
				mongoose.models[modelName].where(
					field, new RegExp('^'+val+'$', 'i')
				).count(function(err,n){
					// false when validation fails
					cb( n < 1 )
				})
			} else {
				cb( true )
			}
		} else { // raise error of unique if empty // may be confusing, but is rightful
			cb( false )
		}
	}
}

exports.emailFormat = function( val ){
	// false when validation fails
	return (/^([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})$/i).test( val )
}

exports.cannotBeEmpty = function( val ){
	console.log('pass val is:', val)

	// not all passwords should be set, BUT when string, should be encoded
	if( typeof val === 'string' ){
		if( val.length ){ // when it comes empty, something went wrong!
			return true
		} else {
			return false
		}
	} else {
		return false
	}
}

This file contains some very important function, being intuitive, I will only comment on the first.
uniqueFieldInsensitive is actually a validation creator. It uses JS’s closure ability along with the powerful this object to wrap the context where this will represent whichever Model is calling it.
The intent of the function is to search the db for any repeated occurrence of the field, case-insensitive. This way the system will block a user “John” from signinup, if “john” is present.

Helper: form_helper.js


//app.helpers.displayErrors = require('./helpers/form_helper.js').displayErrors

stripErrors = function(mongooseErr){
	var prop, list = [];
	for( prop in mongooseErr.errors ){
		list.push( [mongooseErr.errors[prop].path, mongooseErr.errors[prop].type] )
	}
	return list
}

/*
 * Translate mongoose errors into a <li> of errors
 */
exports.displayErrors = function( mongooseErr ){

	console.log( 'mongoose errs', mongooseErr )

	var list = stripErrors( mongooseErr )

	var output = []
	list.forEach(function(e,i){
		switch( e[1] ){
			case( 'unique' ):
				output.push( e[0]+" is taken" )
				break;
			case( 'required' ):
				output.push( e[0]+" is "+e[1] )
				break;
			case( 'format' ):
				output.push( e[0]+" has a bad format" )
				break;
			case( 'password' ):
				output.push( "password should be at least 6 char long" )
				break;
			case( 'loginpass' ):
				output.push( "login+password not found" )
				break;

			default:
				output.push( e[0]+": "+e[1] )
				break;
		}
	})
	if( output.length ){
		// condense all items in an error list
		output = [output.join( '</li><li>\n' )]
		output.unshift( '<ul><li>' )
		output.push( '</li><ul>' )

		// wrap in a div
		output.unshift( "<div class='error block'>" )
		output.push('</div>')
	}
	return output.join('\n')
}

This file provides a translation from Mongoose errors to user-readable errors. In order to offer a good UX, it is important to give accurate feedback. Notice this file could benefit from some refinement 🙂

Don’t forget to download the code from github, it plays well along everyauth module 🙂

Cheers!

Ruby’s fear-cancer

This video: High Performance Ruby: Threading Versus Evented by Dr Nic Williams, Engine Yard

It meant so much to me!

For around 4 months I’ve been using Node.js, and before that, for 3 years, programming Rails.

As soon as I started on Node.js, I could feel that something was different. My little Computer Science bachelor conscience was starting to tell me: now you are starting to do it right 🙂

But then, lets go back, Ruby’s syntax is lovely.. Rails API is sugar! But still, it’s neverland. Why??

It does not fit real world! It is a thin layer of happiness, so delicate, that we feel fear of touching it and breaking it. So what we puny humans do? We delegate it to those genius heroes who will be able to take that code and maybe squeeze a few more couples of requests/sec!

But not even that is the cancer I mean.

The cancer here is represented by clown hosting the video. “Oh those scientific articles mess with our head and stuff O.o Let me chew that evil reality and I will give you the RIGHT (sugar coated) solution!” and all other non fun nor assertive statements. By the way, he may be referring to the C10K problem, and alike, articles.

The problem here is, VP  EngineYard and want ppl addicted on his junk. He does not host Node.js.

Ruby always had so many language implementations to solve some important language problems, as well as servers. This is not the first time a good solution for ruby is born (JRuby+Trinidad), but why would they share it before if he could have people paying for so much RAM?

The reason I believe he felt compelled to share it now, is that Node.js unoptimized code can outperform Rails by a LOT,  under 150Mb.

But my main problem is fear of complexity. The host talks about it all over the initial part of the video. He refers his audience under constant fear of it’s own ignorance all the time. This is a awful, and that’s how I felt using Rails. Such a large stack, complex language design, only understanding the framework code itself was hard task.

To keep people in bliss ignorance he just goes like: “Oh I promissed evented? We are past that, right?! HAAAA” so just be happy and keep the status quo. You would’t want to mess your pretty little head with asynchronous code, would you?

I like Node.js, besides performance, language is simple; it is JavaScript, open Objects, closure and more. Of course, don’t assume well-written JS is something easy to do. No great code in any modern language is easy to achieve.

If you’ve developed Rails, you have concepts of MRI, thin, mongrel, jruby, 1.8.7, 1.9.2, rubinius, unicorn, and LOT MORE, whilst in node.js world all that stands for: node.js. It is a unification point for language and server.

What about gems?  npm, a easier system of distributing packages.

Wrapping it up, node.js is no silver bullet, it just made me realize the problems I had while programming ruby, as much as ruby did it for me on php. If nothing else, node.js helped improve ruby’s community by adding options to the web development mainstream.

Would I work with Rails again? If a employer would point me that out as the only chosen solution, then yes, and I’d probably try that stack, but I prefer confidence instead of fear.

When to Ruby on Rails, when to Node.js

(update) Take this post as a naive overview, it may not reflect the most accurate reality

Hello!

I am trying to do a sort of indirect comparison between Rails and Node.js. The very main reason of being indirect, is that Rails is a Framework, while Node.js is a runtime with custom libraries.

If it were be to put in a simple phrase, Rails is resourceful and Node.js is light and fast.

Lets elaborate some more..

Rails

Is the most complete open-source framework available (that I know of)! Big companies use it. It can do lots of stuff well, in an organized manner; this meaning, Rails is more than just MVC, it has a full stack of features well-integrated, at the same time being very modular. Some of the features included out of the box:

  • Database adapter for the majority of them, supporting plug your own.
  • Database migrations, so multiple dev can sync and experiment with their DB.
  • Powerful engines for Views, Controllers and Models.
  • Support to code generator.
  • Has structure to all sorts of tests and friendly to TDD.
  • Really awesome documentation.
  • Model has all kinds of hooks, validations and associations.
  • Controller has support to handle XML/JSON in the same action that serves HTML.
  • Gems that integrate, for instance, Memcached, MongoDB, Auth and lots more.
So Rails is war-proven, capable of integrating lots of features together without harass. There is also a very cool post of Fabio Akita in the refs. about how it made possible to develop systems in periods before impossible.

Node.js

Two things make this platform suitable for web:

Its engine, V8 is very fast! In a very loose average, 8 times faster than Python (or even up to 200 at peak). Python already outperforms Ruby (ref. bottom)

Second point; and this argument is separated from the above, is that it async-driven (is built around reactor pattern). As in the case, requests can be performed in parallel, without any blocking I/O. A single server can handle a lot. (update) And with >0.6.0 Cluster API, it can scale to use all of available CPU cores.

So, it is a very new sort of backend language, but huge players, besides Joyent, who invented it, are adopting it, including LearnBoost and LinkedIn, which has an awesome article about using. The language, and it’s main web framework, Express, deserve a list of features (you can check more info in the references below).

  • It´s web server is able to handle a HUGE number of connections out of the box
  • Various libraries can be run on browser, the same as in the server
  • Very friendly to Websockets (real-time web apps)
  • Lots of libraries are being ported to it from other langs.
  • Express, inspired in ruby´s Sinatra; is very light on memory but also very powerful
Running a simple benchmark against a single server instance, I were able to get 587 req/s accessing MySQL without any external cache support. This number could scale, if I used Cluster to spawn at least a process per processor.

Summarizing, When to use each?

Rails really shines when..

  • The database is complex in terms of associations.
  • The app structure is well defined.
  • Business rules are complex, and validation is needed.
  • When the number of requests isn´t the a decisive factor.
  • Administrative interfaces.
  • Many developers in parallel keep the DB up-to-date with migrations
  • The database to be used is undefined, or may vary.
What about Node.js?
  • APIs
  • Real-time web/mobile apps.
  • Application that should scale to lots of concurrent requests.
  • Little memory footprint

This being said, there is no reason at all, a web-site or service can´t easily integrate both.

— I’d appreciate if you could leave a comment, either to talk about your case, or add up.

References

 

UPDATE: http://www.mikealrogers.com/posts/a-new-direction-for-web-applications-.html

http://guides.rubyonrails.org/

http://railscasts.com/

http://blog.heroku.com/archives/2011/6/22/the_new_heroku_2_node_js_new_http_routing_capabilities/

http://nodejs.org/

http://akitaonrails.com/2011/04/16/twitter-muda-de-ruby-para-java-ruby-e-3x-mais-lento-que-java

http://venturebeat.com/2011/08/16/linkedin-node/

http://blog.bossylobster.com/2011/08/lesson-v8-can-teach-python-and-other.html

https://github.com/LearnBoost

http://www.readwriteweb.com/hack/2011/01/how-3-companies-are-using-node.php

http://twitter.com/#!/FlockonUS/status/104655096956190720

https://github.com/LearnBoost/cluster