Testing a Node.js Express API server with Vows (functional)

How to test an API that must be authenticated via session?

vows: plural of vow

Noun: A solemn promise.
Verb: Solemnly promise to do a specified thing: “one fan vowed, “I’ll picket every home game.””.

Oh wait..

Vows.js
Asynchronous behaviour driven development for Node.

I should… test every method I expose to the client..
Since the Vows documentation is very awesome, and I already have another post that explains the strategy used on the API, there is no need to much talk; just not that this code is still very early in maturity.
Straight to current code [ test/api-test-authed.js ]

/*
 * INSTRUCTIONS
 *
 * run the site at localhost, port 8010
 *
 * run vows --spec test/api-test-authed.js
 *
 */


var request = require('request'),
    vows = require('vows'),
    assert = require('assert'),
    apiUrl = "http://localhost:8010/",
    cookie = null


var apiTest = {
  general: function( method, url, data, cb ){
    //console.log( 'cb?', cb )
    request(
      {
        method: method,
        url: apiUrl+(url||''),
        json: data || {},
        headers: {Cookie: cookie}
      },
      function(req, res){
        cb( res )
      }
    )
  },
  get: function( url, data, cb  ){ apiTest.general( 'GET', url, data, cb    )  },
  post: function( url, data, cb ){ apiTest.general( 'POST', url, data, cb   )  },
  put: function( url, data, cb  ){ apiTest.general( 'PUT', url, data, cb    )  },
  del: function( url, data, cb  ){ apiTest.general( 'DELETE', url, data, cb )  }
}

function assertStatus(code) {
  return function (res, b, c) {
    assert.equal(res.statusCode, code);
  };
}


function assertJSONHead(){
  return function(res, b, c ){
    assert.equal( res.headers['content-type'], 'application/json; charset=utf-8' )
  }
}

function assertValidJSON(){
  return function(res, b ){
    // this can either be a Object or Array
    assert.ok( typeof( res.body ) == 'object' )
    //assert.isObject( res.body)
  }
}





// TODO include unauthed tests
var suite = vows.describe('API Localhost HTTP Authenticated Tests')

// Very first test!
.addBatch({
  "Server should be UP as in: var apiUrl": {
    topic: function(){
      apiTest.get('', {} ,this.callback )
    },

    '/ should repond something' : function(res, b){
      assert.ok(res.body)
    }
  }
})

.addBatch({
  'Authenticate to /login': {
    topic: function(){
      request.post(
        {
          url: "http://localhost:8010/login",
          json: { user:{ username: 'flockin_lab', password: '123456' }}
        },
        this.callback
      );
    },



    'get a valid Cookie': function(req, res, body, err){
      try{
        cookie = res.headers['set-cookie'].pop().split(';')[0]
        console.log("GOT COOKIE!", cookie)
      } catch(e){ }

      assert.ok( typeof(cookie) == 'string' && cookie.length > 10 )
    }
  }
})
.addBatch({
  'Users#index': {
    topic: function(){
      apiTest.get('admin/employees', {}, this.callback)
    },
    'should be 200': assertStatus(200),
    'should have JSON header' : assertJSONHead(),
    'body is valid JSON' : assertValidJSON(),

  },
})
.addBatch({
  'Qrcodes#index': {
    topic: function(){
      apiTest.get('admin/qrcodes', {}, this.callback)
    },
    'should be 200': assertStatus(200),
    'should have JSON header' : assertJSONHead(),
    'body is valid JSON' : assertValidJSON(),

  },
})

//suite.run( )
suite.export( module )

Brief Discussion

This code is still state-of-art,
We depend on the lib ‘request’, which is pretty good,
The server should already be started, since I see no point in having the test being responsible to bring it up and handle it.

Can improve it? Please leave a comment! :)

accepts_nested_attributes_for 3.0.5 :reject_if still have gotchas

Hey

Nested forms are nasty, but many times, a necessary evil :)

Rails tackled it since version 2.3x, when added the ActiveRecord method accepts_nested_attributes_for, main ref here. For its complexity, I believe, it is still hard to master it, and still bugs may happen; this was my lesson.

Ryan Bates, (a ruby hero) has made some screencasts from it, watch part1, part2. Soon after, he and some buds, made a gem for it, targeting Adding and Removing via JavaScript, without harassment. You can find on GitHub, along with the necessary doc.

So, using the gem or not, still there are things to cover over this methods params;

From what I could observe, :reject_if, does not work for ‘rejecting’ a existing record, i.e.: those that have ID. But, it is called for those too! – In this topic, (what may be a bug) is that, if the record gets rejected, it does not get updated (it does not go through it’s model’s validations either).

The solution I found was not using :reject_if,  instead, validating whatever I wanted on the nested Model in order to keep dry.

For this scenario, consider the following setup (rails v3.0.5)

class A < ActiveRecord::Base
 has_many :bs
 accepts_nested_attributes_for :bs,
                               :allow_destroy => true
                               #no use. :reject_if => :combo_elem_zero?

 # wont use this function at all
 def combo_elem_zero?( att )
  #puts(atribs.to_yaml)
  #att['_destroy'] = '1' # wont work here

  # only useful for new records
  if atribs['id'].blank? && att[:some_atrib].to_f < something_blah
   true
  else
   false
  end
 end
end

class B < ActiveRecord::Base
 belongs_to :a

 validate :destruction_update # works both for create/update

 def destruction_update
  if self.some_atrib.to_f < something_blah
   self.mark_for_destruction
  end
 end

end

If on the other hand, you only need to check on the nested new records, :reject_if may do the job.

Migrating from SQLite to Mysql on Rails 3

Common sense on developers use to say: Don’t use sqlite on production, because it sux in performance.

Well, for a system with around 20 users that should be no problem at all in the begin, right? – Right but also Wrong!

I’ve tested using the system on the same settings, but for some reason, after a while the reading on the Database by Rails became very inconsistent. While in console I would find some recently created objects by Rufus Scheduler, the web app would not see then until the server was restarted! Weird results, but a price to pay for not listening the common sense =)

The solution would be to migrate the schema and data to some easy to use Mysql. Schema as widely known is extremely easy to migrate on Rails framework (if used properly).

Searching for solutions for migrating data from different databases may be a pain, but given a great Rails tool it is rather simple! Introducing the gem yaml_db, made by the Heroku people.

As the commands are simple as :dump and :load, I don’t see why go longer in the explanation.

The whole process took less than 2 hours, including a small bug fix in a non-agnostic code =)

At service of BananaBrains, FIRE the social soccer game RPG.

JavaScript Simple String Templating System, Ruby like

Ever wanted to use the beautifulness of the Ruby String template (a.k.a #{}) on JavaScript? Now you can!

  var hero = new Object(); hero.name = 'Conan'; hero.lv = 1;
  S("#{ hero.name } Level Up! He is now Lv:#{ hero.lv+1 } ") // "Conan Level Up! He is now Lv:2 "
  S("Date & Time now: #{ new Date() }") // "Date & Time now: Sat Nov 13 2010 19:20:16 GMT-0200 (BRT)"

The source of this goodness you can be found here: FlockonUS-GitHub long with other Utils :)

When to use it? It is hard to say that there is a limitation to this function.
In my case, I use it on a WebApp ( that requires JavaScript), to eval the response of Ajax before appending the content to the page.

Rails User Action Logger

Hi!

The problem: I need to log my user’s actions on the site ( a game) so I can Datamine it later, It is also very desirable to investigate each user individually.

My solution: (still primitive) A ruby lib that is able to accumulate some actions (in a array) and then append it to a log  File. Aafter_filterapplication_controller that auto-logs every action taken. The action can set a custom warn level, and a custom message.

I made a GIST with this. link1 link2

Usage: The usage is actually automatic, but you can custom with 2 options on a Action, Example:

def create
    @user_session = UserSession.new(params[:user_session])
    
    @bf_errors = 10
    @bf_minutes = 10
    #logger.info ">>>> #{@bfw.to_s}, #{session['create_number']}"
    
    if @bfw = brute_force_warning
      captcha = verify_recaptcha(:model => @user_session, :message => "Favor digite as letras que aparecem distorcidas")
      if (@bfw == :active && !captcha)
        @warn = 8
        @action_obs = "#{@bf_errors} per #{ @bf_minutes} minutes. Email:#{@user_session.try( :email)}, pass: #{@user_session.try( :password)}. "
        render :action => "login"
        return true
      end
    end

Discussion: I chose JSON for being very extremely easy to parse on JavaScript, which has great Libs for data visualization.
I considered the rails Buffered Logger, Beanstalk and direct File Write, but I believe the method I’m using is way faster than those. My assumption is because all I do is native to Ruby, with minimal memory use, and disc access.

TODO:
. A plugin that handles this.
. A controller-action-view to visualize this Data =)

LazyLoad on JavaScript Load only once

What? And what’s the point?

On some web pages, the focus is displaying information fast, but also being aware that some actions that rely on a kinda heavy framework such as jQuery/Prototype/Dojo (Ajax+Animations+etc..) may still be done in page by a minority of users.

I believe the use cases of this technique are sparse, but would be around pages focusing Mobile Phones, where the processing and bandwidth are limited and pages that should prioritize very fast loadings.

To solve several browsers implementation for calling for JS after the page is fully loaded I used this library (LazyLoad 2.0) by a great programmer, a non-evil-doer. With this great lib is pretty easy to call a JS (or css for that matter) but is complicated to know when the Framework  you required is already loaded ( in the case that multiple acions may trigger the load)

For this case I developed a sort of proxy-function that should do the job in the following way:

        function load_and_run( callback, arg ){
		if( typeof($) == 'function' ){
			// This means jQuery is loaded
			callback( (typeof( arg ) != 'undefined' ? arg : true))
		}else{
			LazyLoad.js('/javascripts/jquery-1.4.2.js', function(arg){
				j=$;
			        //as here you can use LazyLoad to require some plugin
				callback( (typeof( arg ) != 'undefined' ? arg : true) );
			}, arg)

		}
	}

The secret sauce there is that I simply verify if the $ ( signature of jQuery, Prototype and others) is already defined, and for that to work I must proxy other actions by load_and_run, as can be seen here:

var spinner = new Image();	spinner.name = "madd-spinner";	spinner.src = "/images/ajax-loader.gif";
	
	function up_request( id, node ){
		load_and_run( up_request_callback )
		node.parentNode.appendChild( spinner )
	}
	function up_request_callback( id ){
		j.ajax({
					url: '/ajax/up',
					data:{definition_id:id},
					//etc...
					})

Remember to always give some feedback about the loading process for your user, in my case, I use a spinner gif preloaded.

Lesson learned from building www.gasa.jaeh.net, can be seen on any definition page

SEO para novos domínios

Terminou hoje um curso de férias muito bom que aconteceu na faculdade Unifil ministrado por @jorgehen, gerente do site GrandeEletro . Só posso agradecer pelo conhecimento que ele transmitiu e a atenção aos interessados, porque não são muitos que fazem isto, ainda mais gratuitamente :P

Aprendi muito na palestra, com enfoque na lógica do negócio experiente, e aprendi ainda várias novidades hoje em SEO, que seria Search Engine Optimization, e vou dividir um pouco em um tópico que ainda hoje não conhecia bem.

Digamos que você faz um site, e como a maioria quer uma boa posição no Google, isto é aparecer nos primeiros. Bom, desde a compra do domínio podem ser colocadas em prática alguns métodos, pois os mecanismos de busca avaliam entre MUITAS outras coisas o tempo de vida do site e seu acesso. Com isso em mente vou dar uns tiros aqui de coisas em ordem que eu posso imaginar de relevância (aceito correções):

  • Registrou o domínio? Então já coloque logo um index.html para os buscadores saberem que você existe.
  • No index, coloque palavras chaves variadas que tratam do assunto futuro do site
    • * Mito
  • Coloque alguns links que levem à sites com boa colocação no Google
    • *Ainda mais importante, coloque links para este domínio em sites reconhecidos, (um Tweet por ex.)
    • *Registrar na DMOZ.
  • ! Não coloque muitos links proporcionalmente ao texto, ou seu site é marcado como spam
  • ! Sites Spam são a escória e ficam lááá nas últimas páginas
  • Depois de lançado o site pode ser interessante gastar um pouquinho com AdSense
  • Advertising contribuirá para que seu site ganhe uma impulso para alcançar os existentes

Bom, creio que é isso para ter um bom nascimento, em muitos outros sites você deve encontrar técnicas super relevantes como sitemap, url semântica, coerência, etc..

Update – 28/03/2011 : Atualizados itens com * segundo recomendações do meu amigo @richardrx, profissional de SEO.

*Menos de 50% das pessoas clica após o 2º link exibido em uma busca do Google, para estar em primeiro consulte um profissional :)

Live Validation simple File Validation

Taking a breath from this speeding job, lol, it probably makes no sense, but has been crazy here.. anywho!

This is a simple snippet or recipe for using the marvelous Live Validation on a simple file extension verification.

The advantage of this is huge, because you can check the kind of file being sent without your server handling the full download of the file! And the user has a quicker feedback! Win – Win

Validate.Anexo = function(value, options) {
    var extensoes = options.ext.split(" ") || ['png', 'jpg', 'bmp', 'gif', 'jpeg']
    var message = options.failureMessage || "A extensão do arquivo está incorreta, são aceitos: " + extensoes.join (' ');
    if (  validateArquivo(value, extensoes)  )
        Validate.fail(message);
    return true;
}

function validateArquivo(val, extensoes){
	var exp = ".*\.(" + extensoes.join('|')+")$" 
	exp = new RegExp(exp,'i') // ex: /.*.(png|jpg|bmp|gif|jpeg)$/i
	if (val.length > 0) {
		return ( ! exp.test(val) )
	}
	return true;
}

Then you use on the view through the arg -> ‘ext’, that should be a string insensitive, where every extension is separated by space

var file = new LiveValidation('document_atach', {validMessage: " ", onlyOnSubmit: true} )
					.add(Validate.Anexo, { ext : 'xml txt doc' } )

Otimização de chamada de função em JS

Em algumas situações a chamada repetida de ajax e javascripts pesados e custosos pode virar um incomodo, tanto para o cliente (processamento, demora), quanto para o server (processamento, banda). Com uma tecnica muito simples você pode elimina-las.

O Problema: Em algumas situações, como carregamento dinamico de página, alguns campos podem ser ativados, chamando funções repetidamente.

A Solução: Criar uma função intermediária que chama a função custosa por meio de setTimeout! Veja:

Seu código antigo:

function valida_parcelamento() {
    bla()
    bla()
    //...
    new Ajax.Request('/acao_demorada/', {} )
}

Código otimizado:


var timeout = 9999
function valida_parcelamento(){
        // veja mais detalhes sobre esta var em Discussão
	var limite = 1000
	jQuery("#aviso").html( " É legal por alguma mensagem de feedback aqui, ou spinner" );
	// Cancela qq chamada de funcao menos recente, dentro do tempo limite
	clearTimeout(timeout)
	// Mantem a chamada de funcao + recente
	timeout = setTimeout("valida_parcelamento_slave()", limite)
}

function valida_parcelamento_slave() {
    bla()
    bla()
    //...
    new Ajax.Request('/acao_demorada/', {} )
}


Discussão: O modo de fazer é bastante simples, a função antiga tem seu nome concatenado com algum diferencial, no meu caso, escolhi chama-las de _slave e uma nova função é criada com o nome da antiga. Esta nova função tem uma função de gateway  para com a antiga, na prática ignorando as chamadas mais antigas e consecutivas.

Pelo lado positivo, esta tecnica não requer alterar nenhum HTML, e pode reduzir em muitas vezes a número de requisições.

Pelo lado negativo, a var limite é muito complicada de ser balanceada devido ao seu tradeoff: um número muito pequeno se torna inútil, pois em um computador lento as coisas demoram a carregar, então a função _slave acabaria sendo chamada varias vezes; por outro lado um número grande é ruim, pois obriga o usuário esperar d+ por uma resposta (tempo de execução da function + limite(ms)).

O ideal seria calcular a eficiencia do computador do usuário uma vez por meio de um benchmark simples e armazenar em cookie para usa-la de limite.

Caso, a funcionalidade de gateway seja desejável somente durante o tempo de carregamento, é possível desabilitar depois. Esta parte fica como tarefa :P