NodeJS Cron Job with Kue

At Cozy we needed to write some jobs of this kind for our Mails app. For that we used Kue a simple task manager for NodeJS that requires Redis to run. The documentation is great but there was no sample for writing a cron task. That’s why I share this little snippet that runs a 2 seconds long job every 3 seconds, in case you need something similar.

NB : The delay between each job is set in minutes.

kue = require 'kue'

Job = kue.Job

jobs = kue.createQueue()

# Set up server if you want to see your task progression with a beautiful UI
kue.app.listen 3003

# Your cron timing
delay = 3000

# Function used to launch a job.
repeatJob = ->
    job = jobs.create "test complete",
        title: "my job"
        info: "job is working"

    job.on 'promotion', () ->
        console.log job.data.title + " #" + job.id + " promoted"

    job.delay(delay)
    job.save()

# The job to run, a fake task that is 2 seconds long.
cronTask = (job, done) ->
    global.currentJob = job.id
    console.log job.data.title + " #" + job.id + " job started"
    setTimeout ->
        console.log "my job is done"
        repeatJob()
        done()
    , 2000

# Register job
jobs.process "test complete", myFunc = cronTask

# Check for new job every 3s, change this value to set your cron timing.
jobs.promote(delay)

# Run the cron job for the first time.
repeatJob()
Advertisements

Whoosh: full-text search with Python

To add an efficient search function to the product I work on, I was looking for a good indexer. Elastic Search, a Java indexer that is managed through a REST api, looks good but it requires to set-up a dedicated server: it’s not a library but a full software. Another option was Xapian, looks efficient, but not very well documented.

Then I discovered Whoosh, a Python library which offers indexing and search features. The documentation and the API makes it really easy to use. The performance are probably worst than the Elastic Search or Xapian but it should be enough for a lot of projects. The library provides a lot of search strategies and functionalities (stemming, faceting, highlighting…). In conclusion, if you have a Python project that requires full-text search, you should definitely have a look at it.

To illustrate this article here is a little snippet I wrote that index a list of blog posts located in MongoDB database.

import os

from whoosh.fields import Schema, ID, KEYWORD, TEXT
from whoosh.index import create_in
from whoosh.query import Term

from pymongo import Connection
from bson.objectid import ObjectId

# Set index, we index title and content as texts and tags as keywords.
# We store inside index only titles and ids.
schema = Schema(title=TEXT(stored=True), content=TEXT,
                nid=ID(stored=True), tags=KEYWORD)

# Create index dir if it does not exists.
if not os.path.exists("index"):
    os.mkdir("index")

# Initialize index
index = create_in("index", schema)

# Initiate db connection
connection = Connection('localhost', 27017)
db = connection["cozy-home"]
posts = db.posts

# Fill index with posts from DB
writer = index.writer()
for post in posts.find():
    writer.update_document(title=post["title"],
                           content=post["content"],
                           nid=unicode(post["_id"]),
                           tags=post["tags"])
writer.commit()

# Search inside index for post containing "test", then it displays
# results.
with index.searcher() as searcher:
    result = searcher.search(Term("content", u"test"))[0]
    post = posts.find_one(ObjectId(result["nid"]))
    print result["title"]
    print post["content"]

How to quickly start a single-page application with Node.js

A problem I experienced while starting coding with NodeJS environment and Express framework is that I had difficulties to quickly make a well structured app. I spent too much time on organizing my modules and writing helpers. Paradoxally with Express, I like the fact that I am free to do what I want and don’t encounter too much constraints due to the framework. At last, it was harder to design front end code than back end code.

To deal with that, I used two tools that fit with my requirements :

  • CompoundJS : a lightweight framework on top of express. It offers the structure and the vital functions I need to write a good  backend and configure express properly.
  • Brunch : an application assembler to organize and build cleanly my front-end code.
  • Edit: If you want to build a small single page-app, have a look at Americano a lightweight framework easy to learn (based on ExpressJS).

The bad thing with that choice is that there are no out of the box integration. So, in this article, I will give you a way to make them work together and obtain a well structured single page-app  in a minute (see the result here). I won’t cover in this article how code is organized, I can just tell you that it is a typical MVC structure for backend and a Backbone MVC for frontend. Look at their documentations for more.

1. Generate files

I assume you have already installed NodeJS. So install needed tools via npm, the node package manager :

npm install compound -g
npm install brunch -g

Then generate backend with railway:

compound init blog --coffee
cd blog
npm install # dependencies installation

With brunch generate frontend after removing frontend stuff from railway:

rm -rf public
brunch new client
cd client 
brunch build
cd ..

BackboneJS is the MVC framework configured by Brunch by default. If you want to work with AngularJS, you should run the project creation this way :

brunch new client --skeleton https://github.com/scotch/angular-brunch-seed

2. Plug Brunch on Railway

Then configure backend to handle brunch folder instead of old ones. In config/environment.coffee Replace this line :

app.use express.static(cwd + '/public', maxAge: 86400000)

with :

app.use express.static(cwd + '/client/public', maxAge: 86400000)

or for AngularJS case:

app.use express.static(cwd + '/client/_public', maxAge: 86400000)

In base template (index) app/views/layouts/application_layout.ejs , change stylesheet and javascript links :

<%- stylesheet_link_tag('bootstrap', 'style', 'bootstrap-responsive') %>

with

<%- stylesheet_link_tag('app') %>

and

<%- javascript_include_tag('http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js', 'bootstrap', 'rails', 'application') %>

with :

<%- javascript_include_tag('vendor', 'app') %>

For AngularJS cas use these lines:

<%- stylesheet_link_tag('css/app') %>
<%- javascript_include_tag('js/vendor', 'js/app') %>

Then clear body to replace it by :

<body>            
        <%- body %>
</body>

3. Make entry point and set a route to it

Create a controller to generate an entry point in app/controllers/application_controller.coffee:

action 'index', ->
     render title: "my single page app"

Then create template file for entry point:

mkdir app/views/application 
vim app/views/application/index.ejs

fill it with:

<script>require('initialize');</script>

Then add a route to confige/routes.coffee file:

exports.routes = (map) ->     
    map.get "/", "application#index"

Check that everything is fine by starting server and opening http://localhost:3000/ in your browser (Brunch logo should be displayed):

compound s

4. Use

To make Brunch automatically rebuild you app after each modification, go into the client directory and type:

brunch watch

You can write all your UI code inside client directory. Brunch handles templates too, so don’t worry about writing your html code on client side. Another good thing is that Compound and Brunch offer generators to build your models and controllers faster. Finally, Compound is very efficient for writing REST API : sending and parsing JSON are easy. As you understand, now you have all the stuff you need to build an awesome single-page web application!

Feel free to comment this tutorial and share any thoughts about this.

Latest Node.js frameworks trends

Hey, since my last blog post about latest Node.js trends, I get deeper in RailwayJS, a rails-like framework for Node.JS. Despite the lack of documentation, I definitely like it : it is lightweight and allow to start quickly a solid web application. The counterpart is that you have to follow the framework logic. Fortunately, it’s easy to learn. Moreover Railways offers scaffolding, so you can see the way it works. For the bad part, I noticed some bugs with the coffeescript mode (files are still generated in javascript), I hope that they will be corrected soon.

I also discovered Tower.js a complete framework dedicated to coffee script applications. Tower is inspired by Rails too and looks promising (lot of cool ideas) but for now it is a little bit the mess, too much features were added too quickly : documentation is incomplete and performances look poor (see my CPU charge grows of 6% for a single request). Despite of that, the overall design looks great and based on good tools, so after a little pause and a little cleaning it could really become a killer framework. One other good point is that it handles the client part too, so common code for backend and frontend should be easy to set up. Whatever, for now, I would recommend Railway over it, because it is easier to start with and developers seem to focus better on important features.

At last I will talk about Brunch, the Coffeescript client framework, which finally gets to 1.0. Now Brunch has no more weird bugs with watch command (automatic compilation after file modifications was broken) and seems to be faster. The overall project organization is better and dependencies are very well handled. Another good news is that you can use other MVC framework than Backbone: Ember and Batman are availables. As you understand, Brunch is still my favorite client side framework.

Little remark: until now upgrades of Brunch were painful. I hope with that last release, they will stabilize the directory organization and makes updates more seamless.

Finally, via Brunch I discovered Chaplin which is a proposal of BackboneJS app architecture. It has been shared after being built for a real case. It doesn’t look trivial, I hope I could get deeper, I can’t tell a lot for the moment. Whatever have a look at it if you need some inspirations !

Latest Javascript and Node.js trends

RailwayJS  is a nice framework built on the same principle as Rails. It is built on top of express JS : it looks easy to learn and provides a lot of helpers (localization, caching…) to build a solid web application. There is a small bad point : the ORM/ODM, called JugglingBD is weird because it claims to be able to manage both relational DB and non-relational DB.  The good fact is that it allows to do some validations on the model fields and have functions to set actions before and after an operation is done on models. Railways allows to do scaffolding, that means he prebuilt your resources and your routes supposing you will do CRUD operations on a given entity kind. Finally Railways looks like a good framework to start a web app with Node.js but don’t use it if you want to develop a small and lightweight application (there are a lot of unuseful features in this case).

Iced Coffee script is a language built on top coffee script. In fact it is the same language with some nice features added. As you know, Javascript relies a lot on callback/closures. So, notably with node.js, your code can become unreadable due to the number of nested callback. Iced allow to write asynchronous code in a synchronous way without breaking the asynchronous feature. Here is an example :

  await $.getJSON url, defer json
  cb json.results

instead of :

  $.getJSON url, (json) ->
    cb json.results

Iced brings other helpers to deal cleanly with simultaneous asynchronous calls.

Brunch framework is the must have to write browser client side of web applications. It offers all the tools needed to make good developments without dumb constraints. But beware, this great tool is not fully functional yet. I have already written about this subject but things look to progress well.

Mocha is a test framework which attracts me a lot. He has two qualities : he offers a very simple way to deal with asynchronous methods testing and it runs both in the browser and in node.js. Before that I used Jasmine for the browser and Vows for the backend. But Jasmine is only for the browser (or hard to make it works on the backend) and Vows is not easy to use at all. Another good point is that it will be integrated soon to Brunch. So if I should start a project now, I will definitely go for Mocha, even if it is not as mature as the one quoted above.

Ember.js isanother MVC framework for client side. I did not use it but it looks good and more complete than Backbone. It looks more like a true Model View Controller than Backbone. It offers notably better binding between views and models. Unfortunately it is not integrated into Brunch, so if I should start a new application, I won’t use it. But let’s keep an eye on it, it could become quickly the new JS MVC king.

Date and Node.js

Recently, I did some small developments on Ponyo that made me deal with dates. Unfornately I found a big bug in Node.js about them. By default when you create a date, it changes it and set it one hour before. When you try to modify it, it doesn’t change at all. So I decided to include node-time to deal with timezones and moment to have more human functions to manage dates. But the problem still occurs. There is no good reason for that, when you force a date, it should not be updated to current timezone.

> new Date(2012, 01, 03, 0, 0, 0, 0)
Thu, 02 Feb 2012 23:00:00 GMT

I finally find one solution, that I share here, hoping it will save time to people in the same case as me. To avoid this, the timezone must not be set on the date you work on but on the node process. In your “main” just add this at the beginning :

process.env.TZ = 'UTC'

Now you can use date like in any other programming environment.

Edit : Apparently it’s not a Node bug, it’s the way javascript works…

Web application development workflow with Node.js

Context

The aim of the article is to describe a good workflow to use when developing web applications. I always read advice on good workflow practices but they are never aggregated together, that’s why I do it right now.

UI Driven Development

We are going to follow a Behavior Driven Development style. Moreover we are making a web application, so we decide to focus on user: we will start by writing code for the UI. Then for UI and backend, we will write our specs/tests first.

Technos

As technology we are going to use :

  • Node.js and its framework Expess.JS because of their popularity and to get more familiar with it.
  • The language is Coffeescript for its readability.
  • Backbone is the UI framework.
  • Mongoose is the ODM.
  • Git is the version control system.
  • Jasmine is the BDD framework for UI and Vows is the BDD Framework for backend.

Use case

The application on which I will apply the workflow is Ponyo, a simple app that actually does nothing apart of allowing to create and browse “categories”. To illustrate the workflow we are going to add a new feature : allow category deletion.

Category list

Category display

Workflow

Here are all steps we need to do proper developments. Of course most of the time pressure push us to do shortcuts, but it is good to keep the good way in mind. Moreover this complete workflow could help to think about what could be optimized/automatized:

  1. Make a branch
  2. Write UI specs + commit
  3. Write UI specs code + commit
  4. Write UI code + commit
  5. Write backend resource specs + commit
  6. Write backend resource tests + commit
  7. Write backend resource code + commit
  8. Run all tests
  9. Test your app manually
  10. Rebase branch
  11. Merge branch + push
  12. Refactor if needed

Details

1. Make a branch

First of all, don’t bother other programmers right now with our commits, so let’s create a branch called feature-delete-category.

With Git

    git branch feature-delete-category
    git checkout feature-delete-category

2. Write UI Specs

We are doing UI Driven Development, so first write client side code specs. Then, check with your browser (here the URL is http://localhost:3000/tests/) that the Jasmine entries fail.

With Jasmine

describe 'Category deletion', ->
  it 'When I display newly created category', ->
      expect(false).toBeTruthy()
  it 'And I click on delete category button from a category page', ->
      expect(false).toBeTruthy()
  it 'Then it brings me back to category list', ->
      expect(false).toBeTruthy()
  it 'And deleted activity is no more in the list', ->
      expect(false).toBeTruthy()

With Git, commit

    git commit tests/categories.coffee -m "Add specs for category deletion."

3. Write UI specs code

Now that you know what you want to do, you can write corresponding tests and checks that they fail.

With Jasmine

describe 'Category deletion', ->
it 'When I display newly created category', ->
    runs ->
      $("#category-jasmine").click()
    waits(500) # Waits to be sure that everything is done before testing
it 'And I click on delete category button from a category page', ->
    runs ->
      $("#delete-category-button").click()
    waits(500) # Waits to be sure that everything is done before testing
it 'Then it brings me back to category list', ->
    runs ->
      expect($("#category-list").length).not.toEqual 0
it 'And deleted activity is no more in the list', ->
    runs ->
      expect($("#category-jamsine").length).toEqual 0

With Git, commit

    git commit tests/categories.coffee -m "Add tests for category deletion."

4. Write UI code

Now we are going to write the UI code, it is needed to know what we expect from server. We add a button to the template displaying a category, then we code the button behavior. After that we check that our tests still fail (backend does not support request for deletion).  Finally we commit.

Modify the template with Eco

<p>
  <a id="delete-category-button" href="#home">
    Delete category<br />
  </a>
</p>

Write behavior with Backbone

categoryViewTemplate = require('../templates/category_view')
Category = require('../models/category').Category

class exports.CategoryView extends Backbone.View
  id: 'category-view'

  constructor: ->
    super()

 render: (category) ->
   $("#nav-content").html null
   $.get "/categories/#{category}/", (data) =>
     $("#nav-content").html categoryViewTemplate(category: data)
     @model = new Category data
     @deleteButton = $("#delete-category-button")
     @deleteButton.click(@onDeleteButtonClicked)

 onDeleteButtonClicked : (event) =>
   event.preventDefault()
   @model.destroy
     success: ->
       app.routers.main.navigate(&quot;home&quot;, true)
     error: ->
       alert "An error occured, category was probably not deleted."
       app.routers.main.navigate("home", true)

With Git, commit

    git commit public/ -m "Add deletion button to UI"

5. Write backend resources specs

Now we know that we need  a resource to delete category, so let’s write our category deletion resource specs and commit.

With Vows

.addBatch
  'DELETE /categories/category-02/':
    topic: () ->
      apiTest.del 'categories/category-02/', @callback

    'response should be with a 200 OK': (error, response, body) ->
      assert.ok false
    'GET /categories/category-02/':
      topic: () ->
        apiTest.get 'categories/category-02/', @callback
      'response should be with a 404 Not Found': (error, response, body) ->
        assert.ok false
    'DELETE /categories/category-02/':
      topic: () ->
        apiTest.del 'categories/category-02/', @callback
      'response should be with a 404 Not Found': (error, response, body) ->
        assert.ok false

With Git

 git commit test/ -m "Add backend resources specs"

6. Write backend resource tests

Now we write our test code, we just check that returned HTTP code are expected ones and that once category is deleted, it cannot be reached anymore. We commit.

With Vows

.addBatch
  'DELETE /categories/category-02/':
    topic: () ->
      apiTest.del 'categories/category-02/', @callback

    'response should be with a 200 OK': assertStatus 200
    'GET /categories/category-02/':
      topic: () ->
        apiTest.get 'categories/category-02/', @callback
      'response should be with a 404 Not Found': assertStatus 404
    'DELETE /categories/category-02/':
      topic: () ->
        apiTest.del 'categories/category-02/', @callback
      'response should be with a 404 Not Found': assertStatus 404

With Git

 git commit test/ -m "Add backend resources specs code"

7. Write backend resource code

Now we write code : we add a new route that will link to a new resource dedicated to category deletion. We commit.

With Express

  app.del "/categories/:category/", routers.deleteCategory

With Express and Mongoose

exports.deleteCategory = (req, res) ->
  categoryProvider = new CategoryProvider

  categoryProvider.getCategory req.params.category, (err, docs) ->
    if err
      console.error(err.stack)
      res.json 'An error occured', 500
    else if docs.length > 0
      docs[0].remove (err) ->
        if err
          console.error(err.stack)
          res.json 'An error occured', 500
        else
          return res.json  success: true
    else
      res.json 'I dont have that', 404

With Git

 git commit test/ -m "Add category deletion resource"

8. Run all tests

We run our backend tests and our UI tests through browser and we are glad to see they all work.

With Vows

vows --spec test/resources.coffee

9. Test your app manually

Once you launch all your tests, test your application as a normal user. BDD is great but it will never replace a manual test, we often miss something that is not revealed by our tests.

Category deletion is possible


10. Rebase branch

We want to add our commits like we did them from last version of master branch, so we use rebase command.

With Git

git rebase master

11. Merge branch

Then we can merge our features to the master trunk. A push to master branch will validate that work is done !

With Git

git checkout master
git merge feature-delete-category
git push
git branch -d feature-delete-category

12. Refactor if needed

UI Driven development has the nice advantage to not let you develop unuseful resources but it does not let you think as good as possible the way to develop your backend. So you will probably need some refactoring. Fortunately, with your tests refactoring will be easier and safer. Moreover patterns you see when you develop UI first push you to think about refactoring that match better to your needs.

NB: Feel free to comment and criticize this article so I could improve it and correct what is wrong.