Importing data from MongoHQ, and sending to MongoLab (as Heroku plugin)

Up to present date, I have not found a complete guide about it, so I am assembling it here, but please, have in mind that I am a MongoDB noob

What I want to do:
– Get all collection I have in MongoHQ, and backup locally
– Also, lets dump it into binary
– Get all collection of documents locally, and transmit to MongoLab (lets say, because they give 200MB of host free *Please see comments on more insight about this subject )

To do so, I first got to: http://www.mongodb.org/display/DOCS/Import+Export+Tools, but then, they warn it is easier to achieve by: http://www.mongodb.org/display/DOCS/Copy+Database+Commands command: db.copyDatabase

Step1, backup

On the folder associated to heroku (where you commit and push your app) run:
heroku config

focus on the key:
MONGOHQ_URL => mongodb://heroku:some_pass@flame.mongohq.com:27021/app9000

You can have more info about MongoHQ here http://devcenter.heroku.com/articles/mongohq

So, after figuring out this string, you just log on your local mongo console via: mongo

Now, you plan to backup to some db name, as ‘my_heroku_app_bckp’
run: db.copyDatabase( ‘app9000’, ‘my_heroku_app_bckp’, ‘flame.mongohq.com:27021/app9000’, ‘heroku’, ‘some_pass’ )

And you should get: { “ok” : 1 } ftw! If you get some #fail, refer to someone wiser than me, like www.StackOverflow.com :L

If all went right now you have you database replicated in your PC, and you can do whatever you want with it. For extra paranoid run: db.some_coll_you_know_has_stuff.count()

STEP 2, dump

Lets dump all data locally, using:
mongodump -o mongo_bckp -d my_heroku_app_bckp

That sould create a folder and the data inside, as bson.

STEP 3, migrate

Lets add MongoLab at heroku; follow this guide: http://devcenter.heroku.com/articles/mongolab

All right? Again run: heroku config

Now focus on:
MONGOLAB_URI => mongodb://heroku_app9000:other_pass@dbh22.mongolab.com:27227/heroku_app9000

Lets make a straigth import to them, using:
mongorestore -h dbh22.mongolab.com:27227 -d heroku_app9000 -u heroku_app9000 -p “mongo_bckp/my_heroku_app_bckp”

Enter the huge password when propted, and if everything ran smooth you will see many importations and no #fail

All is left to do is to switch the DB connection inside your app.

Now, commit cross your fingers! Check your app, all good? 🙂

* If you get a empty DB, check the database name

End Notes

MongoLabs and MongoHQ have different ways of counting and charging for data, in ways that can make a huge difference. For instance, for the same 6,582 doccuments under 1 collection, I may get 2.18Mb in one, while in the other I get 6.91 MB! (see comments)

Finally, by a very very very loose benchmark, their response has pretty much the same speed, being MongoLabs a tiny bit faster

Advertisements

4 thoughts on “Importing data from MongoHQ, and sending to MongoLab (as Heroku plugin)

  1. The reason the new size was smaller was that in performing the mongodump and subsequent mongorestore, the database is compacted. The file size and the size of the documents have not inherently changed, they just use less room. There is no strangeness in how MongoHQ is handling data sizes…it is just how MongoDB works.

    As a point of clarification, MongoHQ actually charges you based on the storageSize (data + indexes) attribute and MongoLab charges you based on the fileSize attribute (which is the total size of all the files, not your docs, allocated to the database. You may or may not be using these files and, as you may know, MongoDB plans ahead and allocates larger and larger files as you hit certain thresholds.

    To show how this can make a difference, I looked at one of our current customer’s databases and he had the following:

    storage size: 14,848 (so, 14KB)
    file size: 50331648 (so, 50MB)

    So, in that example, MongoHQ would say that you had 11K of storage used and MongoLab would say that you had 50MB of storage used! This is a big difference. This example is an easy way to see how they claim they are offering more storage space, but are actually using MongoDB’s filesize, which means that you may (or may not) be using that space and the database just needs to be compacted again.

    Jason
    MongoHQ

    • Thanks for clarification Jason!
      Made updates to the post, so readers, as well as myself, can have a better understanding of the subject.

      • Awesome … happy to help. All the different sizes of data in MongoDB can definitely be a bit confusing. It took me a while to get a full understanding of them as well! 🙂

        Cheers!

        Jason
        MongoHQ

  2. Hi Jason,
    In my app, MongoHQ seem a bit slow…
    Well, I launch my app beta yesterday, and are using the free plan on heroku…
    so… when I start pay, it will be faster? Or is it a MongoHQ limitation?

    Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s