Thursday 7 June 2012

Raspberry Pi Project

You should all of heard of the Raspberry Pi by now, but in case you haven't head over to   http://www.raspberrypi.org/faqs for a preview. As a summary: A Raspberry Pi is a credit card sized  Linux-powered computer that costs between $25 and $35. That price tag makes it VERY interesting to me.

Although the Raspberry Pi is designed for educational purposes there is a growing community using them for projects ranging from home automation to robotics. Recently I have been writing a program that scrapes data from both websites and xml feeds, aggregates that data, and does some useful things with it... I ordered a Raspberry Pi about 6 weeks ago (due to arrive next month) and I intend to use it to run my web scraping program.

As my project has grown I have had to distribute the scraping work and much of the calculations across multiple AWS servers while keeping the database on a single server. The database server I will scale upwards onto larger hardware due to the high write to read ratio (yes more writes), but the scraping work I hope to distribute across a small farm of Raspberry Pi computers back home. If this works I should be able to massively cut down my AWS bill.

I am yet to find out whether the Raspberry Pi will be able to run my scraping program in just 256MB of RAM, but if all goes well I will be buying at least 10 of these little gems!

I'll post again when my Raspberry Pi arrives.

No comments:

Post a Comment