Clouds on the Horizon

In spring of last year (2009) we realized that the vast amount of data being collected by our internet server system was quickly becoming unwieldy. After all, every sensor value change in our commercial installations and homes (temperatures, instantaneous power fluctuations, etc.) ends up recorded in our server middleware database. Additionally, a large number of health and site operational statistics are constantly streaming into our system from sites all over the country (soon to be international). Although we are constantly tuning database indexes and stored procedures in order to insure that all critical queries and operations are performed successfully, a traditional relational database has fundamental limitations in cases such as this. We we were rapidly facing a decision point; either make some serious infrastructure investments in database hardware/software or look to public cloud computing services for managing our growing data storage needs.

Initially we looked at Google Big Table and Amazon S3 services for offloading storage of historical sensor data (our largest storage need). These services utilize state-of the-art, non-relational storage technologies that are very attractive in terms of cost and anticipated performance. We also reviewed Microsoft Azure services and once we began to understand all that was available as part of the Azure framework all other choices quickly fell off of the table. Not only is the Azure framework much more comprehensive in terms of service components and features, the integration with our primary software development tool, Visual Studio, made Azure a no-brainer for us.

In the summer of last year (2009) we began migrating our server-based application components to the Azure framework. We started with our internal system management web applications and then moved on to version 2 of our primary end-user application for our commercial product, EcoView Web. Late in the year we began experimenting with techniques and approaches for offloading our sensor historical data - a process that is continuing as I write this blog entry. The tricky bit here is to seamlessly integrate the Azure table storage technology into a running system that processes nearly 100k sensor update transactions per hour.

The eventual goal is to migrate our entire server application to Azure. Though I had originally projected that we would complete this task by the end of Q1 2010, it is now clear that this is going to take quite a bit more time. This is because we dont want to simply relocate our application. Rather, we need to fully leverage the core services and features of the Azure framework, an effort that involves learning some new approaches and technologies. It is already clear, however, that this move will be extremely beneficial in terms of our long term ability to adapt and extend our server processes and application features.

It is clear to me that the future is indeed full of clouds. Cloud computing frameworks, especially Microsoft Azure, promise to completely change the landscape of internet-based services. The ability to leverage robust computational and data storage services in a cost effective and incremental fashion will no doubt lead to a host of entirely new applications.

If you are interested in learning more about our move to the Microsoft Azure cloud computing framework, click the below link to read the Microsoft case study that was put together in November.

Microsft Case Study: Advanced Telemetry