Cloud computing is bringing cool software start-ups and talent to the infrastructure space again—a $50B+ market that had been hitting the snooze button for a while.

One of the hottest areas is services and programming libraries built on top of cloud infrastructure APIs. Infrastructure is cool again because the capital required for a hardware start-up is out of the reach of most angel investors and incubators—just try doing a hardware start-up on the funding from a typical Y-Combinator deal.  The programmatic codification of API controlled processing and storage resources; however, turns out to be pretty interesting proposition to small scale software driven ventures.

This wave of ambitious coder adventuring will only be further fueled by the coming arms race to the the largest possible developer ecosystem for each provider’s cloud API. But what if this won’t be like the last episode, and a single implementation (ex Red Hat) won’t gain dominant developer market-share? What if technologies to liberate the resources become the real power brokers?

Hello libcloud
The balance of cloud infrastructure power is open question—but one way to track the collision is watching a new open project called libcloud. Its ambition is simple—to mask the API differences of various cloud providers and create one unified Python library for controlling them all with the same syntax. The public face of the project is less than a month old, and you can get a good idea of their ambitions form this table:

Interestingly Eucalyptus makes their list despite being another API abstraction framework. While the scope of cloud interactions seems limited (list, reboot, create destroy) the breadth of providers is beyond most projects I’ve seen.

(Just to show how hot the space/idea is RedHat itself has a project with the same name.)

This stuff isn’t easy
Managing dependencies across eight providers isn’t easy, and it will be interesting to see how they engineer away massive maintenance cycles for the project. I’ve enjoyed the projects chatter on the right approach to abstraction:

I feel the goal of all this Interface and base class stuff
should be to get the point where adding a new provider doesn’t require any significant implementation details to be rewritten.
All these providers have REST-ful(ish) interfaces, a default parameter list that needs to be passed for each request, a default return format that you can parse in a predictable way, etc. All of these can be extracted into piece-meal methods that need to be defined by derivative classes. If it is done correctly, we won’t even need to use mock HTTP clients because it will just be a matter of,
“Okay, if I give `build_base_params` this stuff, it should set these
variables to this” and “If I feed this XML to `parse_response` it should return this dictionary (or whatever)”.

This makes the whole thing sound pretty simple—but anyone who has been through the UNIX wars remembers how even POSIX compliant OS’s still created lock-ins. It would be pretty amazing if somehow a consistent abstraction layer really wins out this time around—but I’ll stay tuned.

My guess is that by keeping the libraries’ scope simple they will be able to maintain the project. But what if you want to do something very specific only available on Amazon? What if that function is of great value—now you are no longer in your neat libCloud interface.

This constant tension between standards and differentiation never goes away does it? Later this week I’ll be hanging out with Eucalyptus, maybe they can show me the light and explain why this time it will all be different.

(BTW: follow the founder of libcloud for updates, he is @polvi, he is currently working on a Y combintator start-up called CloudKick as well)

@wattersjames