Paradigm shift: How Apache Mesos is reshaping the data center into a giant PC
Most people couldn’t get much done on their computers without an operating system. Working the hardware directly would require the specialized knowledge and near-infinite patience to manually feed instructions to each processor on the motherboard in its specific language for every single action.
It’s an impossible proposition. Yet the infrastructure professionals who keep the world’s data centers humming along manage a comparable feat on an incomparably larger scale as a matter of daily routine. They determine workload requirements, allocate the necessary resources and perform all the other activates by hand that are usually executed automatically on a personal computer.
It doesn’t have to be that way, or at least not according to the backers of an emerging open-source project called Apache Mesos that promises to bridge the chasm between the consumer and enterprise worlds. The framework offers to provide an operating system for the data center.
The concept is a radical departure from the current reality of manual management that traces its roots back to the mainframe era. Mesos ofers a fresh approach that takes a page directly from world’s largest data center operators.
The platform was originally developed at UC Berkeley’s famed AMPLab, the birthplace of pioneering open-source technologies like Apache Spark and GraphX, with direct input from Google. The search giant was working on a new system to manage its data centers at the time and took the opportunity to exchange key lessons with the creators of Mesos.
After its release, the open-source framework went on to find a home at other web giants such as Twitter, Inc. and Airbag, which funneled the lessons they gleaned from running large-scale production implementations back into the upstream initiative. The end result is a platform that not only provides the conveniences of an operating system but acts like one, too.
Mesos elevates distributed workloads to the same level at which traditional single-tenant applications execute on a single computer. The complex details of the underlying hardware are hidden away under a programmatic interface that exposes – or more accurately, offers – resources for consumption on an as-needed basis.
Services, which are deployed in containers to provide the scale-out equivalent of process isolation, determine how much capacity to accept through a built-in scheduler that can be optimized for the specific requirements of the workload. That two-tier approach makes Mesos more flexible than other clustering technologies that make the choice independently of the application and puts it an entire world apart from how most organizations provision their infrastructure today.
Business processes are typically deployed in virtual machines that are complicated to manage and immensely difficult to re size, which leads to a situation in which there are almost always too few or too many resources allocated to an instance. Organizations didn’t have much choice but to accept that situation until now.
While Mesos addresses that trade-off, the operating system analogy doesn’t fully hold for a data center with thousands of computers. A container that ends up on a server with less bandwidth than the other nodes in a cluster, for instance, will inevitably create a bottleneck regardless of how many layers of separation there are between the hardware and the administrator. Some problems will always require moving down the stack, just as not every workload can be effectively accommodated in a fully abstracted environment.
Mesosphere, the startup that members of the original AMPLab team created to commercialize the project, has been working to fill the gaps with its commercial version of the platform. Unveiled in December, the distribution combines the framework with a number of other components from the open-source ecosystem to address key needs such as the ability to effectively scale services. But the question whether the enterprise is truly ready for a data center operating system still looms large.
Since you’re here …
… We’d like to tell you about our mission and how you can help us fulfill it. SiliconANGLE Media Inc.’s business model is based on the intrinsic value of the content, not advertising. Unlike many online publications, we don’t have a paywall or run banner advertising, because we want to keep our journalism open, without influence or the need to chase traffic.The journalism, reporting and commentary on SiliconANGLE — along with live, unscripted video from our Silicon Valley studio and globe-trotting video teams at theCUBE — take a lot of hard work, time and money. Keeping the quality high requires the support of sponsors who are aligned with our vision of ad-free journalism content.
If you like the reporting, video interviews and other ad-free content here, please take a moment to check out a sample of the video content supported by our sponsors, tweet your support, and keep coming back to SiliconANGLE.