Linux Mint - Free and powerful

Tuesday, 28 May 2013

Apache Mesos: Dynamic Resource Sharing for Clusters

Apache Mesos: Dynamic Resource Sharing for Clusters:

'via Blog this'

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop,MPIHypertableSpark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator.
Mesos block diagram

You can use Mesos to:
  • Run Hadoop, MPI, Spark, and other frameworks on a dynamically shared pool of nodes.
  • Run multiple instances of Hadoop on the same cluster to isolate production and experimental jobs, or even multiple versions of Hadoop.
  • Run long-lived services (e.g. Hypertable and HBase) on the same nodes as batch applications and share resources between them.
  • Build new cluster computing frameworks without reinventing low-level facilities for farming out tasks, and have them coexist with existing ones.


  • Fault-tolerant replicated master using ZooKeeper.
  • Scalability to 10,000s of nodes using fast, event-driven C++ implementation.
  • Isolation between tasks with Linux Containers.
  • Multi-resource scheduling (memory and CPU aware).
  • Efficient application-controlled scheduling mechanism (resource offers) that lets frameworks achieve their own placement goals (e.g. data locality).
  • Java, Python and C++ APIs for developing new parallel applications.
  • Web UI for viewing cluster state.


Mesos is being used to manage clusters at TwitterAirBnbConvivaUC Berkeley, and UC San Francisco.

Research Project

Mesos started out as a research project in the UC Berkeley AMP Lab before being open sourced and joining the Apache Incubator. You can find more about the research behind Mesos, including our papers, on our research page.

Apache Incubator Notice

Apache Mesos is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.


Post a Comment

Thank you for taking the time to comment. Your opinion is important and of value and we appreciate the positive feedback! If you are "Negative Nancy" then please do us, and humanity, a favor, and piss off.

Total Pageviews

Google+ Followers


Blog Archive

Popular Posts

Recent Comments

Rays Twitter feed


Web sites come and go and information is lost and therefore some pages are archived. @rayd123. Powered by Blogger.