How To Set Up Mesos Cluster on Amazon EC2

Mesos is a cluster manager for various applications like Hadoop, spark, docker etc..You can manage clusters for any application using Mesos. To know more about the architecture and design of Mesos visit Mesos websites.

It is used by many big organizations like twitter for managing their fleet of servers in cluster mode.

Set Up Mesos Cluster on Amazon EC2

This guide will help you to set up a Mesos cluster with one master and multiple slaves. In this, I am using t2.micro instances for setting up Mesos master and slaves.

Setting Up Mesos Master

1. Update the server

apt-get -y update

2. Add the necessary repositories.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E56151BF

DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')

CODENAME=$(lsb_release -cs)

echo "deb http://repos.mesosphere.com/${DISTRO} ${CODENAME} main" | \
  sudo tee /etc/apt/sources.list.d/mesosphere.list

sudo apt-get -y update

3. Install oracle java8 as java 8 is a dependency for marathon.

sudo add-apt-repository ppa:webupd8team/java -y
sudo apt-get update
sudo apt-get install oracle-java8-installer

4. Install Mesos using the following command.

sudo apt-get -y install mesos marathon

5. Open /etc/zookeeper/conf/myid and set any number between 1-255. I am setting the master id as 1.

6. Open /etc/mesos/zk and replace localhost with the masters IP address.

For example,

zk://172.31.17.14:2181/mesos

7. Restart the master service using the following command.

sudo service mesos-master restart

8. Start the marathon service using the following command.

 sudo service marathon start

9. Now you can access the marathon and Mesos UI on the following URLs.

http://:8080
http://:5050

Setting Up Slave

1. add the required repositories shown in step 1 of the master setup.

2. Install Mesos

sudo apt-get -y install mesos

3. Edit /etc/mesos/zk file and replace localhost with the masters IP address.

3. Stop and disable zookeeper.

sudo service zookeeper stop
sudo sh -c "echo manual > /etc/init/zookeeper.override"

4. Now, Start the slave service.

sudo service mesos-slave restart

Now, if you navigate to slaves option in the Mesos dashboard, you will see a new entry for the slave you have configured.