CentOS 7, Systemd, Mesos, etc.

Quick instructions on how install CentOS 7, Mesos and Marathon from scratch.

Install CentOS 7

Disk partition (your choice of partitions)

/       200GB
/home   200GB
swap    32GB

Install with SOFTWARE SELECTION -> Infrastructure Server
- Options: (or your choice of other options)
DNS
Email
Hardware Monitoring Utilities
Java Platform
Network Filesystems Client
Performance Tools
Remote Management for Linux
Development Tools

Supply root password

Reboot after installation

Configure network bond

if you are concerned that a single switch failure might knock down a dozen of your servers

Select "Edit a connection"

Select "Add"

Select "Bond" and “Create”
In Device, type in bond name (for example “bond1”), and in BOND Slaves: add network interfaces

Continue with bond network configuration, IP address, netmask, etc.

2. Some commands to see configured network

ip -d addr

route -n

cat /proc/net/bonding/bond1

Other CentOS7 configurations

Load EPEL

wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-2.noarch.rpm

rpm -ivh epel-release-7-2.noarch.rpm

yum update

Install package groups

add below lines to /etc/yum.conf

# Setting to install every package from a group
group_package_types=default, mandatory, optional

Install Development Tools group

yum groupinstall -y "Development Tools"

Install other packages

yum install -y python-devel java-1.7.0-openjdk-devel maven zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 ntp whois

Allow 8192 open file limit for all

Increase system wide hard limit, and the number of open files in inotify watches

edit /etc/sysctl.conf

fs.file-max = 65535

fs.inotify.max_user_watches=65535

Increase soft limit

edit /etc/security/limits.conf

* hard nofile 8192

* soft nofile 8192

make effect sysctl -p

new CentOS 7 Firewall commands

View varies firewall settings

firewall-cmd --list-all

firewall-cmd --get-zones

firewall-cmd --zone=public –list-ports

Open a port to public

firewall-cmd --zone=public --add-port=5050/tcp --permanent

firewall-cmd --reload

SYSTEMD Explained (init services)

Migrate init services to Systemd

Run your services on old system in a normal way
Remember your main process ID, and come up with a service name you want to use

For example, your PID is 12234, you want to call it foobar

Download this program and run as a root:

migrate_to_systemd.py 12234 foobar

You will see 2 files created, foobar and foobar.service

manually verify and edit these 2 text files, as you see fit

there are plenty comments in these files

On new CentOS 7 system, copy foobar to /etc/sysconfig/foobar and foobar.service to /etc/systemd/system/foobar.service
Enable and start the foobar service on CentOS 7

systemctl enable foobar

systemctl start foobar

“systemctl” command

Show and Change system init level

systemctl set-default multi-user.target (default init level 3)

systemctl set-default graphical.target (level 5)

systemctl get-default

System commands
- systemctl poweroff|reboot
- systemctl rescue
  put into single user mode
- systemd-analyze blame
- systemctl list-dependencies
- systemctl list-dependencies memcached --all
  list all dependent services "memcached" has, “--all” is to display all levels

Set resource limits for a particular service

systemctllist-units

systemctl show ntpd.service |grep CPU

DropInPaths=/etc/systemd/system/ntpd.service.d/90-CPUShares.conf
LimitCPU=18446744073709551615
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUSchedulingResetOnFork=no
CPUAccounting=no
CPUShares=1024

systemctl set-property ntpd.serviceCPUShares=256

Other resource limitations:

LimitNOFILE=4096 MemoryLimit=18446744073709551615 MemoryAccounting=no

systemctl show ntpd.service |grep CPU

DropInPaths=/etc/systemd/system/ntpd.service.d/90-CPUShares.conf
LimitCPU=18446744073709551615
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUSchedulingResetOnFork=no
CPUAccounting=no
CPUShares=256

Start ntpd.service

systemctl enable ntpd.service

systemctl start ntpd.service

SYSTEMD service example

set up foobar.service to be started after ntpd.service and squid.service

Edit environment file: /etc/sysconfig/foobar
```
FOO1=12345
FOO2=98765
```

Edit service config file: /etc/systemd/system/foobar.service

[Unit]
Description=My FooBar
After=ntpd.service squid.service
Requires=ntpd.service squid.service

[Service]
#  the type can be:
#      forking – the startup program will fork a subprocess
#      normal  -- the startup program is a long running service, 
#                 so systemd will fork a subprocess for this service
#      oneshot – the service will run and exit
#
Type=oneshot
#   define environment variables used in startup program
EnvironmentFile=-/etc/sysconfig/foobar

UMask=0002

#  create a directory before starting the service
#     starting “-” means continue even this command failed
ExecStartPre=-/bin/mkdir /tmp/foobar_cache

#  command line variable ${FOO1} will use /etc/sysconfig/foobar
#    ${FOO1} and ${FOO2} are also visible inside /root/foobar.sh
ExecStart=/root/foobar.sh ${FOO1}
User=mylogin
Group=users

[Install]
WantedBy=multi-user.target

Enable and run foobar service

systemctl enable foobar.service

systemctl start foobar.service

Install Mesos

Download Mesos (check for newer version)

wget http://www.apache.org/dist/mesos/0.20.1/mesos-0.20.1.tar.gz

tar zxf mesos-0.20.1.tar.gz -C /usr/local

Build mesos

cd /usr/local/mesos-0.20.1

mkdir build

cd build

../configure

make

make check

This will build and run test programs, [failure in CpuIsolator] - check #ifdef SYSTEMD and invoke cgroups differently. [fixed in 0.21.0]

make install

Deployed mesos on /usr/local/bin/, /usr/local/lib/, /usr/local/include/mesos/, /usr/local/libexec/mesos/, etc.

Install and run zookeeper

wget http://mirror.cogentco.com/pub/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz

tar xzvf zookeeper-3.4.6.tar.gz -C /usr/local/

cd /usr/local/zookeeper-3.4.6/conf

cp zoo_sample.cfg zoo.cfg

Edit zoo.cfg

server.1=192.168.1.100:2888:3888

#server.2=192.168.1.101:2888:3888

Create file /tmp/zookeeper/myid with 1; directory is defined as dataDir in zoo.cfg

Start Zookeeper and test with CLI client

cd /usr/local/zookeeper-3.4.6

bin/zkServer.sh start

bin/zkCli.sh -server 192.168.1.100:2181

help

ls /

create /my_data my_test_data

ls /

get /my_data

set /my_data my_test_data2

get /my_data

delete /my_data

ls /

exit

Start Mesos master and slave

start master and slave

mesos-master --ip=192.168.1.100 --zk=zk://192.168.1.100:2181/mymesos --work_dir=/usr/lib/mesos –quorum=1

mesos-slave --master=zk://192.168.1.100:2181/mymesos --attributes="rack:2"

Verify

Go to http://192.168.1.100:5050/

run test framework

/usr/local/mesos-0.20.1/build/src/examples/java/test-framework 192.168.1.100:5050

Other important directory, files

/usr/local/var/mesos/deploy

mesos-master-env.sh, mesos-deploy-env.sh, mesos-slave-env.sh

/usr/local/sbin/

mesos-daemon.sh, mesos-master, mesos-slave, mesos-start-cluster.sh, mesos-start-masters.sh, mesos-start-slaves.sh, mesos-stop-cluster.sh, mesos-stop-masters.sh, mesos-stop-slaves.sh

Install and run Marathon

get the software

Download tarball: http://mesosphere.com/downloads/#marathon

tar xvfz marathon-0.7.0.tgz -C /usr/local/

cd /usr/local/marathon-0.7.0/

bin/build-distribution

Start Marathon

/usr/local/marathon-0.7.0/bin/start --master 192.168.1.100:5050 --zk_hosts localhost:2181 &

Verify

Send a JSON configuration to Marathon

Prepare a test JSON input file marathon.json

{
    "id": "/product/service/myapp",
    "cmd": "env && sleep 300",
    "cpus": 1.5,
    "mem": 256.0
}

Launch this app to Marathon

curl -X POST -d @marathon.json http://localhost:8080/v2/apps --header "Content-Type:application/json"

Check running statuses

Mesos Master console http://192.168.1.100:5050/

go to Master -> Active Tasks; note that the app got restarted repeatedly

Marathon console: http://192.168.1.100:8080/

From there, click the app and do scale, it will dynamically scale out