Continuous Delivery of a Simple Web Application Tutorial – Part 2

In Part 1 we gave the general outline of what we are trying to do, the tools we’re using, and the architecture of the application.

In this part (Part 2) we’re going to work on building the development environment with Ansible. This includes the Jenkins, Gitlab, a private Docker Registry, and a proxy server so we can point DNS to the site.

In Part 3 we configure the load balancers and the web site.

In Part 4 we configure Jenkins and put it all together.

The Ansible code can be found on Github in the Cisco Live 2015 repo.

Get the CoreOS Image

We’ll need an image to work with. While we can do this on the command line, its not something we’re going to repeat to often so I think we’re ok doing this the lame way and use the GUI.

A simple search takes us to the OpenStack page on the CoreOS site. I just used the current stable version. Its pretty simple. You follow their instructions:

$ wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
$ bunzip2 coreos_production_openstack_image.img.bz2

1 2	$ wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2 $ bunzip2 coreos_production_openstack_image.img.bz2

I downloaded this to some Linux server that was on the Internet. From there, I went into the Cisco OpenStack private Cloud dashboard and under images created a new one.

You can also do this through the command line just to make sure you’re still connecting correctly:

glance image-create --name CoreOS647.2.0 \
--container-format bare --disk-format qcow2 \
--file coreos_production_openstack_image.img \
--is-public True

glance image-create --name CoreOS647.2.0 \

--container-format bare --disk-format qcow2 \

--file coreos_production_openstack_image.img \

--is-public True

Ok, now we have a base image to work with. Let’s start automating this.

Ansible Base Setup

I’ve set up a directory called ~/Code/lawngnomed/ansible where all my Ansible configurations will live. I’ve spoken about setting up Ansible before so in this post we’ll just go over the things that are unique. The first item we need to do is setup our Development environment. Here’s the Ansible script for creating the development node, which I gave the hostname of ‘ci’:

- name: Ensure CI environment is ready
  connection: local
  hosts: local
  vars_files:
   - vars/metacloud_vars.yml
  tasks: 
  - name: Ensure CI environment is deployed
    nova_compute:
      state: present
      auth_url: "{{ lookup('env', 'OS_AUTH_URL') }}"
      login_username: "{{ lookup('env', 'OS_USERNAME') }}"
      login_password: "{{ lookup('env', 'OS_PASSWORD') }}"
      login_tenant_name: "{{ lookup('env', 'OS_TENANT_NAME') }}"
      name: ci
      image_name: "CoreOS 633.1.0"
      key_name: "{{ keypair }}"
      # 3 is m1.large
      flavor_id: 3
      meta: 
        group: ci
      security_groups: "{{ security_group }}"
      floating_ip_pools:
      - "{{ floating_ip_pool }}"
      user_data:  "{{ lookup('file', 'files/cloud-config.sh') }}"

- name: Make sure CI stuff is ready 
  hosts: ci
  roles:
    - registry
    - gitlab
    - jenkins
    - ci-proxy

- name: Ensure CI environment is ready

connection: local

hosts: local

vars_files:

- vars/metacloud_vars.yml

tasks:

- name: Ensure CI environment is deployed

nova_compute:

state: present

auth_url: "{{ lookup('env', 'OS_AUTH_URL') }}"

login_username: "{{ lookup('env', 'OS_USERNAME') }}"

login_password: "{{ lookup('env', 'OS_PASSWORD') }}"

login_tenant_name: "{{ lookup('env', 'OS_TENANT_NAME') }}"

name: ci

image_name: "CoreOS 633.1.0"

key_name: "{{ keypair }}"

# 3 is m1.large

flavor_id: 3

meta:

group: ci

security_groups: "{{ security_group }}"

floating_ip_pools:

- "{{ floating_ip_pool }}"

user_data: "{{ lookup('file', 'files/cloud-config.sh') }}"

- name: Make sure CI stuff is ready

hosts: ci

roles:

- registry

- gitlab

- jenkins

- ci-proxy

This playbook does the following:

Creates a new server called ‘ci’
1. ci will use a security group I already created
2. ci will use a key-pair I already created.
3. ci will use the cloud-config.sh script I created as part of the boot up.
Once the node is created creates the following roles on it: registry, gitlab, jenkins, and ci-proxy

The metacloud_vars.yml file contains most of the environment variables. Here is the file so you can see it. Replace this with your own:

---
keypair: tco-gold
# added port 22 to the security group
security_group: default,loadbalancer

# we use Coreos for this application
#coreos_image_id: bc3c7ad4-33d5-4702-a01a-b4b65b5c14a3
#coreos_image_name: "CoreOS 633.1.0"
coreos_image_name: "coreos-jenkins-slave-6331"

#image
m1large: 3

# this is the floating IP pool that we are able to get IPs from. 
floating_ip_pool: nova

---

keypair: tco-gold

# added port 22 to the security group

security_group: default,loadbalancer

# we use Coreos for this application

#coreos_image_id: bc3c7ad4-33d5-4702-a01a-b4b65b5c14a3

#coreos_image_name: "CoreOS 633.1.0"

coreos_image_name: "coreos-jenkins-slave-6331"

#image

m1large: 3

# this is the floating IP pool that we are able to get IPs from.

floating_ip_pool: nova

You can see I used a few images as I tried this out and eventually settled on using the same coreos image that my jenkins slaves run on. We’ll get to that soon.

You’ll need to create a security group so that all the services can be access. My security group looked as follows:

The other security group allows port 80 and 22 so I can ssh and go to the web browser.

The next important file is the files/cloud-config.sh script. With the script I needed to accomplish 2 things:

Get Java on the instance so that Jenkins could communicate with it.
Get Python on the instance so Ansible could run on it.
Make it so docker would be able to communicate with an insecure registry.

CoreOS by itself tries to be as bare as it gets so after trolling the Internet for a few days I finally cobbled this script together that would do the job.

#!/bin/bash

## Part 1:  Set up python
PYPY_VERSION=2.4.0
HOME=/home/core
mkdir -p $HOME
cd $HOME
block-until-url https://bitbucket.org/pypy/pypy/downloads/pypy-$PYPY_VERSION-linux64.tar.bz2
#
wget -O - https://bitbucket.org/pypy/pypy/downloads/pypy-$PYPY_VERSION-linux64.tar.bz2 |tar -xjf -
mv -n pypy-$PYPY_VERSION-linux64 pypy
#
## library fixup
mkdir -p pypy/lib
ln -snf /lib64/libncurses.so.5.9 $HOME/pypy/lib/libtinfo.so.5

mkdir -p $HOME/bin
#
cat > $HOME/bin/python <<EOF
#!/bin/bash
LD_LIBRARY_PATH=$HOME/pypy/lib:$LD_LIBRARY_PATH exec $HOME/pypy/bin/pypy "\$@"
EOF
#
chmod +x $HOME/bin/python
$HOME/bin/python --version

## part 2: insecure docker registry
mkdir -p /etc/systemd/system/
cat > /etc/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target
Requires=docker.socket early-docker.target

[Service]
Environment=TMPDIR=/var/tmp
EnvironmentFile=-/run/flannel_docker_opts.env
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// --insecure-registry ci:5000 $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl restart docker

#!/bin/bash

## Part 1: Set up python

PYPY_VERSION=2.4.0

HOME=/home/core

mkdir -p $HOME

cd $HOME

block-until-url https://bitbucket.org/pypy/pypy/downloads/pypy-$PYPY_VERSION-linux64.tar.bz2

wget -O - https://bitbucket.org/pypy/pypy/downloads/pypy-$PYPY_VERSION-linux64.tar.bz2 |tar -xjf -

mv -n pypy-$PYPY_VERSION-linux64 pypy

## library fixup

mkdir -p pypy/lib

ln -snf /lib64/libncurses.so.5.9 $HOME/pypy/lib/libtinfo.so.5

mkdir -p $HOME/bin

cat > $HOME/bin/python <<EOF

#!/bin/bash

LD_LIBRARY_PATH=$HOME/pypy/lib:$LD_LIBRARY_PATH exec $HOME/pypy/bin/pypy "\$@"

EOF

chmod +x $HOME/bin/python

$HOME/bin/python --version

## part 2: insecure docker registry

mkdir -p /etc/systemd/system/

cat > /etc/systemd/system/docker.service <<EOF

[Unit]

Description=Docker Application Container Engine

Documentation=http://docs.docker.com

After=docker.socket early-docker.target network.target

Requires=docker.socket early-docker.target

[Service]

Environment=TMPDIR=/var/tmp

EnvironmentFile=-/run/flannel_docker_opts.env

MountFlags=slave

LimitNOFILE=1048576

LimitNPROC=1048576

ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// --insecure-registry ci:5000 $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl restart docker

Roles

A few directories with files were created:

roles
|
|-ci-proxy
|  |
|  |- files/default.conf
|  |- tasks/main.yml
|
|-gitlab
|  |
|  |- vars/main.yml
|  |- tasks/main.yml
|
|- jenkins
|  |- tasks/main.yml
|
|- registry
   |- tasks/main.yml

roles

|-ci-proxy

| |

| |- files/default.conf

| |- tasks/main.yml

|-gitlab

| |

| |- vars/main.yml

| |- tasks/main.yml

|- jenkins

| |- tasks/main.yml

|- registry

|- tasks/main.yml

Let’s go through each task:

ci-proxy

This role is creates a docker container that acts as the reverse proxy. So that when requests like http://jenkins.lawngnomed.com come in, the proxy redirects the request to the right container.

The task file below copies the nginx configuration file and then mounts it into the container. Then it runs the container.

---

- name: Ensure nginx files are copied. 
  copy: src={{item.src}} dest={{item.dest}}
  with_items:
    - { src: ../files/default.conf, dest: /vol/nginx/ }
  sudo: true

- name: Ensure NGINX proxy is up
  docker: image="nginx" name=proxy volumes="/vol/nginx:/etc/nginx/conf.d" ports=80:80,443:443

---

- name: Ensure nginx files are copied.

copy: src={{item.src}} dest={{item.dest}}

with_items:

- { src: ../files/default.conf, dest: /vol/nginx/ }

sudo: true

- name: Ensure NGINX proxy is up

docker: image="nginx" name=proxy volumes="/vol/nginx:/etc/nginx/conf.d" ports=80:80,443:443

The contents of the nginx default config file that will run in /etc/nginx/conf.d/default.conf is the following:

server {
  server_name jenkins.lawngnomed.com;
  location / { 
    proxy_pass http://ci:8080/;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout   150;
    proxy_send_timeout      100;
    proxy_read_timeout      100;
    proxy_buffers           4 32k;
    client_max_body_size    8m; 
    client_body_buffer_size 128k;
  }
}

server {
  listen 80; 
  server_name gitlab.lawngnomed.com www.gitlab.lawngnomed.com;
  location / { 
    proxy_pass http://ci:10080;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout   150;
    proxy_send_timeout      100;
    proxy_read_timeout      100;
    proxy_buffers           4 32k;
    client_max_body_size    8m; 
    client_body_buffer_size 128k;
  }
}

server {
  server_name registry.lawngnomed.com;
  location / { 
    proxy_pass http://ci:5000;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout   150;
    proxy_send_timeout      100;
    proxy_read_timeout      100;
    proxy_buffers           4 32k;
    client_max_body_size    8m; 
    client_body_buffer_size 128k;
  }
}

server {

server_name jenkins.lawngnomed.com;

location / {

proxy_pass http://ci:8080/;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_connect_timeout 150;

proxy_send_timeout 100;

proxy_read_timeout 100;

proxy_buffers 4 32k;

client_max_body_size 8m;

client_body_buffer_size 128k;

}

server {

listen 80;

server_name gitlab.lawngnomed.com www.gitlab.lawngnomed.com;

location / {

proxy_pass http://ci:10080;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_connect_timeout 150;

proxy_send_timeout 100;

proxy_read_timeout 100;

proxy_buffers 4 32k;

client_max_body_size 8m;

client_body_buffer_size 128k;

}

server {

server_name registry.lawngnomed.com;

location / {

proxy_pass http://ci:5000;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_connect_timeout 150;

proxy_send_timeout 100;

proxy_read_timeout 100;

proxy_buffers 4 32k;

client_max_body_size 8m;

client_body_buffer_size 128k;

}

There could be some issues with this file, but it seems to work. There are occasions when jenkins and gitlab redirect to bad urls, but everything works with this configuration. I’m open to any ideas to changing it.

Once this role is up you can access the URL from the outside.

Gitlab

Gitlab requires a Redis container for key value store and a PostGrsql database. We use Docker for both of these and link them together. The Ansible playbook file looks as follows:

---
- name: Ensure Redis is up for Gitlab
  docker: image="sameersbn/redis:latest" volumes=/vol/redis/data:/var/lib/redis name=redis

- name: Ensure PostGresql is up
  docker: image="sameersbn/postgresql:latest" volumes=/vol/postgresql/data:/var/lib/postgresql env="DB_NAME=gitlabhq_production,DB_USER=gitlab,DB_PASS={{gitlab_db_password}}" name=postgresql
  register: result

- name: Wait for a few seconds if the postrgres just came up...
  pause: seconds=5
  when: result|changed

- name: Ensure Gitlab is up
  docker: image="sameersbn/gitlab:latest" volumes=/vol/gitlab/data:/home/git/data env="DB_TYPE=postgres,GITLAB_PORT=10080,GITLAB_SSH_PORT=10022" links="postgresql:postgresql,redis:redisio" ports=10022:22,10080:80 name=gitlab

---

- name: Ensure Redis is up for Gitlab

docker: image="sameersbn/redis:latest" volumes=/vol/redis/data:/var/lib/redis name=redis

- name: Ensure PostGresql is up

docker: image="sameersbn/postgresql:latest" volumes=/vol/postgresql/data:/var/lib/postgresql env="DB_NAME=gitlabhq_production,DB_USER=gitlab,DB_PASS={{gitlab_db_password}}" name=postgresql

- name: Wait for a few seconds if the postrgres just came up...

pause: seconds=5

when: result|changed

- name: Ensure Gitlab is up

docker: image="sameersbn/gitlab:latest" volumes=/vol/gitlab/data:/home/git/data env="DB_TYPE=postgres,GITLAB_PORT=10080,GITLAB_SSH_PORT=10022" links="postgresql:postgresql,redis:redisio" ports=10022:22,10080:80 name=gitlab

Notice that the gitlab_db_password is an environment variable created in the ../var/main.yml file. I set this up and then encrypted the file using Ansible Vault. See my post on how that is accomplished because its a pretty cool technique I learned from our Portland Ansible Users Group.

Jenkins

The Jenkins Ansible installation script is pretty straight forward. The only catch is to make sure the directory owner is jenkins and that you mount the directory.

---
- name: Make sure file is in place
  file: path=/vol/jenkins_home owner=1000 state=directory
  sudo: yes 

- name: Ensure Jenkins is up
  docker: image="jenkins" volumes=/vol/jenkins_home:/var/jenkins_home name=jenkins ports=8080:8080,50000:50000

---

- name: Make sure file is in place

file: path=/vol/jenkins_home owner=1000 state=directory

sudo: yes

- name: Ensure Jenkins is up

docker: image="jenkins" volumes=/vol/jenkins_home:/var/jenkins_home name=jenkins ports=8080:8080,50000:50000

Registry

No tricks, here, we’re just using the latest from the docker registry. This goes out and pulls the registry.

---
- name: Ensure Docker Registry is up.
  docker: image="registry:latest" volumes=/vol/docker-registry:/docker \
env="STORAGE=local,STORAGE_PATH=/docker" ports=5000:5000 name=registry

---

- name: Ensure Docker Registry is up.

docker: image="registry:latest" volumes=/vol/docker-registry:/docker \

env="STORAGE=local,STORAGE_PATH=/docker" ports=5000:5000 name=registry

Loose Ends

There are a few parts that I didn’t automate that should be done.

The instance I created mounts a persistent storage device that I created in Metacloud. There are two pieces missing:
1. It doesn’t create the volume in OpenStack if its not there yet.
2. It doesn’t mount the volume onto the development server.
For speed, its better to pull the docker containers from the local registry. So technically we should tag all the images that we’re using and put them in the local registry. This is a chicken and an egg problem because you need the registry up before you can download images from it. So I left it that way.
There are still some things I needed to finish like putting some keys and other items I needed for Jenkins in the /vol directory. Its not perfect but its pretty good.

Creating the volume and mounting was pretty quick once the image was up. First I created the volume and assigned it using the Horizon Dashboard that Metacloud provides.

This was just a 20GB volume. Once the instance was up I ran a few commands like:

fdisk -l # walk through menu create a new fs, accept defaults: e.g: n,enter,enter,m to write it out. 
mkfs.ext2 /dev/vdb1
mkdir -p /vol
mount /dev/vdb1 /vol

fdisk -l # walk through menu create a new fs, accept defaults: e.g: n,enter,enter,m to write it out.

mkfs.ext2 /dev/vdb1

mkdir -p /vol

mount /dev/vdb1 /vol

This way all of our information persists if the containers terminate and if the instances terminate.

Finishing up the Development Server

Once you get to this point, you should be able to bring it all up with:

ansible-playbook ci.yml

1	ansible-playbook ci.yml

That should do it! Once you are in you may want to tag all of your images so that they load in the local docker registry. For example, once you log in you could run:

core@ci ~ $ docker tag registry ci:5000/registry
core@ci ~ $ docker push ci:5000/registry

1 2	core@ci ~ $ docker tag registry ci:5000/registry core@ci ~ $ docker push ci:5000/registry

At this point the idea is that you should be able to go to whatever public IP address was assigned to you and be able to access:

jenkins:  <IP>:8080
gitlab: <IP>:10080
registry: <IP>:5000

jenkins: <IP>:8080

gitlab: <IP>:10080

registry: <IP>:5000

If you’re there then you can get rolling to the next step: Ansible scripts to deploy the rest of the environment.

In Part 3 we’ll cover Ansible for bringing up the load balancers and web servers. We’ll also snapshot an image to make it a jenkins slave.