{"id":3330,"date":"2015-06-05T14:50:15","date_gmt":"2015-06-05T20:50:15","guid":{"rendered":"http:\/\/benincosa.com\/?p=3330"},"modified":"2015-06-15T11:53:52","modified_gmt":"2015-06-15T17:53:52","slug":"continuous-delivery-of-a-simple-web-application-tutorial-part-2","status":"publish","type":"post","link":"https:\/\/benincosa.com\/?p=3330","title":{"rendered":"Continuous Delivery of a Simple Web Application Tutorial – Part 2"},"content":{"rendered":"

In Part 1<\/a> we gave the general outline of what we are trying to do, the tools we’re using, and the architecture of the application.<\/p>\n

In this part (Part 2) we’re going to work on building the development environment\u00a0with Ansible. \u00a0This includes the Jenkins, Gitlab, a private Docker Registry, and a proxy server so we can point DNS to the site.<\/p>\n

In Part 3<\/a> we configure the load balancers and the web site.<\/p>\n

In Part 4<\/a> we configure Jenkins and put it all together.<\/p>\n

The Ansible code can be found on Github in the Cisco Live 2015<\/a> repo.<\/p>\n

Get the CoreOS Image<\/h3>\n

We’ll need an image to work with. \u00a0While we can do this on the command line, its not something we’re going to repeat to often so I think we’re ok doing this the lame way and use the GUI.<\/p>\n

A simple search takes us to the OpenStack page on the CoreOS<\/a> site. \u00a0I just used the current stable version. \u00a0Its pretty simple. \u00a0You follow their instructions:<\/p>\n

$ wget http:\/\/stable.release.core-os.net\/amd64-usr\/current\/coreos_production_openstack_image.img.bz2\r\n$ bunzip2 coreos_production_openstack_image.img.bz2<\/pre>\n
I downloaded this to some Linux server that was on the Internet. \u00a0From there, I went into the Cisco OpenStack private Cloud dashboard and under images created a new one.<\/p>\n
\"Screen<\/a><\/p>\n
You can also do this through the command line just to make sure you’re still connecting correctly:<\/p>\n
glance image-create --name CoreOS647.2.0 \\\r\n--container-format bare --disk-format qcow2 \\\r\n--file coreos_production_openstack_image.img \\\r\n--is-public True<\/pre>\n
Ok, now we have a base image to work with. \u00a0Let’s start automating this.<\/p>\n
Ansible Base Setup<\/h3>\n
\"\"
Ansible<\/figcaption><\/figure>\n
I’ve set up a directory called ~\/Code\/lawngnomed\/ansible where all my Ansible configurations will live. \u00a0I’ve spoken about setting up Ansible before<\/a> so in this post we’ll just go over the things that are unique. \u00a0The first item we need to do is setup our Development environment. \u00a0Here’s the Ansible script for creating the development node, which I gave the hostname of ‘ci’:<\/p>\n
- name: Ensure CI environment is ready\r\n  connection: local\r\n  hosts: local\r\n  vars_files:\r\n   - vars\/metacloud_vars.yml\r\n  tasks: \r\n  - name: Ensure CI environment is deployed\r\n    nova_compute:\r\n      state: present\r\n      auth_url: \"{{ lookup('env', 'OS_AUTH_URL') }}\"\r\n      login_username: \"{{ lookup('env', 'OS_USERNAME') }}\"\r\n      login_password: \"{{ lookup('env', 'OS_PASSWORD') }}\"\r\n      login_tenant_name: \"{{ lookup('env', 'OS_TENANT_NAME') }}\"\r\n      name: ci\r\n      image_name: \"CoreOS 633.1.0\"\r\n      key_name: \"{{ keypair }}\"\r\n      # 3 is m1.large\r\n      flavor_id: 3\r\n      meta: \r\n        group: ci\r\n      security_groups: \"{{ security_group }}\"\r\n      floating_ip_pools:\r\n      - \"{{ floating_ip_pool }}\"\r\n      user_data:  \"{{ lookup('file', 'files\/cloud-config.sh') }}\"\r\n\r\n- name: Make sure CI stuff is ready \r\n  hosts: ci\r\n  roles:\r\n    - registry\r\n    - gitlab\r\n    - jenkins\r\n    - ci-proxy<\/pre>\n
This playbook does the following:<\/p>\n
    \n
  1. Creates a new server called ‘ci’\n
      \n
    1. ci will use a security group I already created<\/li>\n
    2. ci will use a key-pair I already created.<\/li>\n
    3. ci will use the cloud-config.sh script I created as part of the boot up.<\/li>\n<\/ol>\n<\/li>\n
    4. Once the node is created creates the following roles on it: registry, gitlab, jenkins, and ci-proxy<\/li>\n<\/ol>\n
      The metacloud_vars.yml file contains most of the environment variables. \u00a0Here is the file so you can see it. \u00a0Replace this with your own:<\/p>\n
      ---\r\nkeypair: tco-gold\r\n# added port 22 to the security group\r\nsecurity_group: default,loadbalancer\r\n\r\n# we use Coreos for this application\r\n#coreos_image_id: bc3c7ad4-33d5-4702-a01a-b4b65b5c14a3\r\n#coreos_image_name: \"CoreOS 633.1.0\"\r\ncoreos_image_name: \"coreos-jenkins-slave-6331\"\r\n\r\n#image\r\nm1large: 3\r\n\r\n# this is the floating IP pool that we are able to get IPs from. \r\nfloating_ip_pool: nova<\/pre>\n
      You can see I used a few images as I tried this out and eventually settled on using the same coreos image that my jenkins slaves run on. \u00a0We’ll get to that soon.<\/p>\n
      You’ll need to create a security group so that all the services can be access. \u00a0My security group looked as follows:<\/p>\n
      \"Screen<\/a><\/p>\n
      The other security group allows port 80 and 22 so I can ssh and go to the web browser.<\/p>\n
      The next important file is the files\/cloud-config.sh script. \u00a0With the script I needed to accomplish 2 things:<\/p>\n
        \n
      1. Get Java on the instance so that Jenkins could communicate with it.<\/li>\n
      2. Get Python on the instance so Ansible could run on it.<\/li>\n
      3. Make it so docker would be able to communicate with an insecure registry.<\/li>\n<\/ol>\n
        CoreOS by itself tries to be as bare as it gets so after trolling the Internet for a few days I finally cobbled this script together that would do the job.<\/p>\n
        #!\/bin\/bash\r\n\r\n## Part 1:  Set up python\r\nPYPY_VERSION=2.4.0\r\nHOME=\/home\/core\r\nmkdir -p $HOME\r\ncd $HOME\r\nblock-until-url https:\/\/bitbucket.org\/pypy\/pypy\/downloads\/pypy-$PYPY_VERSION-linux64.tar.bz2\r\n#\r\nwget -O - https:\/\/bitbucket.org\/pypy\/pypy\/downloads\/pypy-$PYPY_VERSION-linux64.tar.bz2 |tar -xjf -\r\nmv -n pypy-$PYPY_VERSION-linux64 pypy\r\n#\r\n## library fixup\r\nmkdir -p pypy\/lib\r\nln -snf \/lib64\/libncurses.so.5.9 $HOME\/pypy\/lib\/libtinfo.so.5\r\n\r\nmkdir -p $HOME\/bin\r\n#\r\ncat > $HOME\/bin\/python <<EOF\r\n#!\/bin\/bash\r\nLD_LIBRARY_PATH=$HOME\/pypy\/lib:$LD_LIBRARY_PATH exec $HOME\/pypy\/bin\/pypy \"\\$@\"\r\nEOF\r\n#\r\nchmod +x $HOME\/bin\/python\r\n$HOME\/bin\/python --version\r\n\r\n## part 2: insecure docker registry\r\nmkdir -p \/etc\/systemd\/system\/\r\ncat > \/etc\/systemd\/system\/docker.service <<EOF\r\n[Unit]\r\nDescription=Docker Application Container Engine\r\nDocumentation=http:\/\/docs.docker.com\r\nAfter=docker.socket early-docker.target network.target\r\nRequires=docker.socket early-docker.target\r\n\r\n[Service]\r\nEnvironment=TMPDIR=\/var\/tmp\r\nEnvironmentFile=-\/run\/flannel_docker_opts.env\r\nMountFlags=slave\r\nLimitNOFILE=1048576\r\nLimitNPROC=1048576\r\nExecStart=\/usr\/lib\/coreos\/dockerd --daemon --host=fd:\/\/ --insecure-registry ci:5000 $DOCKER_OPTS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\nEOF\r\n\r\nsystemctl daemon-reload\r\nsystemctl restart docker<\/pre>\n
        Roles<\/h4>\n
        A few directories with files were created:<\/p>\n
        roles\r\n|\r\n|-ci-proxy\r\n|  |\r\n|  |- files\/default.conf\r\n|  |- tasks\/main.yml\r\n|\r\n|-gitlab\r\n|  |\r\n|  |- vars\/main.yml\r\n|  |- tasks\/main.yml\r\n|\r\n|- jenkins\r\n|  |- tasks\/main.yml\r\n|\r\n|- registry\r\n   |- tasks\/main.yml\r\n\r\n<\/pre>\n
        Let’s go through each task:<\/p>\n
        ci-proxy<\/strong><\/p>\n
        This role is creates a docker container that acts as the reverse proxy. \u00a0So that when requests like http:\/\/jenkins.lawngnomed.com come in, the proxy redirects the request to the right container.<\/p>\n
        The task file below copies the nginx configuration file and then mounts it into the container. \u00a0Then it runs the container.<\/p>\n
        ---\r\n\r\n- name: Ensure nginx files are copied. \r\n  copy: src={{item.src}} dest={{item.dest}}\r\n  with_items:\r\n    - { src: ..\/files\/default.conf, dest: \/vol\/nginx\/ }\r\n  sudo: true\r\n\r\n- name: Ensure NGINX proxy is up\r\n  docker: image=\"nginx\" name=proxy volumes=\"\/vol\/nginx:\/etc\/nginx\/conf.d\" ports=80:80,443:443<\/pre>\n
        The contents of the nginx default config file that will run in \/etc\/nginx\/conf.d\/default.conf is the following:<\/p>\n
        server {\r\n  server_name jenkins.lawngnomed.com;\r\n  location \/ { \r\n    proxy_pass http:\/\/ci:8080\/;\r\n    proxy_set_header        Host $host;\r\n    proxy_set_header        X-Real-IP $remote_addr;\r\n    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;\r\n    proxy_connect_timeout   150;\r\n    proxy_send_timeout      100;\r\n    proxy_read_timeout      100;\r\n    proxy_buffers           4 32k;\r\n    client_max_body_size    8m; \r\n    client_body_buffer_size 128k;\r\n  }\r\n}\r\n\r\nserver {\r\n  listen 80; \r\n  server_name gitlab.lawngnomed.com www.gitlab.lawngnomed.com;\r\n  location \/ { \r\n    proxy_pass http:\/\/ci:10080;\r\n    proxy_set_header        Host $host;\r\n    proxy_set_header        X-Real-IP $remote_addr;\r\n    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;\r\n    proxy_connect_timeout   150;\r\n    proxy_send_timeout      100;\r\n    proxy_read_timeout      100;\r\n    proxy_buffers           4 32k;\r\n    client_max_body_size    8m; \r\n    client_body_buffer_size 128k;\r\n  }\r\n}\r\n\r\nserver {\r\n  server_name registry.lawngnomed.com;\r\n  location \/ { \r\n    proxy_pass http:\/\/ci:5000;\r\n    proxy_set_header        Host $host;\r\n    proxy_set_header        X-Real-IP $remote_addr;\r\n    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;\r\n    proxy_connect_timeout   150;\r\n    proxy_send_timeout      100;\r\n    proxy_read_timeout      100;\r\n    proxy_buffers           4 32k;\r\n    client_max_body_size    8m; \r\n    client_body_buffer_size 128k;\r\n  }\r\n}<\/pre>\n
        There could be some issues with this file, but it seems to work. \u00a0There are occasions when jenkins and gitlab redirect to bad urls, but everything works with this configuration. \u00a0I’m open to any ideas to changing it.<\/p>\n
        Once this role is up you can access the URL from the outside.<\/p>\n
        Gitlab<\/strong><\/p>\n
        Gitlab requires a Redis container for key value store and a PostGrsql database. \u00a0We use Docker for both of these and link them together. \u00a0The Ansible playbook file looks as follows:<\/p>\n
        ---\r\n- name: Ensure Redis is up for Gitlab\r\n  docker: image=\"sameersbn\/redis:latest\" volumes=\/vol\/redis\/data:\/var\/lib\/redis name=redis\r\n\r\n- name: Ensure PostGresql is up\r\n  docker: image=\"sameersbn\/postgresql:latest\" volumes=\/vol\/postgresql\/data:\/var\/lib\/postgresql env=\"DB_NAME=gitlabhq_production,DB_USER=gitlab,DB_PASS={{gitlab_db_password}}\" name=postgresql\r\n  register: result\r\n\r\n- name: Wait for a few seconds if the postrgres just came up...\r\n  pause: seconds=5\r\n  when: result|changed\r\n\r\n- name: Ensure Gitlab is up\r\n  docker: image=\"sameersbn\/gitlab:latest\" volumes=\/vol\/gitlab\/data:\/home\/git\/data env=\"DB_TYPE=postgres,GITLAB_PORT=10080,GITLAB_SSH_PORT=10022\" links=\"postgresql:postgresql,redis:redisio\" ports=10022:22,10080:80 name=gitlab<\/pre>\n
        Notice that the gitlab_db_password is an environment variable created in the ..\/var\/main.yml file. \u00a0I set this up and then encrypted the file using Ansible Vault. \u00a0See my post <\/a>on how that is accomplished because its a pretty cool technique I learned from our Portland Ansible Users Group.<\/p>\n
        Jenkins<\/strong><\/p>\n
        The Jenkins Ansible installation script is pretty straight forward. \u00a0The only catch is to make sure the directory owner is jenkins and that you mount the directory.<\/p>\n
        ---\r\n- name: Make sure file is in place\r\n  file: path=\/vol\/jenkins_home owner=1000 state=directory\r\n  sudo: yes \r\n\r\n- name: Ensure Jenkins is up\r\n  docker: image=\"jenkins\" volumes=\/vol\/jenkins_home:\/var\/jenkins_home name=jenkins ports=8080:8080,50000:50000<\/pre>\n
        Registry<\/b><\/p>\n
        No tricks, here, we’re just using the latest from the docker registry. \u00a0This goes out and pulls the registry.<\/p>\n
        ---\r\n- name: Ensure Docker Registry is up.\r\n  docker: image=\"registry:latest\" volumes=\/vol\/docker-registry:\/docker \\\r\nenv=\"STORAGE=local,STORAGE_PATH=\/docker\" ports=5000:5000 name=registry<\/pre>\n
        Loose Ends<\/h3>\n
        There are a few parts that I didn’t automate that should be done.<\/p>\n
          \n
        1. The instance I created mounts a persistent storage device that I created in Metacloud. \u00a0There are two pieces missing:\n
            \n
          1. It doesn’t create the volume in OpenStack if its not there yet.<\/li>\n
          2. It doesn’t mount the volume onto the development server.<\/li>\n<\/ol>\n<\/li>\n
          3. For speed, its better to pull the docker containers from the local registry. \u00a0So technically we should tag all the images that we’re using and put them in the local registry. \u00a0This is a chicken and an egg problem because you need the registry up before you can download images from it. \u00a0So I left it that way.<\/li>\n
          4. There are still some things I needed to finish like putting some keys and other items I needed for Jenkins in the \/vol directory. \u00a0Its not perfect but its pretty good.<\/li>\n<\/ol>\n
            Creating the volume and mounting was pretty quick once the image was up. First I created the volume and assigned it using the Horizon Dashboard that Metacloud provides.<\/p>\n
            \"Screen<\/a><\/p>\n
             <\/p>\n
            This was just a 20GB volume. \u00a0Once the instance was up I ran a few commands like:<\/p>\n
            fdisk -l # walk through menu create a new fs, accept defaults: e.g: n,enter,enter,m to write it out. \r\nmkfs.ext2 \/dev\/vdb1\r\nmkdir -p \/vol\r\nmount \/dev\/vdb1 \/vol<\/pre>\n
            This way all of our information persists if the containers terminate and if the instances terminate.<\/p>\n
            Finishing up the Development Server<\/h3>\n
            Once you get to this point, you should be able to bring it all up with:<\/p>\n
            ansible-playbook ci.yml<\/pre>\n
            That should do it! \u00a0Once you are in you may want to tag all of your images so that they load in the local docker registry. \u00a0For example, once you log in you could run:<\/p>\n
            core@ci ~ $ docker tag registry ci:5000\/registry\r\ncore@ci ~ $ docker push ci:5000\/registry<\/pre>\n
            At this point the idea is that you should be able to go to whatever public IP address was assigned to you and be able to access:<\/p>\n
             <\/p>\n
            jenkins: \u00a0<IP>:8080\r\ngitlab: <IP>:10080\r\nregistry: <IP>:5000<\/pre>\n
            If you’re there then you can get rolling to the next step: \u00a0Ansible scripts to deploy the rest of the environment.<\/p>\n
            In Part 3 we’ll cover Ansible for bringing up the load balancers and web servers. We’ll also snapshot an image to make it a jenkins slave.<\/p>\n","protected":false},"excerpt":{"rendered":"
            In Part 1 we gave the general outline of what we are trying to do, the tools we’re using, and the architecture of the application. In this part (Part 2) we’re going to work on building the development environment\u00a0with Ansible. \u00a0This includes the Jenkins, Gitlab, a private Docker Registry, and a proxy server so we…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1013,744,990,745,676],"tags":[530,734,732,1011,176,740],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/3330"}],"collection":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3330"}],"version-history":[{"count":7,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/3330\/revisions"}],"predecessor-version":[{"id":3377,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/3330\/revisions\/3377"}],"wp:attachment":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3330"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3330"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3330"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}