This will be the first of a series of posts showing how we can do continuous delivery of a simple web application. This will be written tutorial style to show all the different components used. We are using Open Source tools on Cisco OpenStack private cloud, but the majority of the instructions here could be used in any cloud. This first part in the series is going to introduce the architecture and the components.
In Part 2 we configure the build environment with Ansible.
In Part 3 we configure the load balancers and the web site.
In Part 4 we configure Jenkins and put it all together.
Code for this can be found on Github in my Cisco Live 2015 repo.
If you want to see the end result of what we’re building, check out this video
A New Startup!
Our startup is called Lawn Gnomed. We specialize in lawn gnomes as a service. Basically, suppose your friend has a birthday. You engage with our website and on the day of their birthday they wake up and there are 50 lawn gnomes on their front yard with a nice banner that says: “Happy Birthday you old man!”. We set up the gnomes, we take them down. Just decorations.
The Requirements & the Stack
We need to react fast to changes in our business model. Right now we’re targeting birthdays, but what if we want to target pool parties? What about updates to our site for Fathers’ Day or Mothers’ Day? Instead of listening to the HiPPO (The Highest Paid Person’s opinion) we need to be able to react quicker. Try things out fast, if we’re wrong change, and do it fast.
1. We have a private cloud. We are part of a large corporation already. This app fits under the category of “System of Innovation” as defined by Gartner. We’re going to develop this on our Private Cloud. In this case Cisco OpenStack Private Cloud (formerly Metacloud) fits the bill for this nicely.
2. Our executives think we should leverage as much internally as possible. Our goals are to keep things all in our private data center. Most of these tools could use services and cloud based tools instead, but there are already plenty of tutorials out there for those types of environments. Instead, we’re going to focus on keeping everything in house.
3. We want to use containers for everything and keep everything ephemeral. We should be able to spin this environment up as quickly as possible in other clouds if we decide to change. So we are avoiding lockin as much as possible. This may be a bad idea as some argue, but this is the choice we are going with.
So here is our stack we’re developing with:
- OpenStack. In this example we are using Cisco OpenStack Private Cloud (formerly Metacloud) but we may instead decide that we want to do this on a different cloud platform, like Digital Ocean, AWS, or Azure.
- CoreOS. CoreOS is a popular lightweight operating system that works great for running containers.
- Docker. Our applications will be delivered and bundled in Docker Containers. They are quick and fast.
- Gitlab. We are using the free open source version of what Github offers to most organizations. We will be using Gitlab to check in all of our code.
- Jenkins. Our Continuous Integration service will be able to listen to Gitlab (or Github if you used that) and not only do our automated test cases when new changes are pushed, but will also be able to update our live servers.
- Slack. This is the only service we don’t host in house. Slack allows our team to be alerted anytime there is a new build or if something fails.
- Ansible. We are using Ansible to deploy our entire environment. Nothing should have to be done manually (where possible) if we can automate all the things. We’ve mostly followed that in this article, but there are some hands on places that we feel are ok for now, but can automate later.
In this series, we will not be concentrating so much on what the app does nor the database structure, but in an effort to be complete, we will add that for now we are using a simple Ruby on Rails application that uses BootStrap with a MariaDB backend.
The Application Stack
The application will be a set of scalable web services behind a pair of load balancers. Those in turn will talk to another set of load balancers that will house our database cluster.
The diagram below gives a high level view of our application.
- Blue circles represent the instances that are running in our cloud.
- Red circles represent the containers
- Green circles represent the mounted volumes that are persistent even when containers or instances go away.
We will probably add multiple containers and volumes to each instance, but for simplicity we show it running this way.
We have several choices on metacloud as to where we put the components. Cisco OpenStack Private Cloud has the concept of Availability Zones which are analogous to AWS Regions. If we have more Metacloud has theWe could if we were to do A/B testing put several components inside a different availability zone or a different project. Similarly, we could put the database portion inside its own project, or separate projects depending on what types of experiments we are looking to run.
Diving in a little deeper we can make each service a project. In this case the application could be a project and the database could be a separate project within each AZ.
Autoscaling the Stack
Cisco OpenStack Private Cloud does not come with an Autoscaling solution. Since Ceilometer is not part of the solution today, we can’t use that to determine load. We can, however use third party cloud management tools like those that come from Scalr or RightScale. These communicate with Cisco OpenStack Private Cloud via the APIs as well as agents installed on the running instances.
There is also the ability to run a poor mans autoscaling system that can be cobbled together with something like Nagios and scripts that:
- Add or Remove instances from a load balancer
- Monitors the CPU, memory, or other components on a system.
We would like the instances to run on separate physical hosts to increase stability. Since the major update in the April release we have that ability to add anti-affinity rules to accomplish this.
nova server-group-create –policy anti-affinity group-1
nova boot –image IMAGE_ID –flavor FLAVOR_ID –hint \
nova boot –image IMAGE_ID –flavor FLAVOR_ID –hint \
This rule will launch web01 and web02 on different physical servers. We mention this now as we won’t be going over it in the rest of the articles.
Logging and Analytics
Something we’ll be going over in a future post (I hope!) is how to log all the activity that happens in this system. This would include a logging system like Logstash that would consolidate every click and put it into place where we can run analytics applications. From this we can determine what paths our users are taking when they look at our website. We could also analyze where are users come from (geographically) and what times our web traffic gets hit the hardest.
Cisco OpenStack Private Cloud allows us to carve up our hypervisors into aggregates. An aggregates is a collection of nodes that may be dedicated to one or more projects. In this case, it could be hadoop.
The blue arrow denotes the collection of servers we use for our analytics.
Continuous Delivery Environment
A simple view of our Continuous Delivery environment is shown below
- A developer updates code and pushes it to Gitlab. This Gitlab server is the place where all of our code resides.
- When Gitlab sees that new code has been received he notifies Jenkins. Gitlab also notifies Slack (and thus all the slackers) that there was a code commit.
- Jenkins takes the code, merges it, and then begins the tests. Jenkins also notifies all the slackers that this is going to happen.
- As part of the build process, Jenkins creates new instances on Cisco Openstack Private Cloud / Metacloud. Here’s what the instances do when they boot up:
- Download the code from gitlab that was just checked in.
- Perform a ‘Docker build’ to build a new container.
- Run test cases on the container.
- If the tests are successful, the container is pushed to a local Docker Registry where it is now ready for production. Slack is notified that new containers are ready for production.
- A second Jenkins job has been configured to automatically go into each of our existing web hosts, download the new containers, and put them into production and remove the new ones. This only happens if a new build passed.
This whole process in my test environment takes about 5 minutes. If we were to run further test cases it could take longer but this illustrates the job pretty quickly.
The Build Environment
Our build environment is pretty simple. It consists of a single instance with a mounted volume. On this instance we are running 4 containers:
- NGINX. This does our reverse proxying so that subdomains can be hit.
- Jenkins. This is the star of our show that runs the builds and puts things into production.
- Registry. This is a local docker registry. We’re using the older one here.
- Gitlab. This is where we put all our code!
This shows the power of running containers. Some of these services need their own databases and redis caches. Putting that all on a single machine and coordinating dependencies is crazy. By using containers we can pile them up where we need them.
The other thing to note is that all of the instances we create in OpenStack are the same type. CoreOS 633.1.0 right now.
The last piece of this first part is that we’ll need to gain access to our cloud. Not just GUI access but command line access so that we can interface with the APIs.
Once you login to your project you can go to the Access & Security menu and select API Access. From there you can download the OpenStack RC file.
Test it out with the nova commands:
$ nova list
| ID | Name | Status | Task State | Power State | Networks |
| ad4dfbc4-3048-4f8a-a104-6adaffbeab45 | ci | ACTIVE | - | Running | trial2-3003=10.2.3.7, 126.96.36.199 |
| d97f38d7-ce0d-47dd-a105-281de586e6c8 | demo-server | ACTIVE | - | Running | trial2-3003=10.2.3.5, 188.8.131.52 |
| 445d4040-e1bf-4af4-b2c9-4f60c5bd64a5 | load-balancer-a | ACTIVE | - | Running | trial2-3003=10.2.3.2, 184.108.40.206 |
| 2d5c5436-6dae-4c6a-a36f-093ac853f704 | load-balancer-b | ACTIVE | - | Running | trial2-3003=10.2.3.3, 220.127.116.11 |
| 1ae2e4bd-9936-43a9-a870-c12f465312b5 | web01 | ACTIVE | - | Running | trial2-3003=10.2.3.4 |
| 31356f78-f068-4b3b-8c66-c1b3bdac00e7 | web02 | ACTIVE | - | Running | trial2-3003=10.2.3.6 |
| 25c2d915-5dae-4105-af58-73189ca7748a | web03 | ACTIVE | - | Running | trial2-3003=10.2.3.8 |
While you may not see all the instances that I’m showing here, you should at least see some output that shows things are successful.
The next sections will be more hands on. Refer back to this section for any questions as to what the different components do. The next section will talk about:
- Getting Image for CoreOS
- Ansible Configuration
- Cloud Init to boot up our instances
- Deploying load balancers
- Deploying web servers
- Deploying Jenkins Slaves.