BENINCOSAArmchair Notes

An old man’s take on tech, systems, and trying to make sense of the world.

Technology

CI/CD server behind a firewall

By Vallard BenincosaApril 10, 20184 min read
The goal of this post is to show how to create a CI/CD server that sits behind a firewall.  When we do a git push then we would like a webhook to be fired off to our CI/CD server.  The CI/CD server would then run our jobs and create our artifacts.  Normally this is pretty simple when you have a server that sits at a public IP address on the internet.  But then I'd have to pay for that.  Why pay for it when I have lots of computing resources behind my corporate firewall? DRONE First we set up our CI/CD server.  I'll use drone just cause I think its cool the way it works using containers to do each step of the CI/CD pipeline.

NGROK

ngrok is recommended by Github as a way to get our private server an internet address.  Create an account by going to their homepage. We download the app and put it on our server 
wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
unzip ngrok*zip
mv ngrok /usr/local/bin
After signing up we get our own authtoken with ngrok.  Run the command to get it set up 
ngrok authtoken 5SJXZvAsM47HKXandsomeotherrandomcharaters
Now let's start this server.  On the free plan you get a new domain name everytime you start it so ideally we'd like to keep this session up for good.  One way we can do that is to create a service that does this.  The other way is to create a screen session.  To be quick and pedantic we will use a screen session to make this work. 
screen -S ngrok
Now in this screen session let's start our service 
ngrok http 9001
ngrok by @inconshreveable                                                                                                                          (Ctrl+C to quit)

Session Status                online
Account                       Vallard N Grok (Plan: Free)
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://1db27123.ngrok.io -> localhost:9001
Forwarding                    https://1db27123.ngrok.io -> localhost:9001

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00
This now shows that our public IP address is ldb27123.ngrok.io.  Cool. We'll use this now to set up our CI/CD server as well as Github.  You can detach from the screen using the Ctl-a-d command and the connection will stay up as long as the server is up. Notice that we told this to point to port 9001 on our server.  This is the port where we will have Drone run.

Github

We now go to our project and we will register a new application (drone in this case).  To do this on Github go to settings, developer settings , and register a new OAuth application. On the next screen you'll be shown the variables for client ID and client Secret.  Make note of those!

Environment Variables for CI/CD server

We're almost ready to bring up our CI/CD server.  First we will put in environment variables into the ~/.bash_profile file of the server. We define them as follows: 
export DRONE_HOST=http://ldb27123.ngrok.io
export DRONE_GITHUB_CLIENT=<github client ID from previous step>
export DRONE_GITHUB_SECRET=<github secret from previous step>
export DRONE_SECRET=<some random string you define>
  From there you need to 'source' the bash_profile by running 
. ~/.bash_profile
Test that its running by doing 
env | grep DRONE
You should see the environment variables print out.

Install Drone

To start up our Drone service on this server you'll need to have docker-compose installed.   We follow the instructions and created a docker-compose file that looks as follows: 
version: '2'

services:
  drone-server:
    image: drone/drone:0.8

    ports:
      - 9001:8000
      - 9000:9000
    volumes:
      - /var/lib/drone:/var/lib/drone/
    restart: always
    environment:
      - DRONE_OPEN=true
      - DRONE_HOST=${DRONE_HOST}
      - DRONE_GITHUB=true
      - DRONE_GITHUB_CLIENT=${DRONE_GITHUB_CLIENT}
      - DRONE_GITHUB_SECRET=${DRONE_GITHUB_SECRET}
      - DRONE_SECRET=${DRONE_SECRET}

  drone-agent:
    image: drone/agent:0.8

    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=drone-server:9000
      - DRONE_SECRET=${DRONE_SECRET}
Now we can start with 
docker-compose up -d
If you have problems, check the logs of the docker containers.  It may be that the environment variables are not set correctly. We made it so anyone can register and will need to change this to something different so only authorized users of our org can use our CI/CD server.  You can see how to do this with more settings.  Check out the documentation on Drone's web page.

Configure Drone

Now that you are installed you should be able to navigate to the drone web interface by going to the grok page.  You'll be redirected to your CI/CD server and be ready to accept push events.  Next up, you'll need to configure drone on your application to do all the wonders you like it to do!  Drone, unlike Jenkins, has a paradigm where you put the configuration of the CI/CD job into the github repo.  This works great as it allows jobs to be configured individually by the different applications. I'll probably write about that in a different post as we explore that with our team!

Notes

  • I did find I had a few errors when putting it in production.  To start out with I was using an older docker version so I got an error that was resolved with this post.
  • Because my docker images were behind a proxy I had to configure proxy settings.  This was done in the environment variable inside the .drone.yml file. See the example here.