Continuous integration (CI) is the hot new thing in the computer infrastructure world. To learn CI I'm going to practise it using one of my favorite python projects - netshow. Hopefully in a few months I can begin to manage compute infrastructure changes using CI.

Platform/Hypervisor

Ubuntu 16.04 Laptop with enough RAM/Disk to hold all the necessary docker components.

Tools

Installing GitLab

Docker-compose is used to setup the image, volume and networking of the various containers required to setup the Continuous Integration environment.

docker-compose.yml for setting up gitlab

version: '2'

networks: gitlab_net: driver: bridge ipam: driver: default config: - subnet: 192.168.201.0/24 gateway: 192.168.201.1 services:

prodpypi: image: linuxsimba/devpi containername: prodpypi ports: - "9200:3141" volumes: - ${HOME}/prodpypi:/data networks: gitlabnet: ipv4address: 192.168.201.13 environment: - DEVPI_PASSWORD=mysecretpass

stagingpypi: image: linuxsimba/devpi containername: stagingpypi networks: gitlabnet: ipv4address: 192.168.201.12 ports: - "9100:3141" volumes: - ${HOME}/stagingpypi:/data environment: - DEVPI_PASSWORD=mysecretpass

gitlab: image: gitlab/gitlab-ce:latest containername: gitlab hostname: gitlab.linuxsimba.test ports: - "9080:80" - "9443:443" - "9022:22" volumes: - ${HOME}/gitlab/config:/etc/gitlab - ${HOME}/gitlab/logs:/var/log/gitlab - ${HOME}/gitlab/data:/var/opt/gitlab networks: gitlabnet: ipv4_address: 192.168.201.10

gitlab-runner: image: gitlab/gitlab-runner:latest containername: gitlab-runner dnssearch: linuxsimba.test networks: - gitlabnet extrahosts: - "gitlab.linuxsimba.test:192.168.201.10" volumes: - ${HOME}/gitlab-runner/config:/etc/gitlab-runner - /var/run/docker.sock:/var/run/docker.sock environment:

the Docker network resolves the docker name has a DNS name. That is

why you can say http://staging_pypi:3141 or http://gitlab from within

this docker container. Read more about docker networking from the docker

website.

    - CI_SERVER_URL=http://gitlab/ci

This registration token can only be used once a Project/git repo is recreated.

    - REGISTRATION_TOKEN=
    - RUNNER_NAME=python_docker
    - RUNNER_EXECUTOR=docker
    - DOCKER_IMAGE=python:2.7.12-alpine

Notice that it matches exactly the network found in docker network ls

    - DOCKER_NETWORK_MODE=gitlab_gitlab_net
    - DOCKER_EXTRA_HOSTS=gitlab.linuxsimba.test:192.168.201.10

network layout of docker containers

Docker-compose Network Topology

CLI commands on the Platform/Hypervisor

apt-get install python-setuptools
sudo easy_install pip
sudo pip install docker-compose

mkdir gitlab-docker
cd gitlab-docker
mkdir $HOME/staging_pypi
mkdir $HOME/prod_pypi

sudo chown 1000:1000 $HOME/staging_pypi
sudo chown 1000:1000 $HOME/prod_pypi
wget https://goo.gl/FHIzgO -o docker-compose.yml

docker-compose up -d
!! Wait a few minutes for Gitlab to load

Initial Setup of Gitlab

TODO: Use python-gitlab to automate the process described below

Add a root password

Go to http://localhost:9080 and create a root password. In this test case it is cn321cn321!

enter root password

Create a new user

Refresh http://localhost:9080 and create new user with the following parameters

  • username: demo
  • password: cn321cn321!
  • name: demo
  • email: demo@example.com

new demo user

Create a new project

Select "new project" and then import the github netshow-core To perform this import, one needs to create a Github API Token

import project from github

Add a SSH Key to the Demo user profile

When the project is accessed for the first time, it will alert saying to add a SSH key. Take the $HOME/.ssh/id_rsa.pub key and add it to the SSH key Profile section

Obtain the gitlab runner token.

Gitlab uses a Go-Erlang application to manage docker containers that will run the test builds. This app is called gitlab-runner. It is stored in its own container. Each project needs to be associated with a gitlab-runner if continuous integration is to take place.

To view runners for a project, click on the top right side dropdown and select runner. Or use a link that looks like this:

runner_dropdown

One bug with the GUI is that if you have a screen that doesn't cover the whole screen, then the top right dropdown is not visible.

On the runner page, note the regsitration token. This will be used in the next step

project registration token

Register Gitlab-runner

From the hypervisor CLI run the following commands

~:% docker exec -it gitlab-runner gitlab-runner register
Running in system-mode.                            

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
[http://gitlab/ci]:   [hit enter]     
Please enter the gitlab-ci token for this runner:
B4qz89Eattey2XLVx86s
Please enter the gitlab-ci description for this runner:
[python_docker]: [hit enter]
Please enter the gitlab-ci tags for this runner (comma separated):
netshow
[ wait a few seconds to get the next option ]
Registering runner... succeeded                     runner=B4qz89Ea
Please enter the executor: parallels, shell, virtualbox, docker, docker-ssh, ssh, docker+machine, docker-ssh+machine, kubernetes:
[docker]:     [hit enter]
Please enter the default Docker image (eg. ruby:2.1):
[python:2.7.12-alpine]: [hit enter]
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

~:% docker exec -it gitlab-runner gitlab-runner list    
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
python_docker                                       Executor=docker Token=dd4b0a773e63ac56d45fde0f5fe9df URL=http://gitlab/ci

Gitlab is now setup to perform continuous integration on the netshow-core project. The next article will cover how to setup the CI steps for the project and how to use CI to validate changes to the Git repository.