Investigating CI/CD: Local Setup

GOAL:

Automate testing and image creation of a Node/Express API that relies on Postgres for persistence.

TECHNOLOGY:

Node
Express
Postgres
Docker

REQUIREMENTS:

bash script
run tests
report success or failure
on success:
- increment version
- tag the code in github with new version
- build and tag a docker image with new version
- push docker image to docker hub

Local Setup

To better understand the CI/CD process, I built a script that I can run locally to accomplish all of the requirements.

This script assumes the user has a Docker Hub account, a local Postgres database named basic_db_test, and that my Node API is dockerized. For reference I have included the Dockerfiles below.

Steps:

be sure the directory is clean
clear the test database and apply migrations
run the tests in the docker container
if tests pass, continue
update and commit application version
push new version and tag to git
build container
tag it
sign in to docker hub
push both latest and $version to docker hub

#!/bin/bash

# 1. be sure the directory is clean
# 2. clear the test database and apply migrations
# 3. run the tests in the docker container
# 4. if tests pass, continue
# 5. update and commit application version
# 6. push new version and tag to git
# 7. build container
# 8. tag it
# 9. sign in to docker hub
# 10. push both latest and $version to docker hub

# 1. be sure the directory is clean
if ! (git status | grep 'nothing to commit'); then
  echo "\nWARNING\n"
  echo "You have uncommited changes. Please commit changes before building.\n"
  echo "Exiting build script.\n"
  exit 1
fi

# 2. clear the test database and apply migrations
BASIC_DB_TEST=postgres://localhost:5432/basic_db_test \
    knex migrate:rollback --env test --knexfile ./knexfile.js
BASIC_DB_TEST=postgres://localhost:5432/basic_db_test \
    knex migrate:latest --env test --knexfile ./knexfile.js

# 3. run the tests in the docker container
echo 'Running tests ... '
docker build -f Dockerfile.test -t topleft/api-boiler-test .
CONTAINER_ID=\
    $(docker run -d --env-file=env_file.test topleft/api-boiler-test:latest)
echo "Container ID: $CONTAINER_ID"

STATUS=$(docker inspect $CONTAINER_ID --format='{{.State.Status}}')
until [ $STATUS = 'exited' ]; do
  sleep 2
  STATUS=$(docker inspect $CONTAINER_ID --format='{{.State.Status}}');
done

EXIT_CODE=$(docker inspect $CONTAINER_ID --format='{{.State.ExitCode}}')
# 4. if tests pass, continue
if [ $EXIT_CODE -eq 0 ];
  then
    echo "\nTests past!\n"

    # 5. update and commit application version
    VERSION=$(npm version patch)
    # 6. push new version and tag to git
    git push
    git push --tags

    # 7. build container
    docker build -f Dockerfile -t topleft/api-boiler:$VERSION .

    # 8. tag it
    docker tag topleft/api-boiler:latest topleft/api-boiler:$VERSION

    # 9. sign in to docker hub
    echo $DOCKER_PASSWORD | docker login --username topleft --password-stdin

    # 10. push both latest and $version to docker hub
    docker push topleft/api-boiler:latest
    docker push topleft/api-boiler:$VERSION

    echo "DOCKER IMAGE: topleft/api-boiler:$VERSION\n"
    echo "Build successful!\n"
  else
    echo "\n Tests failed.\n"
    echo "For details run --> \n docker logs $CONTAINER_ID\n"
    exit 1
fi

Follow the comments for an explanation of what is going on.

To run this script for the first time:

$ create_db basic_db_test
$ npm i -g knex
$ DOCKER_PASSWORD=mypassword && sh local_build.sh

Dockerfiles

I wrote two Dockerfiles: one for testing that includes all the dependencies to run the test suite and one for production that produces an image with only the code and dependencies necessary for runtime.

Production build:

FROM node:10

WORKDIR /usr/src

COPY knexfile.js ./
COPY package*.json ./

RUN npm install --production

COPY ./src ./src/.

EXPOSE 3030

CMD [ "npm", "start" ]

Explanation line by line:

FROM node:10 - use Node 10.xx.xx as the base image for this docker container. This image includes a Debian flavor of Linux with Node, NPM and many other relevant programs/dependencies already installed. In other words, this is the operating system (batteries included) on which your Node application will run.

WORKDIR /usr/src - set the working directory. after running this instruction $ pwd would return /usr/src.

COPY knexfile.js ./ - the COPY instruction takes files/directories relative to the root directory of your project and copies them to the working dir of the docker container. Typically files that are not in your source code (application code), but are needed for building or testing are copied in piecemeal.

COPY package*.json ./ - same as above.

RUN npm install - install our dependencies. This command will result in modules being installed to /usr/src/node_modules/

COPY ./src ./src/. - here we are copying our source code into the container. The nuts and bolts of our application.

EXPOSE 3030 - The expose instruction serves as 'code as documentation'. It does not actually expose the port. You could remove this line and the container would behave in the same exact way. Its purpose is to tell other programmers that port 3030 is where the app is running on the inside of the container. In order to access port 3030 from the outside of the container, for example localhost:3030, it will need to mapped: port 3030 from within the container to port 3030 on the host machine. This instruction will be given in the command when the container is actually run. Finally, for clarity, the host machine is the computer that the container is being run on. That could be your computer, and AWS EC2 instance, etc.

CMD [ "npm", "start" ] - finally the CMD instruction provides the command that will be executed when the container is run.

Test Build

FROM node:10

WORKDIR /usr/src

COPY knexfile.js ./
COPY package*.json ./
COPY .eslintrc ./
COPY resetTestDb.sh ./migrate.sh
COPY docker-test-entrypoint.sh ./

ENV PATH /usr/src/node_modules/.bin:$PATH

RUN npm install

COPY ./src ./src/.
COPY ./test ./test/.

CMD [ "npm" , "run" , "test" ]