Using Pulumi with Jenkins

This post will demonstrate how to setup Jenkins to install the necessary Pulumi Python dependencies, and show an example deployment pipeline to setup a couple of EC2 instances on AWS.

Here is an overview of the deployment process.

Note. As this is for demonstration purposes, the pipeline will also destroy the EC2 instances on AWS it initially created. This is executed via the stage: Pulumi Cleanup.

All the resources required can be found in the following Gitlab repository:

https://gitlab.com/spengler84/public-repo.git

Setup Jenkins

I’ll setup the Jenkins master inside a docker container for this demonstration.

Dockerfile

I’m using Ubuntu as the base image, then installing Jenkins along with additional dependencies.

  • entrypoint.sh - a shell script taken from here - this is used to launch Jenkins in the background.

  • Pulumi only supports Python versions >=3.6

FROM ubuntu:latest

ARG user=jenkins
ARG group=jenkins
ARG REF=/usr/share/jenkins/ref
ARG JENKINS_HOME=/var/jenkins_home
ARG PULUMI_VERSION=latest
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000

ENV JENKINS_SLAVE_AGENT_PORT ${agent_port}
ENV REF $REF
ENV JENKINS_HOME $JENKINS_HOME

ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

RUN mkdir -p /etc/pki/tls/certs

RUN apt-get update -qq

RUN apt-get install wget gnupg curl default-jdk python3-venv python3 python3-pip  -qq

RUN wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | apt-key add -

RUN sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
    /etc/apt/sources.list.d/jenkins.list'

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9B7D32F2D50582E6

RUN apt-get update -qq

RUN apt-get install jenkins -qq

ENV PATH=$PATH:/root/.pulumi/bin

RUN curl -fsSL https://get.pulumi.com/ | sh

RUN pip3 install grpcio pipenv pulumi-aws

RUN mkdir -p ${REF}/init.groovy.d ${JENKINS_HOME}

RUN chown -R ${user}:${group} "$JENKINS_HOME" "$REF"

RUN update-ca-certificates && cd /etc/pki/tls/certs/ && ln -s /etc/ssl/certs/ca-certificates.crt ca-bundle.crt

COPY entrypoint.sh /
ENTRYPOINT [ "/entrypoint.sh" ]

Build the image by being in the same directory of your Dockerfile.

docker build -t jenkins-ubuntu:0.1 .

docker-compose.yml

version: '3.7'
networks:
  default:
    external: 
      name: dev

services:
  jenkins:
    restart: always
    ports:
      - 9090:8080
    image: jenkins-ubuntu:0.1
    container_name: jenkins-master
    volumes:
      - /volume1/docker-data/jenkins-master/:/var/jenkins_home
  • I’ve mapped an external volume to manage the Jenkins home directory. This way the directory is retained even if I destroy the container.
  • Port forwarding is also enabled
  • A separate network called ‘dev’ is used to isolate this container from accessing other Docker resources

Bring up Jenkins

docker-compose up -d

Check the logs of the container for any problems related to the startup of Jenkins.

docker logs -f jenkins-master

Create Jenkins credential

This will be used to store the AWS Access key ID & AWS Secret access key.

Navigate to https://jenkins_host/credentials/store/system/domain/_/newCredentials

This will open the following.

Username: Enter the AWS access Key Id

Password: Enter the AWS Secret access key

Description: Enter a meaningful description

Once you click on OK.

Jenkins will save this Credential and generate a hash for the ID. Copy this hash as you will need to enter into the Jenkins pipeline.

It will look similar to.

Jenkins pipeline

#!groovy

pipeline {
agent any
environment {
DEVOPS_WORKSPACE = "pulumi-automation"
PULUMI_STACK = "pulumi-aws-simple"
}
parameters {
string(defaultValue: '<replace with jenkins credential hash>', description: 'AWS credentials', name: 'AWS_CREDENTIAL')
string(defaultValue: '<replace with AWS region>', description: 'AWS Region', name: 'AWS_REGION')
password(defaultValue: '<replace with pulumi access token>', description: 'Pulumi Access token', name: 'PULUMI_ACCESS_TOKEN')


}
options {
// Only keep the 5 most recent builds
buildDiscarder(logRotator(numToKeepStr: '5'))
skipDefaultCheckout(true)
}
stages {

stage('Pulumi : Login') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh "export PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN}; pulumi login"
    }
    }
  }
  }
}

stage('Pulumi : Initialize stack') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh ""
      "#!/bin/bash
      stack = \$(pulumi stack ls | grep "pulumi-aws-simple" | awk {
      'print \$1'
      })
      if [-z\ $stack];
      then
      pulumi stack init\ $PULUMI_STACK
      else
      pulumi stack select\ $PULUMI_STACK
      fi
      pulumi config set aws: region\ $AWS_REGION ""
      "
    }
    }
  }
  }

}

stage('Pulumi : Deployment') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    withCredentials([
    [$class: 'UsernamePasswordMultiBinding', credentialsId: "${params.AWS_CREDENTIAL}", usernameVariable: 'AWS_ACCESS_KEY_ID', passwordVariable: 'AWS_SECRET_ACCESS_KEY']
    ]) {
    timestamps {
      script {
      sh ""
      "    #!/bin/bash
      echo "==> Cleanup Python temp files."
      rm - rf venv Pipfile.lock __pycache__

      python3 -m venv venv
      . venv/bin/activate
      set AWS_ACCESS_KEY_ID = $ {
        AWS_ACCESS_KEY_ID
      }
      set AWS_SECRET_ACCESS_KEY = $ {
        AWS_SECRET_ACCESS_KEY
      }

      echo "==> Install Python modules."
      pip3 install - r requirements.txt;
      echo "==> Pulumi deployment."
      pulumi up - y

      deactivate
        ""
      "
      }
    }
    }
  }
  }
}

stage('Pulumi : Cleanup') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    withCredentials([
    [$class: 'UsernamePasswordMultiBinding', credentialsId: "${params.AWS_CREDENTIAL}", usernameVariable: 'AWS_ACCESS_KEY_ID', passwordVariable: 'AWS_SECRET_ACCESS_KEY']
    ]) {
    timestamps {
      script {
      sh ""
      "  #!/bin/bash
      echo "==> Destroy AWS EC2 instances."
      python3 - m venv venv
      . venv/bin/activate
      set AWS_ACCESS_KEY_ID = $ {
        AWS_ACCESS_KEY_ID
      }
      set AWS_SECRET_ACCESS_KEY = $ {
        AWS_SECRET_ACCESS_KEY
      }

      pulumi destroy - y


      pulumi stack rm $ {
        PULUMI_STACK
      } - y

      deactivate
      rm -rf venv
        ""
      "
      }
    }
    }

  }
  }

}

stage('Pulumi : Logout') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh "pulumi logout"
    }
    }
  }
  }
}
}
}

Jenkins pipeline stages

Pulumi: Login

  • Authenticate the user to make calls to the Pulumi service.

  • Setting the environment variable PULUMI_ACCESS_TOKEN allows login to be achieved via a script.

export PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN}; pulumi login
Pulumi: Initialize stack
  • Initilizes a stack else selects an existing one * Configures pulumi to set AWS region to deploy into
stack=$(pulumi stack ls | grep "pulumi-aws-simple" | awk {'print $1'})
if [ -z $stack  ]; then
  stack init $PULUMI_STACK
  stack select $PULUMI_STACK
fi
pulumi config set aws:region $AWS_REGION
Pulumi: Deployment
  • Cleans temporary files
  • Creates Python virtual environment
  • sets AWS access credentials environment variables
  • installs Pulumi dependencies
  • starts Pulumi deployment
  • Deactivates Python virtual environment
echo "==> Cleanup Python temp files."
rm -rf venv Pipfile.lock __pycache__
python3 -m venv venv
  . venv/bin/activate
set AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
set AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

echo "==> Install Python modules."                                
pip3 install -r requirements.txt;
echo "==> Pulumi deployment."
pulumi up -y
                                      
deactivate
echo "==> Cleanup Python temp files."
rm -rf venv Pipfile.lock __pycache__
Pulumi : Cleanup
  • Destroys recently deployed infrastructure * Removes Pulumi stack
echo "==> Destroy AWS EC2 instances."
python3 -m venv venv
. venv/bin/activate
set AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
set AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

pulumi destroy -y
pulumi stack rm ${PULUMI_STACK} -y

deactivate

Pulumi files

main.py

This is where you define what Pulumi will do.

  • Creates 2 EC2 instances in AWS
  • Creates 1 security group to allow port 22 for SSH access
  • this should not be open to the internet like I have done, but be confined to a range of IP-Addresses (for demo purposes I’ve left it open) - this is controlled by line 21
  • Selects a centos image
import pulumi
import pulumi_aws as aws

size = 't1.micro'
no_servers = 2

ami = aws.get_ami(most_recent="true",
                  owners=["247102896272"],
                  filters=[{
                      "name":
                      "name",
                      "values": ["aws-parallelcluster-2.0.0-centos7-hvm-*"]
                  }])

group = aws.ec2.SecurityGroup('ssh-secgrp',
                              description='Enable SSH access',
                              ingress=[{
                                  'protocol': 'tcp',
                                  'from_port': 22,
                                  'to_port': 22,

                                  ## DO NOT LEAVE OPEN
                                  'cidr_blocks': ['0.0.0.0/0']
                              }])

for i in range(no_servers):
    server = aws.ec2.Instance('ssh-server-{}'.format(i),
                            instance_type=size,
                            vpc_security_group_ids=[group.id],
                            ami=ami.id)

    pulumi.export('public_ip', server.public_ip)
    pulumi.export('public_dns', server.public_dns)
Pulumi.yaml

Defines the Pulumi program.

name: aws-py-simple
runtime: python
description: A minimal AWS Python Pulumi program
requirements.txt
pulumi>=2.0.0,<3.0.0
pulumi-aws>=2.0.0,<3.0.0

Running the Jenkins pipeline

Once you have setup the Jenkins pipeline, you will be presented with the following once you click on:

I’ve named the pipeline ‘PULUMI-AWS-CREATE-EC2’

Navigate to the Console Output to analyze the logs generated at runtime.

Last updated on 29 Apr 2020
Published on 29 Apr 2020