Using Pulumi with Jenkins

This post will demonstrate how to setup Jenkins to install the necessary Pulumi Python dependencies, and show an example deployment pipeline to setup a couple of EC2 instances on AWS.

Here is an overview of the deployment process.

Note. As this is for demonstration purposes, the pipeline will also destroy the EC2 instances on AWS it initially created. This is executed via the stage: Pulumi Cleanup.

All the resources required can be found in the following Gitlab repository:

https://gitlab.com/spengler84/public-repo.git

Setup Jenkins

I’ll setup the Jenkins master inside a docker container for this demonstration.

Dockerfile

I’m using Ubuntu as the base image, then installing Jenkins along with additional dependencies.

entrypoint.sh - a shell script taken from here - this is used to launch Jenkins in the background.

Pulumi only supports Python versions >=3.6

FROM ubuntu:latest
    
ARG user=jenkins
ARG group=jenkins
ARG REF=/usr/share/jenkins/ref
ARG JENKINS_HOME=/var/jenkins_home
ARG PULUMI_VERSION=latest
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
    
ENV JENKINS_SLAVE_AGENT_PORT ${agent_port}
ENV REF $REF
ENV JENKINS_HOME $JENKINS_HOME
    
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
    
RUN mkdir -p /etc/pki/tls/certs
    
RUN apt-get update -qq
    
RUN apt-get install wget gnupg curl default-jdk python3-venv python3 python3-pip  -qq
    
RUN wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | apt-key add -
    
RUN sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
    /etc/apt/sources.list.d/jenkins.list'
    
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9B7D32F2D50582E6
    
RUN apt-get update -qq
    
RUN apt-get install jenkins -qq
    
ENV PATH=$PATH:/root/.pulumi/bin
    
RUN curl -fsSL https://get.pulumi.com/ | sh
    
RUN pip3 install grpcio pipenv pulumi-aws
    
RUN mkdir -p ${REF}/init.groovy.d ${JENKINS_HOME}
    
RUN chown -R ${user}:${group} "$JENKINS_HOME" "$REF"
    
RUN update-ca-certificates && cd /etc/pki/tls/certs/ && ln -s /etc/ssl/certs/ca-certificates.crt ca-bundle.crt
  
COPY entrypoint.sh /
ENTRYPOINT [ "/entrypoint.sh" ]

Build the image by being in the same directory of your Dockerfile.

    docker build -t jenkins-ubuntu:0.1 .

docker-compose.yml

    version: '3.7'
    networks:
      default:
        external: 
          name: dev
    
    services:
      jenkins:
        restart: always
        ports:
          - 9090:8080
        image: jenkins-ubuntu:0.1
        container_name: jenkins-master
        volumes:
          - /volume1/docker-data/jenkins-master/:/var/jenkins_home

I’ve mapped an external volume to manage the Jenkins home directory. This way the directory is retained even if I destroy the container. * Port forwarding is also enabled * A separate network called ‘dev’ is used to isolate this container from accessing other Docker resources

Bring up Jenkins

    docker-compose up -d

Check the logs of the container for any problems related to the startup of Jenkins.

    docker logs -f jenkins-master

Create Jenkins credential

This will be used to store the AWS Access key ID & AWS Secret access key.

Navigate to https://jenkins_host**/credentials/store/system/domain/_/newCredentials**

This will open the following.

Username: Enter the AWS access Key Id

Password: Enter the AWS Secret access key

Description: Enter a meaningful description

Once you click on OK.

Jenkins will save this Credential and generate a hash for the ID. Copy this hash as you will need to enter into the Jenkins pipeline.

It will look similar to.

Jenkins pipeline

#!groovy

pipeline {
agent any
environment {
DEVOPS_WORKSPACE = "pulumi-automation"
PULUMI_STACK = "pulumi-aws-simple"
}
parameters {
string(defaultValue: '<replace with jenkins credential hash>', description: 'AWS credentials', name: 'AWS_CREDENTIAL')
string(defaultValue: '<replace with AWS region>', description: 'AWS Region', name: 'AWS_REGION')
password(defaultValue: '<replace with pulumi access token>', description: 'Pulumi Access token', name: 'PULUMI_ACCESS_TOKEN')


}
options {
// Only keep the 5 most recent builds
buildDiscarder(logRotator(numToKeepStr: '5'))
skipDefaultCheckout(true)
}
stages {

stage('Pulumi : Login') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh "export PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN}; pulumi login"
    }
    }
  }
  }
}

stage('Pulumi : Initialize stack') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh ""
      "#!/bin/bash
      stack = \$(pulumi stack ls | grep "pulumi-aws-simple" | awk {
      'print \$1'
      })
      if [-z\ $stack];
      then
      pulumi stack init\ $PULUMI_STACK
      else
      pulumi stack select\ $PULUMI_STACK
      fi
      pulumi config set aws: region\ $AWS_REGION ""
      "
    }
    }
  }
  }

}

stage('Pulumi : Deployment') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    withCredentials([
    [$class: 'UsernamePasswordMultiBinding', credentialsId: "${params.AWS_CREDENTIAL}", usernameVariable: 'AWS_ACCESS_KEY_ID', passwordVariable: 'AWS_SECRET_ACCESS_KEY']
    ]) {
    timestamps {
      script {
      sh ""
      "    #!/bin/bash
      echo "==> Cleanup Python temp files."
      rm - rf venv Pipfile.lock __pycache__

      python3 -m venv venv
      . venv/bin/activate
      set AWS_ACCESS_KEY_ID = $ {
        AWS_ACCESS_KEY_ID
      }
      set AWS_SECRET_ACCESS_KEY = $ {
        AWS_SECRET_ACCESS_KEY
      }

      echo "==> Install Python modules."
      pip3 install - r requirements.txt;
      echo "==> Pulumi deployment."
      pulumi up - y

      deactivate
        ""
      "
      }
    }
    }
  }
  }
}

stage('Pulumi : Cleanup') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    withCredentials([
    [$class: 'UsernamePasswordMultiBinding', credentialsId: "${params.AWS_CREDENTIAL}", usernameVariable: 'AWS_ACCESS_KEY_ID', passwordVariable: 'AWS_SECRET_ACCESS_KEY']
    ]) {
    timestamps {
      script {
      sh ""
      "  #!/bin/bash
      echo "==> Destroy AWS EC2 instances."
      python3 - m venv venv
      . venv/bin/activate
      set AWS_ACCESS_KEY_ID = $ {
        AWS_ACCESS_KEY_ID
      }
      set AWS_SECRET_ACCESS_KEY = $ {
        AWS_SECRET_ACCESS_KEY
      }

      pulumi destroy - y


      pulumi stack rm $ {
        PULUMI_STACK
      } - y

      deactivate
      rm -rf venv
        ""
      "
      }
    }
    }

  }
  }

}

stage('Pulumi : Logout') {
  steps {
  dir(DEVOPS_WORKSPACE) {
    timestamps {
    script {
      sh "pulumi logout"
    }
    }
  }
  }
}
}
}

Jenkins pipeline stages

Authenticate the user to make calls to the Pulumi service.
Setting the environment variable PULUMI_ACCESS_TOKEN allows login to be achieved via a script.

    export PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN}; pulumi login

Pulumi: Initialize stack

Initilizes a stack else selects an existing one * Configures pulumi to set AWS region to deploy into

    stack=$(pulumi stack ls | grep "pulumi-aws-simple" | awk {'print $1'})
    if [ -z $stack  ]; then
     stack init $PULUMI_STACK
     stack select $PULUMI_STACK
    fi
    pulumi config set aws:region $AWS_REGION

Pulumi: Deployment

Cleans temporary files * Creates Python virtual environment * sets AWS access credentials environment variables * installs Pulumi dependencies * starts Pulumi deployment * Deactivates Python virtual environment

    echo "==> Cleanup Python temp files."
    rm -rf venv Pipfile.lock __pycache__
    python3 -m venv venv
     . venv/bin/activate
    set AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    set AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    
    echo "==> Install Python modules."                                
    pip3 install -r requirements.txt;
    echo "==> Pulumi deployment."
    pulumi up -y
                                         
    deactivate
    echo "==> Cleanup Python temp files."
    rm -rf venv Pipfile.lock __pycache__

Pulumi : Cleanup

Destroys recently deployed infrastructure * Removes Pulumi stack

    echo "==> Destroy AWS EC2 instances."
    python3 -m venv venv
    . venv/bin/activate
    set AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
    set AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    
    pulumi destroy -y
    pulumi stack rm ${PULUMI_STACK} -y
    
    deactivate

Pulumi files

main.py

This is where you define what Pulumi will do.

Creates 2 EC2 instances in AWS * Creates 1 security group to allow port 22 for SSH access * this should not be open to the internet like I have done, but be confined to a range of IP-Addresses (for demo purposes I’ve left it open) - this is controlled by line 21 * Selects a centos image

    import pulumi
    import pulumi_aws as aws
    
    size = 't1.micro'
    no_servers = 2
    
    ami = aws.get_ami(most_recent="true",
                      owners=["247102896272"],
                      filters=[{
                          "name":
                          "name",
                          "values": ["aws-parallelcluster-2.0.0-centos7-hvm-*"]
                      }])
    
    group = aws.ec2.SecurityGroup('ssh-secgrp',
                                  description='Enable SSH access',
                                  ingress=[{
                                      'protocol': 'tcp',
                                      'from_port': 22,
                                      'to_port': 22,
                                      'cidr_blocks': ['0.0.0.0/0']
                                  }])
    
    for i in range(no_servers):
        server = aws.ec2.Instance('ssh-server-{}'.format(i),
                                instance_type=size,
                                vpc_security_group_ids=[group.id],
                                ami=ami.id)
    
        pulumi.export('public_ip', server.public_ip)
        pulumi.export('public_dns', server.public_dns)

Pulumi.yaml

Defines the Pulumi program.

    name: aws-py-simple
    runtime: python
    description: A minimal AWS Python Pulumi program

requirements.txt

    pulumi>=2.0.0,<3.0.0
    pulumi-aws>=2.0.0,<3.0.0

Running the Jenkins pipeline

Once you have setup the Jenkins pipeline, you will be presented with the following once you click on:

I’ve named the pipeline ‘PULUMI-AWS-CREATE-EC2’

Navigate to the Console Output to analyze the logs generated at runtime.