Accelerating DevSecOps on AWS: Create secure CI/CD pipelines using Chaos and AIOps

BIRMINGHAM—MUMBAI

Accelerating DevSecOps on AWS

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Rahul Nair

Publishing Product Manager: Meeta Rajani

Senior Editor: Sangeeta Purkayastha

Content Development Editor: Yasir Ali Khan

Technical Editor: Shruthi Shetty

Copy Editor: Safis Editing

Project Coordinator: Shagun Saini

Proofreader: Safis Editing

Indexer: Subalakshmi Govindhan

Production Designer: Shyam Sundar Korumilli

Senior Marketing Coordinator: Sanjana Gupta

Marketing Coordinator: Nimisha Dua

First published: April 2022

Production reference: 1060422

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80324-860-8

www.packt.com

To my father, Nagendra Ram.

I would have been a terrible software engineer if you had not bought me a desktop and taught me cut, copy, and paste in Windows 98.

– Nikit Swaraj

Contributors

About the author

Nikit Swaraj is an experienced solution architect. He is well versed in the melding of development and operations to deliver efficient code. Nikit has expertise in designing, developing, and delivering enterprise-wide solutions that meet business requirements and enhance operational efficiency. As an AWS solution architect, he has plenty of experience in designing end-to-end IT solutions and leading and managing complete projects within time and budgetary constraints. He contributes to open source projects and has experience working with start-ups as well as enterprises including financial service industries and public and government sectors. He holds various professional certifications from AWS, Red Hat, CNCF, and HashiCorp. He loves to share his experience with the latest technologies at AWS meetups.

When he's not in front of his computer, you might find him playing badminton or golf or trying out new restaurants in town. He enjoys traveling to new places around the world and learning about various cultures.

I must begin my acknowledgment by thanking a couple of people who have had a significant effect on my profession. First and foremost, I want to thank my mentor and friend, Rahul Natarajan, who basically taught me AWS and continues to help me with architecture challenges whenever I get stuck. My former manager, Jason Carter, taught me a lot about application architecture and security. I'd also like to thank Stephen Brown and Gergely Varga for advancing my career by exposing me to AI and global reach. Finally I want to thank my girlfriend Lee Lee who has supported me in this journey.

About the reviewer

Julian Andres Forero is a DevOps consultant at Endava, with more than 7 years of experience in different private and public companies related to media, payments, education, and financial services. He holds a degree in systems engineering and other professional certifications as a Professional and Associate Solutions Architect on Amazon Web Services and as a Terraform Associate Engineer. He has broad experience with cloud architectures and enterprise DevOps frameworks and helps companies to embrace DevOps and site reliability engineering principles. He is the author of various academic articles and has spoken at multiple events on the IT scene. Outside of work, he is an amateur footballer and enjoys visiting new places around the world.

To my partner, for always being there for me, and supporting me when I need it the most. I really appreciate all the moments that we have shared together. I love you.

Table of Contents

Preface

Section 1:Basic CI/CD and Policy as Code

Chapter 1: CI/CD Using AWS CodeStar

Technical requirements

Introduction to CI/CD, along with a branching strategy

CI

CD

Branching strategy (Gitflow)

Creating a project in AWS CodeStar

Introduction to AWS CodeStar

Getting ready

Creating feature and development branches, as well as an environment

Creating feature and develop branches

Creating a development environment and pipeline

Validating PRs/MRs into the develop branch from the feature branch via CodeBuild and AWS Lambda

Adding a production stage and environment

Modifying the pipeline

Summary

Chapter 2: Enforcing Policy as Code on CloudFormation and Terraform

Technical requirements

Implementing policy and governance as code on infrastructure code

Policy as code

Why use policy as code?

Policy as code in CI/CD

Using CloudFormation Guard to enforce compliance rules on CloudFormation templates

CloudFormation Guard

Installation

Template validation

Writing CloudFormation Guard rules

Using AWS Service Catalog across teams with access controls and constraints

AWS Service Catalog

Integrating Terraform Cloud with GitHub

Terraform Cloud

VCS-driven workflow (GitHub)

Running a Terraform template in Terraform Cloud

Writing Sentinel policies to enforce rules on Terraform templates

HashiCorp Sentinel

Summary

Chapter 3: CI/CD Using AWS Proton and an Introduction to AWS CodeGuru

Technical requirements

Introduction to the AWS Proton service

What is AWS Proton?

Creating the environment template bundle

Writing an environment template

Creating the service template bundle

Writing the service template

Deploying the containerized application by creating a service instance in Proton

Creating a source connection (GitHub)

Deploying the application by creating a service instance

Introduction to Amazon CodeGuru

Integrating CodeGuru with AWS CodeCommit and analyzing the pull request report

Summary

Section 2:Chaos Engineering and EKS Clusters

Chapter 4: Working with AWS EKS and App Mesh

Technical requirements

Deep diving into AWS EKS

Kubernetes components

Deploying an EKS cluster

Introducing AWS App Mesh

Are microservices any good?

AWS App Mesh

Deploying an application (Product Catalog) on EKS

Implementing traffic management

Installing the App Mesh controller

Getting observability using X-Ray

Enabling mTLS authentication between services

Summary

Chapter 5: Securing Private EKS Cluster for Production

Technical requirements

Planning your fully private EKS cluster

Creating your EKS cluster

VPC, subnet, and endpoint creation

Bastion server

Creating a cluster

Verifying the cluster access

Deploying add-ons

Creating copies of container images in ECR

IAM roles for service accounts

Cluster Autoscaler

The Amazon EBS CSI driver

Enabling the App Mesh sidecar injector

Kubernetes hardening guidance using Kubescape

Policy and governance using OPA Gatekeeper

Deploying a stateful application using Helm

Backup and restore using Velero

How does Velero work?

Summary

Chapter 6: Chaos Engineering with AWS Fault Injection Simulator

Technical requirements

The concept of, and need for, chaos engineering

Principles of chaos engineering

AWS FIS

Chaos engineering in CI/CD

Experimenting with AWS FIS on multiple EC2 instances with a terminate action

Experimenting with AWS FIS on EC2 instances with a CPU stress action

Experimenting with AWS FIS on RDS with a reboot and failover action

Experimenting with AWS FIS on an EKS cluster worker node

Summary

Section 3:DevSecOps and AIOps

Chapter 7: Infrastructure Security Automation Using Security Hub and Systems Manager

Technical requirements

Introduction to AWS Security Hub

Deny execution of non-compliant images on EKS using AWS Security Hub and ECR

Importing an AWS Config rules evaluation as a finding in Security Hub

Integrating AWS Systems Manager with Security Hub to detect issues, create an incident, and remediate automatically

Summary

Chapter 8: DevSecOps Using AWS Native Services

Technical requirements

Strategy and planning for a CI/CD pipeline

Monorepos versus polyrepos

Feature branch

Develop branch

Staging branch

Master branch

Creating a CodeCommit repository for microservices

Creating PR CodeBuild stages with CodeGuru Reviewer

Creating a development CodePipeline project with image scanning and an EKS cluster

Creating a staging CodePipeline project with mesh deployment and chaos testing with AWS FIS

Creating a production CodePipeline project with canary deployment and its analysis using Grafana

Canary deployment using Flagger

Updating a new version of the service

Summary

Chapter 9: DevSecOps Pipeline with AWS Services and Tools Popular Industry-Wide

Technical requirements

DevSecOps in CI/CD and some terminology

Why DevSecOps?

Introduction to and concepts of some security tools

Snyk – Security advisory for source code vulnerabilities in real time

Talisman – Pre-commit secrets check

Anchore inline scanning and ECR scanning – SCA and SAST

Open Web Application Security Project-Zed Attack Proxy (OWASP ZAP) – DAST

Falco – RASP

Planning for a DevSecOps pipeline

Using a security advisory plugin and a pre-commit hook

Prerequisites for a DevSecOps pipeline

Installation of DAST and RASP tools

Installing OWASP ZAP

Installing Falco

Integration with DevOps Guru

Creating a CI/CD pipeline using CloudFormation

Testing and validating SAST, DAST, Chaos Simulation, Deployment, and RASP

Summary

Chapter 10: AIOps with Amazon DevOps Guru and Systems Manager OpsCenter

Technical requirements

AIOps and how it helps in IT operations

AIOps using Amazon DevOps Guru

Enabling DevOps Guru on EKS cluster resources

Injecting a failure and then reviewing the insights

Deploying a serverless application and enabling DevOps Guru

Integrating DevOps Guru with Systems Manager OpsCenter

Injecting a failure and then reviewing the insights

Summary

Other Books You May Enjoy

Preface

CI/CD has never been simple, but these days the landscape is more bewildering than ever, its terrain riddled with blind alleys and pitfalls that seem almost designed to trap the less-experienced developer. If you're determined enough to keep your balance on the cutting edge and are equipped with a resource like this book, though, the landscape of CI/CD is one that you will navigate with ease.

Accelerating DevSecOps on AWS will help you discover all the most modern ways of building CI/CD pipelines with AWS by placing security checks, chaos experiment, and AIOps stage in pipeline, taking you step by step from the basics right through to the most advanced topics in this varied domain.

This comprehensive guide wastes no time in covering the basics of CI/CD with AWS. Once you're all set with tools such as AWS CodeStar, Proton, CodeGuru, App Mesh, Security Hub, and CloudFormation, you'll dive into chaos engineering, the latest trend in testing the fault tolerance of your system using AWS Fault Injection Simulator. After that, you'll explore the advanced concepts of AIOps using AWS DevOps Guru and DevSecOps, two highly sought-after skill sets for securing and optimizing your CI/CD systems. The full range of AWS CI/CD features will be covered, including the Security Advisory plugin for IDEs, SAST, DAST, and RASP, giving you real, applicable expertise in the things that matter.

By the end of this book, you'll be confidently creating resilient, secure, and performant CI/CD pipelines using the best techniques and technologies that AWS has to offer.

Who this book is for

This book is for DevOps engineers, engineering managers, cloud developers, and cloud architects. All you need to get started is basic experience with the software development life cycle, DevOps, and AWS.

What this book covers

Chapter 1, CI/CD Using AWS CodeStar, introduces the basic concept of CI/CD and branching strategies, then you will create a basic pipeline using AWS CodeStar and enhance it by adding multiple stages, environments, and branching strategies. Doing this will cover all of the AWS developer toolchain, such as CodeCommit, CodeBuild, CloudFormation, and CodePipeline.

Chapter 2, Enforcing Policy as Code on CloudFormation and Terraform, walks through the concept of policy as code and its importance in security and compliance, and the stage of CI/CD at which infrastructure can be checked. You will use CloudFormation Guard to apply policies on an AWS CloudFormation template. After that, you will learn how to use AWS Service Catalog across multiple teams. You will also do hands-on implementation on Terraform Cloud and policy implementation using HashiCorp Sentinel.

Chapter 3, CI/CD Using AWS Proton and an Introduction to AWS CodeGuru, introduces the new AWS Proton service and how AWS Proton helps both developers and DevOps/infrastructure engineers with their work in software delivery. You will learn the basic blocks of the Proton service and create an environment template to spin up multiple infrastructure environments and service templates to deploy the service instance in the environment. This chapter will also walk you through the code review process and how to find a vulnerability or secret leak using AWS CodeGuru.

Chapter 4, Working with AWS EKS and App Mesh, guides you through the architecture and implementation of an AWS EKS cluster. It explains the importance of and need for the AWS App Mesh service mesh and implementing features such as traffic routing, mutual TLS authentication, and using the X-Ray service for tracing.

Chapter 5, Securing Private EKS Cluster for Production, contains an implementation guide to set up a production-grade secure private EKS cluster. It covers almost all the important implementations on EKS, such as IAM Role for Service Account (IRSA), Cluster Autoscaler, EBS CSI, App Mesh, hardening using Kubescape, policy and governance using OPA Gatekeeper, and the backup and restore of a stateful application using Velero.

Chapter 6, Chaos Engineering with AWS Fault Injection Simulator, covers the concept of chaos engineering and when it is needed. It walks through the principles of chaos engineering and gives insights in terms of where it fits in CI/CD. You will learn how to perform chaos simulation using AWS FIS on an EC2 instance, Relational Database Service (RDS), and an EKS node.

Chapter 7, Infrastructure Security Automation Using Security Hub and Systems Manager, includes some important solutions to automate infrastructure security using AWS Security Hub and Systems Manager. The solutions include enforcing only running compliant images from ECR on an EKS cluster, config rule evaluation as an insight into Security Hub, and integrating Systems Manager with Security Hub to detect issues, create an incident, and remediate it automatically.

Chapter 8, DevSecOps Using AWS Native Services, walks you step by step through creating a DevSecOps CI/CD pipeline with a branching strategy using AWS native security services such as CodeGuru Reviewer and ECR image scanning. It includes the powerful combination of the developer toolchain, App Mesh, and Fault Injection Simulator. It also covers the canary deployment of microservices and analysis using Prometheus and Grafana.

Chapter 9, DevSecOps Pipeline with AWS Services and Tools Popular Industry-Wide, walks you through the planning to create a pipeline. It shows how to implement security at every stage of software delivery, starting from when you write code. It also shows the usage of the Snyk Security Advisory plugin in an IDE, git-secrets to scan sensitive data such as keys and passwords, SAST using Snyk, DAST using OWASP ZAP, RASP using Falco, chaos simulation using AWS FIS, and AIOps using AWS DevOps Guru. It also includes operational activities such as showing a security posture and vulnerability findings using AWS Security Hub.

Chapter 10, AIOps with Amazon DevOps Guru and Systems Manager OpsCenter, introduces the primer artificial intelligence and machine learning concepts. It covers what AIOps is, why we need it, and how it applies to IT operations. You will learn about the AWS AIOps tool DevOps Guru and implement two use cases about identifying anomalies in CPU, memory, and networking within an EKS cluster, and analyzing failure insights and remediation in a serverless application.

To get the most out of this book

All the tools used are the latest version while writing the book.

All the tools used in this book are open source or have a trial version that you can subscribe to.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

It is important to have cloud, DevOps, or development work experience to understand the content of the book.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Accelerating-DevSecOps-on-AWS. If there's an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781803248608_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: To verify the policy, we will issue a command in the EKS cluster to run the node:10 image.

A block of code is set as follows:

{

  detail-type: [Config Rules Compliance Change],

  source: [aws.config],

  detail: {

    messageType: [ComplianceChangeNotification]

  }

}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

$ wget https://raw.githubusercontent.com/PacktPublishing/Modern-CI-CD-on-AWS/main/chapter-07/ecr-compliance.yaml

Any command-line input or output is written as follows:

$ docker push .dkr.ecr.us-east-1.amazonaws.com/node:latest

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: Select System info from the Administration panel.

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Accelerating DevSecOps on AWS, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Section 1:Basic CI/CD and Policy as Code

This part includes chapters that cover how to create a CI/CD pipeline using AWS CodeStar with a branching strategy and adding multiple stages and environments. It covers how to leverage the AWS Proton service to create a CI/CD pipeline for applications and infrastructure at scale. It also covers how to avoid any secrets and vulnerabilities in code by integrating AWS CodeGuru Reviewer with CodeCommit. After that, it covers how to enforce policy on infrastructure code using CloudFormation Guard and HashiCorp Sentinel.

This section contains the following chapters:

Chapter 1, CI/CD Using AWS CodeStar

Chapter 2, Enforcing Policy as Code on CloudFormation and Terraform

Chapter 3, CI/CD Using AWS Proton and an Introduction to AWS CodeGuru

Chapter 1: CI/CD Using AWS CodeStar

This chapter will first introduce you to the basic concepts of Continuous Integration/Continuous Deployment (or Continuous Delivery) (CI/CD) and a branching strategy. Then, we will implement basic CI/CD for a sample Node.js application using Amazon Web Services (AWS) CodeStar, which will deploy the application in Elastic Beanstalk. We will begin by creating a CodeStar project, then we will enhance it by adding develop and feature branches in a CodeCommit repository. We will also add a manual approval process as well as a production stage in CodePipeline. We will also spin up the production environment (modifying a CloudFormation template) so that the production stage of the pipeline can deploy the application. After that, we will create two lambda functions that will validate the Pull Request (PR) raised from the feature branch to develop branch, by getting the status of the CodeBuild project. Doing this entire activity will give you an overall idea of AWS Developer Tools (CodeCommit, CodeBuild, and CodePipeline) and how to implement a cloud-native CI/CD pipeline.

In this chapter, we are going to cover the following main topics:

Introduction to CI/CD, along with a branching strategy

Creating a project in AWS CodeStar

Creating feature and development branches, as well as an environment

Validating PRs/Merge Requests (MRs) into the develop branch from the feature branch via CodeBuild and AWS Lambda

Adding a production stage and environment

Technical requirements

To get started, you will need an AWS account and the source code of this repository, which can be found at https://github.com/PacktPublishing/Accelerating-DevSecOps-on-AWS/tree/main/chapter-01.

Introduction to CI/CD, along with a branching strategy

In this section of the chapter, we will dig into what exactly CI/CD is and why is it so important in the software life cycle. Then, we will learn about a branching strategy, and how we use it in the source code repository to make the software delivery more efficient, collaborative, and faster.

CI

Before getting to know about CI, let's have a brief look at what happens in a software development workflow. Suppose you are working independently, and you have been asked to develop a web application that is a chat system. So, the first thing you will be doing is to create a Git repository and write your code in your local machine, build the code, and run some tests. If it works fine in your local environment, you will then push it to a remote Git repository. After that, you will build this code for different environments (where the actual application will run) and put the artifact in the artifact registry. After that, you will deploy that artifact into the application server where your application will be running.

Now, suppose the frontend of your application is not too good, and you want some help from your frontend developer. The frontend developer will clone the code repository, then contribute to the repository either by modifying the existing code or adding new code. After that, they will commit the code and push it into the repository. Then again, the same steps of build and deploy will take place, and your application will be running with the new User Interface (UI). Now, what if you and the frontend developer both want to enhance the application, whereby both of you will be writing the code, and somehow you both used the same file and did your own changes, and tried to push back to the repository? If there are no conflicts, then your Git repository will allow you to update the repository, but in case there are any conflicts then it will highlight this to you. Now, once your code repository is updated, you must again build the code and run some unit tests. If the tests find a bug, then the build process will fail and you or the frontend developer will need to fix the bug, and again run the build and unit test. Once this passes, you will then need to put the build artifact into the artifact registry and then later deploy it into the application server. But this whole manual process of building, testing, and making the artifact ready for deployment will become quite troublesome and slow when your application gets bigger, and collaborators will increase, which in return will slow the deployment of your application. These problems of slow feedback and a manual process will easily put the project off schedule. To solve this, we have a CI process.

CI is a process where all the collaborators/developers contribute their code, several times a day, in a central repository that is further integrated into an automated system that pulls the code from the repository, builds it, runs unit tests, fails the build, gives feedback in case there are bugs, and prepares the artifact so that it is deployment-ready. The process is illustrated in the following diagram:

Figure 1.1 – CI process

CI makes sure that software components or services work together. The integration process should take place and complete frequently. This increases the frequency of developer code commits and reduces