ANSIBLE ALL THE THINGS

From traditional to unorthodox, Ansible for

Everything
Adam Miller Nicolas FANJEAU
Principal Software Engineer Airbus Infrastructure
Red Hat Summit 2017
AGENDA
AGENDA
WHAT WE’RE GOING TO TALK ABOUT TODAY

● Quick intro to Ansible (just in case) ● Command Line Tooling

● Why on earth would I want to do all the ● Event Based Execution
things with Ansible? ● Workflow Automation
● Automation Tool ● CI/CD
● Configuration Management ● Ansible Container
● Provisioning and Systems ● Ansible Tower
Management ● Case Study: Airbus
● Deployment
● Application Lifecycle Management
● Orchestration
WHAT IS ANSIBLE?
QUICK INTRODUCTION
WAIT, YOU DON’T KNOW WHAT ANSIBLE IS?

Ansible is an automation tool

● Ansible is a simple agentless idempotent task automation tool
○ By default, tasks are executed in-order but we can change that if we want.
● Tasks are performed via modules
● Tasks are grouped together via plays
○ Also via roles, but more on that later
○ A play operates on a set of hosts
● Playbooks can contain one or many plays
○ Can be used with "traditional" configuration management systems
■ There's even a puppet module!
QUICK INTRODUCTION
BEST THING SINCE SLICED BREAD

● Example of an ad-hoc ansible orchestration task

○ Module: yum
○ Arguments: pkg=bash state=installed

$ ansible localhost -m yum -a "pkg=bash state=installed"

localhost | SUCCESS => {
"changed": false,
"msg": "Nothing to do"
}

● What if I wanted to do more than one thing? Playbooks!

BUT FIRST… INVENTORY
INVENTORY
KEEPING TRACK OF YOUR MARBLES… ERR SYSTEMS

Inventory to defines hosts and groups of hosts

○ Special "all" group that is implicitly defined as the sum of all hosts in your inventory.
○ Also, “localhost” is a built-in and does not need to be defined
● Example:
○ Below we have a simple inventory with two groups, appservers and webservers.

[appservers]
app1.example.com
app1.example.com

[webservers]
webserver1.example.com
webserver2.example.com
PLAYBOOKS AND ROLES
PLAYBOOKS
DOING STUFF AND THINGS

Playbooks are a way to combine many tasks, written in YAML, to be carried out against one
or many hosts.
---
- name: common things to run on all hosts - name: webserver-only tasks
hosts: all hosts: webservers
tasks: tasks:
- name: make sure bash is installed - name: start and enable httpd service
yum: service:
pkg: bash name: httpd
state: installed state: started
enabled: yes
INCLUDES
DON’T JUST COPY/PASTE … COWSAY IS WATCHING

Include file defines a set of tasks that can be included by a playbook, this allows sharing
sets of tasks without copy/pasting everywhere.

enablewebservice.yml webserver.yml
--- ---
- name: start and enable httpd - name: Webserver Playbook
service: hosts: webservers
name: httpd tasks:
state: enabled - include: enablewebservice.yml ____________________________
< Don't copy/paste, include! >
----------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
Playbooks can also include other playbooks! || ||
ROLES
YOUR MOM WAS RIGHT, IT’S BETTER TO SHARE

Roles are reusable logical groupings of tasks that (normally) define Typical Role Layout
a service
● Role-level subdirs for namespaced variable defaults, files, myrole/
├── defaults
templates, and handlers │ └── main.yml
● Can pass variables to roles to modify behavior per-use ├── files
├── handlers
● Searched for and/or shared via Ansible Galaxy │ └── main.yml
○ https://galaxy.ansible.com/ ├── tasks
│ └── main.yml
--- ├── templates
- name: using myrole └── vars
- hosts: webservers └── main.yml
- roles:
- myrole
WHAT IS ANSIBLE?
USING ANSIBLE FOR EVERYTHING
WHY WOULD I WANT TO DO THAT?

Ansible is a simple automation tool that can:

● Execute tasks on one or many hosts
● Orchestrate an otherwise complex order of operations, even conditionally based on
system facts or variables provided at runtime.
● Custom modules can be written in any programming language with JSON support

Question of the day:

What are you trying to accomplish that could be automated?

USING ANSIBLE FOR EVERYTHING
ANSIBLE ALL THE THINGS!!!!

What are you trying to do?

● Configuration Management?
● Provision Virtual Machines or IaaS instances?
● Test software?
● Automate workflows?
● Continuous Integration / Continuous Deployment?
● Configure hardware switches, routers, and load balancers?
● Replace terrible shell scripts that have survived too long already?
● Other?
ANSIBLE CAN DO ALL OF THAT! (AND MUCH MORE)
ANSIBLE DOES THAT
CONFIGURATION MANAGEMENT
KEEPING THE TRAIN ON THE TRACKS

What is configuration management?

Systems engineering process for establishing and maintaining consistency of a product's
performance, functional, and physical attributes with its requirements, design, and operational
information throughout its life.

Generally boils down to:

● Managing file content
● Configuration Templating
● System and Service state
● Package Management
● Lifecycle Management
ANSIBLE DOES THAT
OMG, NO WAY?!?!?!

● Service state: service module

● Files and configuration modules: acl archive assemble blockinfile copy fetch file find
ini_file iso_extract lineinfile patch replace stat synchronize tempfile template unarchive xattr
● System state modules: aix_inittab alternatives at authorized_key beadm capabilities cron
cronvar crypttab debconf facter filesystem firewalld gconftool2 getent gluster_volume group hostname
iptables java_cert kernel_blacklist known_hosts locale_gen lvg lvol make modprobe mount ohai
open_iscsi openwrt_init osx_defaults pam_limits pamd parted ping puppet runit seboolean sefcontext
selinux selinux_permissive seport service setup solaris_zone svc sysctl systemd timezone ufw user
● Package Management modules: bower bundler composer cpanm easy_install gem maven_artifact
npm pear pip apk apt apt_key apt_repository apt_rpm dnf dpkg_selections homebrew homebrew_cask
homebrew_tap layman macports openbsd_pkg opkg package pacman pkg5 pkg5_publisher pkgin pkgng pkgutil
portage portinstall pulp_repo redhat_subscription rhn_channel rhn_register rpm_key slackpkg sorcery
svr4pkg swdepot swupd urpmi xbps yum yum_repository zypper zypper_repository

More modules being added all the time...

ADVANCED CONFIGURATION
MANAGEMENT
THAT LITTLE EXTRA

The following categories of Infrastructure Needs are covered extensively by Ansible

modules:
● Clustering ● Network
● Commands ● Notification
● Crypto ● Packaging
● Database ● Remote Management
● Files ● Source Control
● Identity ● Storage
● Inventory ● System
● Messaging ● Utilities
● Monitoring ● Web Infrastructure
PROVISIONING
MAKING SOMETHING FROM NOTHING

What do you want to accomplish?

● Create IaaS compute instances, object stores, or ephemeral resources?
● Provision virtual machines?
● Create storage allocations?
● Set firewall rules?
● Configure highly available load balancers?
● Create VLANs?
● Deploy container orchestration resources?
● Create databases?
● Other?
ANSIBLE CAN DO THAT
WHAT? AGAIN? NO WAY!!

Provisioning support for many IaaS providers: Datacenter and Virtualization:

● Amazon Web Services ● oVirt / RHV
● Apache CloudStack ● libvirt resource management
● Centurylink Cloud ● Joyent SmartOS Virt
● Digital Ocean ● VMWare (VSphere/ESXi)
● DimensionData
Storage:
● Google Cloud
● Linode ● AIX LVM
● Microsoft Azure ● Gluster Volume
● OpenStack ● Infinidat
● Rackspace Public Cloud ● LVM2
● Softlayer Webfaction ● NetApp
● ZFS
PROVISIONING - CONTINUED
OMG, THIS LIST JUST KEEPS GOING…

Networking ● Netvisor
● Open vSwitch
● A10 Networks
● Palo Alto Networks PAN-OS
● Apstra AOS
● Nokia SR OS
● Arista EOS
● VyOS
● Avi Networks
● BigSwitch Databases
● Cisco (ASA, IOS/IOS-XR, and NX-OS)
● InfluxDB
● Cumulus Networks (Cumulus Linux)
● Redis
● Dell EMC (OS6, OS9, and OS10)
● Riak
● F5 BigIP ● MS-SQL
● Fortios Firewall
● MySQL
● JunOS
● Postgresql
● Lenovo CNOS
● Vertica
PROVISIONING - CONTINUED
SERIOUSLY? MORE STUFF?

Web Infrastructure and Clustering

● Apache HTTPD (module and mod_proxy management) ● ZooKeeper
● Consul
● Django Management
● eJabberd
● htpasswd
● JBoss
● Jenkins (Jobs, Plugin, and Jenkinsfile management)
● Jira
● Kubernetes
● Letsencrypt
● Pacemaker
● Supervisord
DOING THINGS WITH ANSIBLE
DEPLOYMENT
I JUST GIT PUSH TO THE CLOUD, RIGHT?

Software Deployment is the act of making software available on systems; most often, this is
a sequence of steps that must be performed in-order. (In-order task execution anyone?)

Example:
● Sync some data
● Database schema migration
● Remove systems from load balancer
● Push new code
● Put systems back in load balancer
○ Rinse/Repeat on previously not upgraded set
● Verify services are functional
● Status update
Remember what a Playbook does?
APPLICATION LIFECYCLE MANAGEMENT
DO IT LIVE!

Managing application lifecycle across one or many hosts

● Ansible can orchestrate both simple and complex lifecycle management
● Lifecycle “order of operations” defined in Playbooks
○ Whatever your requirements are
● Plays can execute on different sets of hosts
○ Multiple plays per playbook
● Plays can use varying execution strategies for various requirements
○ Cluster node management
○ Database schema updates
○ etc
● Sky is the limit
○ (something something … cloud)
ORCHESTRATION AND WORKFLOW
AUTOMATION WITH FEELING

Flow controlled automation by data from the environment allowing the automation tasks to
make “intelligent” decisions.
COMMAND LINE TOOLING
BUT WHAT ABOUT MY PERL ONE-LINERS?

Make Ansible your new command line tooling API, stop re-inventing the wheel
● Ansible provides a very capable Python API for modules
● Modules can be written in any programming language that understands JSON
● Provides a consistent “UX” for all tasks
● Gives you and your ops team an “on ramp” to scaling your tasks across the
infrastructure

$ ansible localhost -m my_task -a “arg1=foo arg2=bar”

EVENT BASED EXECUTION +-----------------+
| |
+---------------+
| |
COWSAY WHAT? | Events +------>| Looper |
| | | (plugin) |
| | | |
+-----------------+ +---------------+
|
+-------------------+ |
Ansible can easily integrate with existing infrastructure to perform | | |
| | |
actions based on events. | Loopabull +<---+
| (Event Loop) |
| |
● Example: loopabull +---------+---------+
|
○ Events in the infrastructure spawn messages on the bus V
+----------+-----------+
○ loopabull listens on the bus, waiting for a “routing key” | |
| ansible-playbook |
that it cares about (message topic) | |
+----------------------+
○ Message payload is injected into Ansible playbooks as
variables, allowing for decisions to be made based on
message contents
CONTINUOUS INTEGRATION
THERE IS ONLY ZUUL … (BUT ALSO OTHER STUFF)

Brief story of OpenStack Zuul and Jenkins Job Builder

● OpenStack CI System (Zuul) - http://status.openstack.org/zuul/
○ 2,000+ jobs-per-hour
■ single-use OpenStack VMs -> create and destroy 2K+ VMs per hour
○ 1731 git repositories to perform gating on
○ Spread across 7 public OpenStack clouds and 4 private OpenStack clouds
■ Hybrid cloud anyone?
● OpenStack wanted to not fiddle with XML for Jenkins Jobs
● Jenkins Job Builder (YAML) was created
● Jenkins Performance issues ran into…
● No more Jenkins, automatically convert JJB YAML into Ansible Playbooks
● Future: Migrate entirely away from JJB, make it all Ansible!
MORE CONTINUOUS INTEGRATION
THE OTHER STUFF

Fedora Taskotron - https://taskotron.fedoraproject.org/

● CI for the entire Fedora Linux Distribution
● “Tasks” definitions originally in YAML
● Tasks for every RPM, ISO, VM Image, Container, etc in the distro
● Automated reporting to the Fedora Updates System (Bodhi)
● Migration from Taskotron YAML to Ansible Playbooks
ANSIBLE CONTAINER
END THE DOCKERFILE MADNESS

Using Ansible playbooks to build you container images

● Stop chaining together shell commands in Dockerfiles
● Create containers the same way you deploy to servers
● roles == services, build your containers using roles
○ Making single-purpose (microservice) containers easy
● Deploy to Container Orchestration Platforms
○ Currently Supports OpenShift and Kubernetes
ANSIBLE TOWER
PRETTY GRAPHS!

The definitive Ansible Centralized Management Portal

● Role Based Access Control
● Centralized Logging, History Visualizations
● Multi-Playbook Workflow Orchestration
● Playbook and System Auditing (System Tracking)
● Self-Service Automation
○ Sanitized form-based playbook runs
● Integrated Notifications (ChatOps, etc)
● REST API
● … and much much more!
ANSIBLE @ Airbus
Automation from End2End

Nicolas FANJEAU
Airbus Infrastructure
Red Hat Summit 2017
Airbus

Passion

-Our global workforce is

united by a passion for
aviation and restless
55,000 €45,8billion 10yrs 400 desire to create better
ways to fly
Employees Annual revenue* Backlog Operators
Information & Communication Technology
Central &
Operational Teams

Moscow
Fuhlsbüttel
Filton Stade Hamburg
Broughton Bremen Buxtehude
St. Nazaire Nantes
Blagnac
Wichita Ashburn Barajas St. Martin
Getafe Beijing
Washington Tianjin
Mobile
Miami Abu Dhabi Dubai

Bangalore

1300 Information System professionals located around the

world wherever Airbus operates. Data to end 2015
 Airbus IT Infrastructure
Suppliers Airbus Airbus Group Customers TOTAL

106 000 users 96 000 users 94 000 users 72 000 users 368 000

21 000 PCs 61 000 PCs 5 000 PCs 87 000

33 000 mailboxes 77 000 mailboxes 34 000 mailboxes 144 000

6 600 printers 6 600

75 000 fixed phones 75 000

32 400 mobile phones 600 mobile phones 33 000

433 000 network ports 5 000 WiFi access points

19 billions transactions
13 000 Servers per year on SAP
4 200 MIPS on Mainframe Data to end 2015

3 17 petabytes on storage 1,2 petaFLOPS on High

Page 37

7 IT Services a Endperformance computing

User Self Service Solution
Open Source at Airbus
Embraces the open way of working

• Improve the motivation and efficiency of our

people and make IT more attractive • Further increase our speed
through: of change
- Transparency - Sharing • Align with the digitalization
- Collaboration - Empowerment initiatives
Boosts the use of Open Source software

• Get classical Open Source advantages (lower TCO, quicker

Use the implementations, better quality & security etc..
opportunity to
• Reduce our dependency from classical software suppliers
• Increase innovation, as in several areas Open Source Software solutions
are more advanced (Cloud, Big Data…)

A Project
• Solves the IT Service
Management (ITSM)
Use the „dilemma“ and reduces the
opportunity to number of tools
Our needs
Functional Solution

Application Infrastructure Maintenance

Entry in Service
Robust
Library

Maintenance Integrated
Job Scheduler Scalable
Deployment
Public Cloud
High Availability
Linux Private Cloud
Secure
Windows
Interoperable
Plug and play Cost
Segregation
Reporting Agent less
 Automation as Self Service
EXPECTATIONS
• Reduce time and cost to deploy application
• Move to DevOps philosophy
• Give back the responsibility to Application Owner
• Simplify process

SOLUTION
• Propose customer oriented service for Automation
• Develop the service for and with the customers
• Propose tailored solutions to all customers via a
catalogue of services
• Awareness on Automation
• Training : Platform usage, How to implement Playbook
• Playbook On Demand, conversion of Install Manual to Playbook
• eLearning, User Manual, Best practices
 From the PoC to the Project
PoC
• Objective is to evaluate the solution Key Figures
• Test the deployment of 5 applications (Win & Linux)
with 6 automation solutions PoC on 100 Hosts
Result
• Despite missing functionalities of Tower vs Target 10 000 Hosts
Competitors, Tower finish first one based on the
criteria matrix First deployment 6 months
• Deployed in Production during the PoC for two
critical applications for
• Release deployment
• Job scheduler
10 000 hosts
10/2016 02/2017 2000 hosts 6 000 hosts
04/2017
PoC Decision Start deployment 2018 2019

Data to end 2015

 Target
• Hosts
• Windows 9 400 W2k8 4100 W2k12 3500
• Linux 5 900 RHEL5 1800 RHEL6 2500 RHEL7 700 Key Figures
• Unix 3 600
• Deployed of dedicated Tower infrastructure depending of Applications 2000
• Location of the Data Center
Germany, France, United Kingdom & Spain… Users 1 000
• Environment
Deployment Infra 2 months
Integration, Validation, Production, DMZ,
Public Cloud...
• Common architecture base on
• Tower, Cluster of two nodes
• PostgreSQL, Cluster of two nodes
• Virtual Machine, RHEL 7
Next, Automation from End to End Consumer

Full automation from the request to the

Catalog Products
delivery
Request
Be user centric and inforce self service usage
Approval
Propose a single catalogue and point to
Change
aggregate all the products

Use the Tower CLI

Task

Fully integrated with ITSM tool to avoid to

Tower
data duplication and interfaces
CMDB
In line with the ITIL best practices Hosts

Data to end 2015

 Key Success Factors
Open Source is a key solution to ensure
innovative application and quick delivery

Involvement of customers in the development

of the solution is a key of the success

A lot of communication & change support to

get users adopt the situation

Self-service is the requirement to reach

customers’ satisfaction and meet company’s
objectives

Data to end 2015

THANK YOU
plus.google.com/+RedHat facebook.com/redhatinc

linkedin.com/company/red-hat twitter.com/RedHatNews ADAM MILLER

maxamillion
youtube.com/user/RedHatVideos
maxamillion

@TheMaxamillion