DevOps Security

John

These are some notes regarding DevOps and security

The DevOps Tool Chain

A measurement for successful DevOps (CALMS)

  • Culture
    • Accept failure
    • Collaboration occurs widely
    • Empower teams
    • High levels of trust
  • Automation
    • Tools
    • Continuously iterate
    • Transparency
    • Consistency
    • Everything as Code
      • Be very good at git
      • scripts, runbooks, configurations, documentation, …
  • Lean
    • Confront bureaucracy
    • Efficiency
    • Leverage automation
    • Just in time prioritization
  • Measurement
    • Measure everything
    • How many vulnerabilities got released
    • What percent of code tested / code coverage tools
    • MTTR (mean time to recover)
    • MTTF (mean time to failure)
  • Sharing
    • When sharing vulnerability data people dont know risk

 

Case Study – Doing DevSecOps at scale

Amazon is an extreme example of DevOps at scale

  • Frequent deployments, thousands per day
  • Thousands of engineers in autonomous two-pizza teams building microservices
  • “You built it, you run it” – developers write, review, deploy and support

Microsoft become leader in sharing resources and tools

  • Secure Development Lifecycle (SDL)
  • Commitment to open-source (purchase Github in 2018)
    • Largest contributor to github, no longer secluded, secret
  • Bug bounty programs, black hat initiatives
  • MSRC – Microsoft Security Response Center, works with researchers and publishers
  • Popular threat modeling, static analysis security tools

 

DevOps Toolchain

Must understand principles, practices, patterns and tools.

  • If teams already adopted DevOps, understand what and how they doing it
  • If team transitioning to DevOps, map out steps to take how to build security early
  • Continuous Delivery is complicated, dont jump too quickly. Think incrementally and dont get in the way of delivery unless absolutely necessary

 

Version Control / Source Code Management

Key players are Github, Azure DevOps, GitLab, Bitbucket – all based on git

GitFlow Workflow – contribute to development and operations workflows. Prevents attackers from compromising the workflow.

 

Continuous Integration (CI)

Approving and completing a merge request creates a commit on the “main” branch and automatically triggers a pipeline.

Builds, tests and runs tools, then gives fast feedback. If the pipeline is broken, developer can address the issue and make another commit.

Continuous Delivery (CD)

Pipeline model and control extending continuous integration. Performs actual deployment and releases.

Some characteristics of CD

  • Changes are deployed directly to production by developers
  • Blue/Green Deployment – changes switch between production environments using a load balancer
  • Canary Deployment – pushed out one node at a time
  • A/B Testing – routes certain users to certain locations, usually divided by geographical locations
  • Dark Launching

 

Continuous Integration Responsibility Model

Today a lot organizations are using cloud hosted CI/CD tools. This was not the case just a few years ago where Jenkins was the dominate tool.

 

Github

Github has become a full CI/CD using Github Repository.

  • YAML pipeline templates stored in the .github/workflows
    • ex: .github/workflows/main.yaml
  • An event triggers one-to-many job(s) that execute on a runner
  • Github hosted runners support Ubuntu, macOS, Windows vm
  • There is a free tier for Free, Pro, Team, and Enterprise

Github Actions marketplace hosts thousands of custom extensions

  • Cloud API integrations (AWS, Azure)
  • Container security
  • Code Analysis tools
  • Component analysis
  • Dynamic security testing

Environment Secrets

  • CI/CD systems should be storing secrets.
    • Bearer Tokens
    • API Keys
    • Database credentials
  • Github calls these Action Secrets
    • Secrets can be stored at project or organization level

GitLabs

GitLab CI/CD provides an integrated way for CI/CD workflows on push events to use GitLab repository. GitLab runs on GCP.

  • YAML pipeline templates stored in .gitlab-ci.yml file
    • Can use terraform
  • GitLab runners support Linux GCE with latest CoreOS and Docker
  • GitLab pipelines rely heavily on ephemeral Docker image builders

GitLab maintains CI/CD templates for numerous frameworks and tools

  • Auto DevOps detects the languages by a repository and automatically runs applicable templates
  • It supports Kubernetes, AWS ECS, EC2..
  • Embedded support for security code, containers, dynamic scanning
  • Secrets Management

 

Attacks to the DevOps pipelines

Version control systems provide the entry point for building and releasing changes through Continuous Integration (CI) and Continuous Delivery (CD) pipelines. Attackers compromising the version control system will inject malicious code into the source code repository, and hope the CI / CD system releases the malware into a production environment.

  • Threat actor compromises a user’s version control credentials and creates an attacker controlled SSH key for accessing source code.
  • Threat actor makes an unauthorized commit to main branch, triggering a new release to production that contains a backdoor or trojan.
  • Threat actor modifies a high risk code file (e.g. IaC, cryptography library, authentication module, etc.) without approvals from security team. Releasing changes to high risk code without approvals may violate separation of duties and change approval board requirements.

Continuous Integration (CI) and Continuous Delivery (CD) servers are responsible for building and delivering changes across environments (e.g. development, staging, and production). For CI / CD servers to perform these critical tasks, pipeline jobs often run under highly privileged service accounts with access to secrets, credentials, and API keys. Attackers compromising a CI / CD server can leverage these characteristics to back door production systems.

  • Threat actor discovers the CI / CD server (Jenkins) exposed on the Internet, and exploits a vulnerable plugin.
  • Threat actor compromises a CI / CD workflow job and injects malicious code into the build artifacts.

Continuous Integration (CI) and Continuous Delivery (CD) servers typically connect to Secrets Management systems, such as HashiCorp Vault, to obtain credentials for accessing cloud APIs and provisioning secrets to workloads. Secrets Management systems hold the crown jewels for the DevOps toolchain and are usually an attacker’s primary target.

  • Threat actor compromises a secrets manager credential by creating a malicious job on the CI / CD server. The secrets manager credential is used to gain access to secrets, code signing certificates, deployment keys, etc. intended for other pipeline jobs.

 

Recent Supply Chain Attacks:

  • Solarwinds
    • System passwords exposed in GitHub repo
    • Malicious code inserted into JetBrains TeamCity CI build
    • Malware signed by the build pipeline and shipped to customer
  • Codecov
    • Tool runs in CI/CD pipeline to see how tests performing
    • Docker image contains GCS credentials
    • Malicious bash uploader script put into GCS bucket
    • Customer CI/CD pipelines execute malicious script and exfiltrate secrets, API keys, code coverage

 

Guidelines to harden CI/CD Pipelines

  • Restrict control flow into prod
  • Limit service account  permissions, long lived credentials
  • Protect the supply chain with allow lists of trusted actions (plugins, packages)
  • Review all changes to workflow files for malicious code execution
  • Patch self hosted CI/CD runners and software aggressively
  • Include audit logs in network and operations monitoring

Some other guidelines

  • Leverage MFA when accessing any of the tools
  • Protect SSH keys
  • Protect branches (block the PRD branch). Must have gates like peer reviews, approvers, etc
  • Protect Project Hooks
  • Manage Jenkins server, or any other server hosting the CI/CD tools
  • Patch underlying servers and the tools
  • Use secret vaults as appropriate

 

Securing Workflows

There are 5 key phases in DevOps workflow. We need to address security in each of these phases

  • Pre-commit
    • Threat Modeling
    • IDE Security Plugins
    • Pre-commit hooks
    • Peer code review
  • Commit (CI)
    • Static Code Analysis
    • Security Unit Tests
    • Dependency Management
    • Container Security
  • Acceptance (CD)
    • Dynamic Security Test
    • Acceptance Test
    • Infrastructure as code
    • Config Management
  • Production (Preparing for PRE)
    • Server hardening
    • Runtime protection
    • Secrets management
    • Safety Checks
  • Operation (continuous monitoring)
    • Blameless Postmortems
    • Continuous Monitoring
    • Fire Drills
    • Threat Intelligence

Centralized Security Scanning Factory

Build a pipeline for security teams to manage requests for work

  • Penetration tests, code/design reviews, training, deep scanning, tool building/integration, audits
  • Leverage workflow automation
  • Shows how security teams can integrate with CI/CD
  • Centralizes security scanning intelligence for analysis in SIEM systems
  • Validate findings and create tickets
  • Scales across enterprise for hundreds/thousands of pipelines

 

Pre-Commit Security Controls

Pre-Commit Security controls include:

  • Threat modeling = incremental design review, repid risk assessment, etc
  • IDE Security Plugins = code editor static analysis and linting plugins
  • Pre-commit hooks = commit hooks to check for embedded secrets and enforce review workflows
  • Peer code review = security code review
  • Branch protections
  • CODEOWNERS
    • A file in the repo (or defined in the DevOps tool like Azure DevOps) assigning which files or folders are owned by whom. Therefore when changes happen in that area, those people are contacted for review and approval.

Rapid Risk Assessments

Code Analysis: IDE Security Extensions

  • VS Code Marketplace
    • Semgrep/DevSkim/PumaScan
    • Checkov/CFN Nag
    • SonarLint/ESLint
    • Hadolint
    • InSpec

Git Commit Workflow hooks

  • Local repository: pre-commit, prepare-commit, commit, post-commit, post-checkout
  • Remote repository: pre-recieve, update, post-receive

Pre-Commit Framework

  • Standardize tools so it can be used for all repos and local dev environments
  • Add a .pre-commit-config.yaml in repository
  • Configure hooks
  • Create default hook
  • Rerun and verify hooks in ci/cd pipelines

 

Commit Security Controls

 

 

Secrets Management

 

 

 

References

eof