AdvancedAutomation

CI Shell Scripts

Write portable, robust CI scripts with the right shebang, set -euo pipefail, idempotent steps, environment-variable secrets, and matrix build patterns.

BashAdvanced12 min read

Recommended first

Makefiles

By the end of this lesson you will be able to:

Use
Apply set -euo pipefail as the standard strict-mode preamble
Write idempotent steps that can be re-run safely
Pass secrets via environment variables, never command-line arguments
Structure matrix build scripts that work across platforms

A CI/CD script runs in a fresh environment on every commit, across multiple operating systems and container images, with no human watching. The rules for writing these scripts are stricter than for interactive use: portability matters, failure modes must be explicit, secrets must be handled carefully, and every step should be idempotent — safe to re-run without side effects.

The shebang: #!/usr/bin/env bash

A script's shebang line determines which interpreter runs it. /bin/bash is hardcoded; on macOS and some Linux distributions, bash may live elsewhere or only an old version may be at /bin/bash. The portable form:

#!/usr/bin/env bash

env searches PATH for bash, finding whatever version is installed — including Homebrew's Bash 5 on macOS, or a system-managed version in a container.

Never use #!/bin/sh for scripts that use Bash features. sh may be dash, ash, or POSIX sh, none of which support arrays, [[ ]], declare -A, or other Bash extensions. If you use Bash features, declare it: #!/usr/bin/env bash.

Strict mode as standard preamble

Every production CI script should open with:

#!/usr/bin/env bash
set -euo pipefail

You covered this in the intermediate tier. In CI context, it matters more:

set -e makes any unexpected failure abort the pipeline step immediately with a clear non-zero exit code, triggering a CI failure
set -u catches typos in variable names, which in CI are often missing environment variables
set -o pipefail surfaces failures in pipeline steps like curl | jq where the curl could fail silently

Idempotent steps

An idempotent operation produces the same result whether run once or many times. CI pipelines are re-run on retries, on rebuilt artifacts, or as part of a re-deploy. Writing idempotent steps prevents "I already did this" errors:

# Not idempotent — fails if the directory already exists
mkdir /opt/myapp

# Idempotent
mkdir -p /opt/myapp

# Not idempotent — adds the user again every run
useradd deploy

# Idempotent
id deploy &>/dev/null || useradd --system --no-create-home deploy

# Not idempotent — appends to config file on every run
echo "DEPLOY_ENV=production" >> /etc/environment

# Idempotent — only writes if not already present
grep -q "DEPLOY_ENV=production" /etc/environment || \
  echo "DEPLOY_ENV=production" >> /etc/environment

Secrets via environment variables

Never pass secrets on the command line — command-line arguments are visible in ps aux, shell history, and process listings. Use environment variables, set via your CI platform's secrets mechanism:

# Wrong — secret visible in process list
curl -H "Authorization: Bearer my-secret-token" https://api.example.com

# Right — read from environment, set as a CI secret
curl -H "Authorization: Bearer ${API_TOKEN}" https://api.example.com

Validate required secrets early — fail fast if they are missing:

#!/usr/bin/env bash
set -euo pipefail

# Validate required secrets at startup
: "${API_TOKEN:?API_TOKEN must be set}"
: "${DEPLOY_HOST:?DEPLOY_HOST must be set}"
: "${DEPLOY_USER:?DEPLOY_USER must be set}"

echo "All required secrets are present"

Mask secrets in CI logs. GitHub Actions, GitLab CI, and CircleCI automatically mask values registered as secrets so they don't appear in logs. But set -x trace mode will still print them — disable set -x around any command that uses a secret, or use { set +x; cmd; set -x; } 2>/dev/null.

Logging and tracing in CI

Structured logging helps diagnose failures in CI logs. Add a simple logger:

#!/usr/bin/env bash
set -euo pipefail

log()  { echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') [INFO]  $*"; }
warn() { echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') [WARN]  $*" >&2; }
die()  { echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') [ERROR] $*" >&2; exit 1; }

log "Starting deployment to ${DEPLOY_HOST}"

For CI platforms that support it, use group/section commands to fold long output:

# GitHub Actions
echo "::group::Installing dependencies"
npm ci
echo "::endgroup::"

# GitLab CI
echo -e "section_start:$(date +%s):install\r\e[0KInstalling dependencies"
npm ci
echo -e "section_end:$(date +%s):install\r\e[0K"

Matrix builds

Many projects need to test across multiple versions or platforms. A matrix build variable makes the script adapt:

#!/usr/bin/env bash
set -euo pipefail

PYTHON_VERSION=${PYTHON_VERSION:-3.11}
OS=${OS:-ubuntu-latest}

log() { echo "[$(date +%T)] $*"; }

log "Running matrix build: Python ${PYTHON_VERSION} on ${OS}"

# Use the matrix variable to select behaviour
case "$PYTHON_VERSION" in
  3.8|3.9)
    log "Legacy Python — running compat tests"
    pytest tests/compat/
    ;;
  3.10|3.11|3.12)
    log "Modern Python — running full test suite"
    pytest tests/
    ;;
  *)
    die "Unknown Python version: $PYTHON_VERSION"
    ;;
esac

Check your understanding

Do it yourself

# Create a sample CI script and test it
cat > /tmp/ci-demo.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

log()  { echo "[$(date +%T)] [INFO]  $*"; }
die()  { echo "[$(date +%T)] [ERROR] $*" >&2; exit 1; }

# Validate required env var
: "${BUILD_ENV:?BUILD_ENV must be set}"

log "Starting build for environment: $BUILD_ENV"

# Idempotent directory creation
mkdir -p /tmp/ci-output

# Write a result
echo "build=$BUILD_ENV timestamp=$(date -u +%s)" > /tmp/ci-output/result.txt
log "Build complete. Output:"
cat /tmp/ci-output/result.txt
EOF

chmod +x /tmp/ci-demo.sh

# Test with required var
BUILD_ENV=staging /tmp/ci-demo.sh

# Test failure (missing required var)
/tmp/ci-demo.sh || echo "Failed as expected: BUILD_ENV not set"

Where to go next

Your CI scripts are now production-grade. The final lesson — Debugging scripts — covers set -x trace mode, bash -n syntax checking, PS4 customisation, and shellcheck to find bugs before they reach production.

Finished reading? Mark it complete to track your progress.

Makefiles

Use make as a language-agnostic task runner — targets, prerequisites, recipes, .PHONY, automatic variables ($@, $<, $^), and pattern rules.

Debugging Scripts

Use set -x trace mode, bash -n syntax checks, PS4 customisation, shellcheck, and quoting/word-splitting analysis to find and fix Bash bugs.

CI Shell Scripts

Check your understanding

On this page