GitLab CI/CD

      Ankush More

      Complete Guide from Beginner to Expert

      Table of Contents

      1. Introduction to GitLab CI/CD
      2. Getting Started
      3. GitLab CI/CD Fundamentals
      4. Pipeline Configuration
      5. Jobs and Stages
      6. Variables and Environment Management
      7. Docker Integration
      8. Testing Strategies
      9. Deployment Patterns
      10. Advanced Features
      11. Security and Best Practices
      12. Monitoring and Optimization
      13. Troubleshooting
      14. Real-World Examples

      1. Introduction to GitLab CI/CD

      What is CI/CD?

      graph LR
    A[Code Commit] --> B[Continuous Integration]
    B --> C[Automated Testing]
    C --> D[Code Quality Checks]
    D --> E[Continuous Deployment]
    E --> F[Production Environment]

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style C fill:#e8f5e8
    style D fill:#fff3e0
    style E fill:#fce4ec
    style F fill:#f1f8e9

      Continuous Integration (CI): The practice of automatically building and testing code changes as they’re committed to version control.

      Continuous Deployment (CD): The automated deployment of tested code to production environments.

      GitLab CI/CD Architecture

      graph TB
    A[GitLab Repository] --> B[GitLab Runner]
    B --> C[Pipeline Execution]
    C --> D[Build Stage]
    C --> E[Test Stage]
    C --> F[Deploy Stage]

    D --> G[Artifacts]
    E --> H[Test Reports]
    F --> I[Production Environment]

    subgraph "GitLab Instance"
        A
        J[Web Interface]
        K[API]
    end

    subgraph "Runner Environment"
        B
        L[Docker]
        M[Shell Executor]
        N[Kubernetes]
    end

      Benefits of GitLab CI/CD

      • Automation: Eliminates manual deployment processes
      • Quality Assurance: Automated testing catches bugs early
      • Speed: Faster time-to-market
      • Consistency: Standardized deployment processes
      • Visibility: Clear pipeline status and history
      • Integration: Built into GitLab ecosystem

      2. Getting Started

      Prerequisites

      • GitLab account (GitLab.com or self-hosted)
      • Basic Git knowledge
      • Understanding of your application’s build process
      • Access to a GitLab Runner (shared or dedicated)

      Your First Pipeline

      Create a .gitlab-ci.yml file in your repository root:

      # Basic pipeline example
stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building the application..."
    - mkdir build
    - echo "Build complete" > build/info.txt
  artifacts:
    paths:
      - build/

test_job:
  stage: test
  script:
    - echo "Running tests..."
    - test -f build/info.txt && echo "Test passed!" || exit 1

deploy_job:
  stage: deploy
  script:
    - echo "Deploying application..."
    - echo "Deployment successful!"
  only:
    - main
      YAML

      Pipeline Flow Visualization

      flowchart TD
    A[Code Push] --> B[Pipeline Triggered]
    B --> C[Build Stage]
    C --> D[Test Stage]
    D --> E{Tests Pass?}
    E -->|Yes| F[Deploy Stage]
    E -->|No| G[Pipeline Fails]
    F --> H[Deployment Complete]
    G --> I[Fix Issues]
    I --> A

      3. GitLab CI/CD Fundamentals

      Pipeline Components

      graph TB
    A[Pipeline] --> B[Stage 1]
    A --> C[Stage 2]
    A --> D[Stage 3]

    B --> E[Job 1.1]
    B --> F[Job 1.2]

    C --> G[Job 2.1]
    C --> H[Job 2.2]
    C --> I[Job 2.3]

    D --> J[Job 3.1]

    style A fill:#ff6b6b
    style B fill:#4ecdc4
    style C fill:#45b7d1
    style D fill:#96ceb4

      Key Concepts

      Stages

      Sequential groups of jobs that run in order:

      stages:
  - build
  - test
  - security
  - deploy
  - cleanup
      YAML

      Jobs

      Individual tasks within stages:

      unit_tests:
  stage: test
  script:
    - npm test

integration_tests:
  stage: test
  script:
    - npm run test:integration
      YAML

      Scripts

      Commands executed by jobs:

      build_application:
  stage: build
  script:
    - npm install
    - npm run build
    - npm run minify
      YAML

      Pipeline Execution Flow

      sequenceDiagram
    participant Dev as Developer
    participant GL as GitLab
    participant Runner as GitLab Runner
    participant Env as Environment

    Dev->>GL: Push Code
    GL->>Runner: Trigger Pipeline
    Runner->>Runner: Execute Jobs
    Runner->>GL: Report Status
    Runner->>Env: Deploy (if successful)
    GL->>Dev: Notification

      4. Pipeline Configuration

      Advanced .gitlab-ci.yml Structure

      # Global settings
image: node:16-alpine
services:
  - postgres:13
  - redis:6

variables:
  POSTGRES_DB: testdb
  POSTGRES_USER: user
  POSTGRES_PASSWORD: password
  NODE_ENV: test

# Cache configuration
cache:
  paths:
    - node_modules/
    - .npm/

# Stages definition
stages:
  - prepare
  - build
  - test
  - security
  - deploy
  - cleanup

# Before script (runs before each job)
before_script:
  - npm ci --cache .npm --prefer-offline

# Jobs
install_dependencies:
  stage: prepare
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules/
    expire_in: 1 hour

build_application:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week
  dependencies:
    - install_dependencies

unit_tests:
  stage: test
  script:
    - npm run test:unit
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

security_scan:
  stage: security
  script:
    - npm audit
    - npm run security:scan
  allow_failure: true

deploy_staging:
  stage: deploy
  script:
    - echo "Deploying to staging..."
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - develop

deploy_production:
  stage: deploy
  script:
    - echo "Deploying to production..."
  environment:
    name: production
    url: https://example.com
  only:
    - main
  when: manual

cleanup_job:
  stage: cleanup
  script:
    - echo "Cleaning up temporary files..."
  when: always
      YAML

      Configuration Patterns

      Conditional Execution

      graph TD
    A[Pipeline Start] --> B{Branch Check}
    B -->|main| C[Full Pipeline]
    B -->|develop| D[Staging Pipeline]
    B -->|feature/*| E[Test Pipeline Only]

    C --> F[Build + Test + Deploy Prod]
    D --> G[Build + Test + Deploy Staging]
    E --> H[Build + Test Only]

      Parallel vs Sequential Jobs

      # Parallel jobs (same stage)
test_unit:
  stage: test
  script: npm run test:unit

test_integration:
  stage: test
  script: npm run test:integration

test_e2e:
  stage: test
  script: npm run test:e2e

# Sequential stages
stages:
  - build      # Runs first
  - test       # Runs after build completes
  - deploy     # Runs after test completes
      YAML

      5. Jobs and Stages

      Job Configuration Deep Dive

      complex_job:
  stage: build
  image: node:16
  services:
    - name: postgres:13
      alias: database
  variables:
    DATABASE_URL: "postgres://user:pass@database:5432/db"
  before_script:
    - apt-get update -qq && apt-get install -y -qq git
  script:
    - npm ci
    - npm run build
    - npm test
  after_script:
    - echo "Job completed"
  artifacts:
    paths:
      - dist/
      - coverage/
    reports:
      junit: test-results.xml
    expire_in: 1 week
  cache:
    key: $CI_COMMIT_REF_SLUG
    paths:
      - node_modules/
  retry: 2
  timeout: 1h
  tags:
    - docker
  only:
    - main
    - develop
  except:
    - schedules
      YAML

      Job Lifecycle

      stateDiagram-v2
    [*] --> Created
    Created --> Pending: Runner Available
    Pending --> Running: Job Starts
    Running --> Success: Script Succeeds
    Running --> Failed: Script Fails
    Running --> Canceled: Manual Cancel
    Success --> [*]
    Failed --> [*]
    Canceled --> [*]

    Failed --> Running: Retry (if configured)

      Advanced Job Patterns

      Matrix Jobs

      .test_template:
  script:
    - npm test
  parallel:
    matrix:
      - NODE_VERSION: ["14", "16", "18"]
        OS: ["ubuntu", "alpine"]

test_matrix:
  extends: .test_template
  image: node:$NODE_VERSION-$OS
      YAML

      Dynamic Child Pipelines

      generate_pipeline:
  stage: generate
  script:
    - python generate_pipeline.py > generated-pipeline.yml
  artifacts:
    paths:
      - generated-pipeline.yml

trigger_pipeline:
  stage: trigger
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate_pipeline
      YAML

      Stage Dependencies

      graph LR
    subgraph "Stage 1: Build"
        A[Compile Code]
        B[Build Assets]
    end

    subgraph "Stage 2: Test"
        C[Unit Tests]
        D[Integration Tests]
        E[Security Scan]
    end

    subgraph "Stage 3: Deploy"
        F[Deploy Staging]
        G[Deploy Production]
    end

    A --> C
    A --> D
    B --> C
    B --> D

    C --> F
    D --> F
    E --> F

    F --> G

      6. Variables and Environment Management

      Variable Types and Hierarchy

      graph TB
    A[Predefined Variables] --> E[Final Variable Value]
    B[Project Variables] --> E
    C[Group Variables] --> E
    D[Job Variables] --> E

    style A fill:#ffebee
    style B fill:#e8f5e8
    style C fill:#e3f2fd
    style D fill:#fff3e0
    style E fill:#f3e5f5

      Variable Configuration Examples

      Project-level Variables (GitLab UI)

      Variable Name: DATABASE_URL
Value: postgres://user:pass@localhost:5432/myapp
Protected: ✓ (only available in protected branches)
Masked: ✓ (hidden in job logs)
      YAML

      Group-level Variables

      Variable Name: DOCKER_REGISTRY
Value: registry.gitlab.com/mygroup
Environment: production
      YAML

      Pipeline Variables in YAML

      variables:
  # Global variables
  DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  ENVIRONMENT: development

  # Feature flags
  ENABLE_FEATURE_X: "true"
  DEBUG_MODE: "false"

build_job:
  variables:
    # Job-specific variables
    BUILD_ENV: production
    OPTIMIZATION_LEVEL: 2
  script:
    - echo "Building with $BUILD_ENV environment"
    - echo "Docker image: $DOCKER_IMAGE"
      YAML

      Environment Management

      # Environment definitions
.deploy_template: &deploy_template
  script:
    - echo "Deploying to $ENVIRONMENT_NAME"
    - kubectl apply -f k8s/
  environment:
    name: $ENVIRONMENT_NAME
    url: https://$ENVIRONMENT_NAME.example.com

deploy_development:
  <<: *deploy_template
  variables:
    ENVIRONMENT_NAME: development
    REPLICAS: 1
  only:
    - develop

deploy_staging:
  <<: *deploy_template
  variables:
    ENVIRONMENT_NAME: staging
    REPLICAS: 2
  only:
    - main

deploy_production:
  <<: *deploy_template
  variables:
    ENVIRONMENT_NAME: production
    REPLICAS: 5
  only:
    - main
  when: manual
      YAML

      Predefined Variables Usage

      show_pipeline_info:
  script:
    - echo "Pipeline ID: $CI_PIPELINE_ID"
    - echo "Commit SHA: $CI_COMMIT_SHA"
    - echo "Branch: $CI_COMMIT_REF_NAME"
    - echo "Project: $CI_PROJECT_NAME"
    - echo "Registry: $CI_REGISTRY_IMAGE"
    - echo "Runner: $CI_RUNNER_DESCRIPTION"
      YAML

      Environment Flow

      flowchart TD
    A[Code Commit] --> B{Branch?}
    B -->|feature/*| C[Development Environment]
    B -->|develop| D[Staging Environment]
    B -->|main| E[Production Environment]

    C --> F[Automated Deployment]
    D --> G[Automated Deployment]
    E --> H[Manual Approval Required]
    H --> I[Production Deployment]

    style C fill:#e8f5e8
    style D fill:#fff3e0
    style E fill:#ffebee

      7. Docker Integration

      Docker in GitLab CI/CD

      graph TB
    A[Source Code] --> B[Dockerfile]
    B --> C[Docker Build]
    C --> D[Container Registry]
    D --> E[Deployment]

    subgraph "GitLab Pipeline"
        F[Build Stage] --> G[Test in Container]
        G --> H[Push to Registry]
        H --> I[Deploy Container]
    end

    C --> F
    D --> I

      Basic Docker Pipeline

      # Using Docker-in-Docker
image: docker:20.10.16

services:
  - docker:20.10.16-dind

variables:
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_DRIVER: overlay2
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

stages:
  - build
  - test
  - deploy

before_script:
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build_image:
  stage: build
  script:
    - docker build -t $IMAGE_TAG .
    - docker push $IMAGE_TAG

test_image:
  stage: test
  script:
    - docker pull $IMAGE_TAG
    - docker run --rm $IMAGE_TAG npm test

deploy_image:
  stage: deploy
  script:
    - docker pull $IMAGE_TAG
    - docker tag $IMAGE_TAG $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main
      YAML

      Multi-stage Docker Builds

      # Multi-stage Dockerfile
FROM node:16-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM node:16-alpine AS runtime
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]
      Dockerfile
      # Corresponding GitLab CI
build_optimized:
  stage: build
  script:
    - docker build --target runtime -t $IMAGE_TAG .
    - docker push $IMAGE_TAG
      YAML

      Container Scanning and Security

      # Include GitLab's container scanning
include:
  - template: Security/Container-Scanning.gitlab-ci.yml

# Custom security scan
security_scan:
  stage: security
  image: 
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE_TAG
  allow_failure: true
      YAML

      Docker Compose in CI

      # Using Docker Compose for testing
test_with_compose:
  stage: test
  image: docker/compose:latest
  services:
    - docker:20.10.16-dind
  script:
    - docker-compose up -d
    - docker-compose exec -T app npm test
    - docker-compose down
      YAML

      8. Testing Strategies

      Testing Pyramid in CI/CD

      graph TB
    A[Unit TestsFast, Isolated, Many] --> B[Integration TestsMedium Speed, Some Dependencies]
    B --> C[End-to-End TestsSlow, Full System, Few]

    style A fill:#e8f5e8
    style B fill:#fff3e0
    style C fill:#ffebee

      Comprehensive Testing Pipeline

      stages:
  - build
  - unit_test
  - integration_test
  - security_test
  - e2e_test
  - performance_test

# Unit Tests
unit_tests:
  stage: unit_test
  image: node:16-alpine
  script:
    - npm ci
    - npm run test:unit
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'

# Integration Tests
integration_tests:
  stage: integration_test
  services:
    - postgres:13
    - redis:6
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: ""
  script:
    - npm run test:integration
  artifacts:
    reports:
      junit: integration-results.xml

# Security Tests
security_tests:
  stage: security_test
  script:
    - npm audit --audit-level=high
    - npm run test:security
  allow_failure: true

# End-to-End Tests
e2e_tests:
  stage: e2e_test
  image: cypress/browsers:node16.14.2-slim-chrome103-ff102
  services:
    - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      alias: app
  script:
    - npm ci
    - npm run e2e:headless
  artifacts:
    when: always
    paths:
      - cypress/videos/
      - cypress/screenshots/
    expire_in: 1 week

# Performance Tests
performance_tests:
  stage: performance_test
  image: loadimpact/k6:latest
  script:
    - k6 run --out json=results.json performance-tests.js
  artifacts:
    reports:
      performance: results.json
      YAML

      Test Result Visualization

      graph LR
    A[Test Execution] --> B[JUnit XML]
    A --> C[Coverage Report]
    A --> D[Performance Metrics]

    B --> E[GitLab Test Reports]
    C --> F[Coverage Visualization]
    D --> G[Performance Graphs]

    E --> H[Merge Request Widget]
    F --> H
    G --> H

      Parallel Testing

      # Parallel test execution
test_parallel:
  stage: test
  parallel: 4
  script:
    - npm run test -- --testNamePattern="$TEST_PATTERN"
  variables:
    TEST_PATTERN: 
      - "User"
      - "Product" 
      - "Order"
      - "Payment"

# Parallel matrix testing
test_matrix:
  parallel:
    matrix:
      - NODE_VERSION: ["14", "16", "18"]
        DATABASE: ["postgres", "mysql"]
  image: node:$NODE_VERSION
  services:
    - name: $DATABASE:latest
  script:
    - npm test
      YAML

      9. Deployment Patterns

      Deployment Strategies Overview

      graph TD
    A[Deployment Strategies] --> B[Blue-Green]
    A --> C[Rolling Update]
    A --> D[Canary]
    A --> E[Feature Flags]

    B --> F[Zero DowntimeFull Switch]
    C --> G[Gradual UpdateMinimal Resources]
    D --> H[Risk MitigationGradual Rollout]
    E --> I[Runtime ControlA/B Testing]

      Blue-Green Deployment

      # Blue-Green deployment pattern
variables:
  BLUE_ENV: "blue"
  GREEN_ENV: "green"
  CURRENT_ENV: "blue"

deploy_green:
  stage: deploy
  script:
    - kubectl apply -f k8s/deployment-green.yaml
    - kubectl wait --for=condition=available --timeout=600s deployment/app-green
    - kubectl apply -f k8s/service-green.yaml
  environment:
    name: production-green
    url: https://green.example.com
  when: manual

switch_traffic:
  stage: deploy
  script:
    - kubectl patch service app-service -p '{"spec":{"selector":{"version":"green"}}}'
    - echo "Traffic switched to green environment"
  environment:
    name: production
    url: https://example.com
  when: manual
  dependencies:
    - deploy_green

cleanup_blue:
  stage: cleanup
  script:
    - kubectl delete deployment app-blue
  when: manual
  allow_failure: true
      YAML

      Canary Deployment

      # Canary deployment with traffic splitting
deploy_canary:
  stage: deploy
  script:
    - helm upgrade --install app-canary ./helm-chart 
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --set replicaCount=1
        --set canary.enabled=true
        --set canary.weight=10  # 10% traffic
  environment:
    name: production-canary
    url: https://canary.example.com

monitor_canary:
  stage: monitor
  script:
    - python scripts/monitor_metrics.py --duration=300 --threshold=0.01
  after_script:
    - |
      if [ $CI_JOB_STATUS == "success" ]; then
        echo "Canary metrics acceptable, proceeding with full deployment"
      else
        echo "Canary metrics failed, rolling back"
        kubectl delete deployment app-canary
      fi

deploy_full:
  stage: deploy
  script:
    - helm upgrade --install app ./helm-chart 
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --set replicaCount=5
  environment:
    name: production
    url: https://example.com
  dependencies:
    - monitor_canary
      YAML

      Rolling Deployment

      # Rolling deployment with health checks
deploy_rolling:
  stage: deploy
  script:
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - kubectl rollout status deployment/app --timeout=600s
  environment:
    name: production
    url: https://example.com

health_check:
  stage: verify
  script:
    - curl -f https://example.com/health || exit 1
    - python scripts/smoke_tests.py
  retry: 3
      YAML

      Multi-Environment Pipeline

      flowchart TD
    A[Code Commit] --> B[Build & Test]
    B --> C[Deploy to Dev]
    C --> D[Automated Tests]
    D --> E{Tests Pass?}
    E -->|Yes| F[Deploy to Staging]
    E -->|No| G[Rollback & Notify]
    F --> H[User Acceptance Testing]
    H --> I{UAT Pass?}
    I -->|Yes| J[Manual Approval for Prod]
    I -->|No| K[Fix Issues]
    J --> L[Deploy to Production]
    L --> M[Monitor & Verify]
    K --> A
    G --> A

      10. Advanced Features

      Include and Extends

      # Common template file: .gitlab/ci-templates.yml
.deploy_template:
  script:
    - echo "Deploying to $ENVIRONMENT"
    - kubectl apply -f k8s/
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual
    - when: never

.test_template:
  script:
    - npm ci
    - npm test
  artifacts:
    reports:
      junit: junit.xml
      YAML
      # Main .gitlab-ci.yml
include:
  - local: '.gitlab/ci-templates.yml'
  - project: 'group/shared-ci'
    file: '/templates/security.yml'
  - remote: 'https://example.com/ci-templates.yml'

# Using extends
unit_tests:
  extends: .test_template
  variables:
    TEST_TYPE: unit

integration_tests:
  extends: .test_template
  variables:
    TEST_TYPE: integration

deploy_production:
  extends: .deploy_template
  variables:
    ENVIRONMENT: production
      YAML

      Child Pipelines

      # Parent pipeline
trigger_child:
  stage: trigger
  trigger:
    include: child-pipeline.yml
    strategy: depend

# Generate dynamic child pipeline
generate_child_pipeline:
  stage: generate
  script:
    - python generate_tests.py > generated-pipeline.yml
  artifacts:
    paths:
      - generated-pipeline.yml

trigger_generated:
  stage: trigger
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate_child_pipeline
      YAML

      Multi-project Pipelines

      # Trigger downstream project
trigger_integration_tests:
  stage: trigger
  trigger:
    project: group/integration-tests
    branch: main
    strategy: depend
  variables:
    UPSTREAM_PROJECT: $CI_PROJECT_NAME
    UPSTREAM_COMMIT: $CI_COMMIT_SHA
      YAML

      Pipeline Scheduling and Rules

      # Complex rules example
build_job:
  script: echo "Building..."
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: always
    - if: '$CI_COMMIT_BRANCH =~ /^feature\/.*/'
      when: manual
      allow_failure: true
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never
    - changes:
        - "src/**/*"
        - "package.json"
      when: on_success

# Scheduled pipeline with different behavior
nightly_build:
  script:
    - echo "Running nightly build with extended tests"
    - npm run test:extended
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: always
      YAML

      Advanced Caching Strategies

      # Multi-level caching
.cache_template:
  cache:
    - key: 
        files:
          - package-lock.json
      paths:
        - node_modules/
    - key: $CI_COMMIT_REF_SLUG
      paths:
        - .npm/
        - dist/
      policy: pull-push

build_with_cache:
  extends: .cache_template
  script:
    - npm ci --cache .npm
    - npm run build
      YAML

      11. Security and Best Practices

      Security Pipeline Integration

      graph TB
    A[Code Commit] --> B[SAST Scan]
    B --> C[Dependency Check]
    C --> D[Container Scan]
    D --> E[DAST Scan]
    E --> F[License Check]
    F --> G[Deploy if Secure]

    B --> H[Security Report]
    C --> H
    D --> H
    E --> H
    F --> H

    style B fill:#ffebee
    style C fill:#fff3e0
    style D fill:#e8f5e8
    style E fill:#e3f2fd
    style F fill:#f3e5f5

      Security Templates

      include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/DAST.gitlab-ci.yml
  - template: Security/License-Scanning.gitlab-ci.yml

# Custom security job
custom_security_scan:
  stage: security
  image: owasp/zap2docker-stable
  script:
    - zap-baseline.py -t https://example.com -J zap-report.json
  artifacts:
    reports:
      sast: zap-report.json
  allow_failure: true

# Secrets scanning
secrets_scan:
  stage: security
  image: trufflesecurity/trufflehog:latest
  script:
    - trufflehog --json filesystem . > secrets-report.json
  artifacts:
    reports:
      secret_detection: secrets-report.json
      YAML

      Best Practices Checklist

      Pipeline Security

      • ✅ Use protected variables for sensitive data
      • ✅ Enable masked variables for secrets
      • ✅ Implement branch protection rules
      • ✅ Use minimal permissions for runners
      • ✅ Regular security scanning integration
      • ✅ Audit pipeline changes

      Performance Optimization

      • ✅ Use caching effectively
      • ✅ Minimize Docker image sizes
      • ✅ Parallel job execution
      • ✅ Artifact cleanup policies
      • ✅ Resource-appropriate runner selection

      Code Quality

      # Quality gates
quality_check:
  stage: quality
  script:
    - sonar-scanner -Dsonar.projectKey=$CI_PROJECT_NAME
    - python scripts/quality_gate.py
  only:
    - merge_requests
    - main

# Automated code review
code_review:
  stage: review
  script:
    - pylint src/ --output-format=json > pylint-report.json
    - black --check src/
    - isort --check-only src/
  artifacts:
    reports:
      codequality: pylint-report.json
      YAML

      Resource Management

      # Resource limits and optimization
.resource_template:
  tags:
    - docker
  resource_group: production
  timeout: 30m
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure

production_deploy:
  extends: .resource_template
  script:
    - deploy.sh
  environment:
    name: production
    deployment_tier: production
      YAML

      12. Monitoring and Optimization

      Pipeline Monitoring Strategy

      graph TB
    A[Pipeline Metrics] --> B[Execution Time]
    A --> C[Success Rate]
    A --> D[Resource Usage]
    A --> E[Cost Analysis]

    B --> F[Performance Optimization]
    C --> G[Reliability Improvement]
    D --> H[Resource Scaling]
    E --> I[Cost Optimization]

    F --> J[Dashboard & Alerts]
    G --> J
    H --> J
    I --> J

      Monitoring Implementation

      # Pipeline monitoring job
monitor_pipeline:
  stage: monitor
  script:
    - python scripts/collect_metrics.py
    - curl -X POST "https://monitoring.example.com/metrics" 
        -H "Content-Type: application/json"
        -d '{"pipeline_id": "'$CI_PIPELINE_ID'", "duration": "'$CI_JOB_STARTED_AT'", "status": "success"}'
  when: always

# Performance tracking
performance_tracking:
  stage: analyze
  script:
    - echo "Pipeline started at: $CI_PIPELINE_CREATED_AT"
    - echo "Job started at: $CI_JOB_STARTED_AT"
    - python scripts/performance_analysis.py
  artifacts:
    reports:
      performance: performance-report.json
  when: always
      YAML

      Optimization Techniques

      Parallel Execution Optimization

      # Optimized parallel testing
test_optimized:
  parallel: 
    matrix:
      - TEST_SUITE: [unit, integration, e2e]
        NODE_VERSION: [16, 18]
  script:
    - npm run test:$TEST_SUITE
  needs: ["build"]  # Only depend on build, not previous stage
      YAML

      Cache Optimization

      # Advanced caching strategy
.cache_optimized:
  cache:
    - key:
        files:
          - yarn.lock
          - package.json
      paths:
        - node_modules/
      policy: pull-push
    - key: $CI_COMMIT_REF_SLUG-build
      paths:
        - dist/
        - .cache/
      policy: push
  before_script:
    - echo "Cache hit ratio:" && ls -la node_modules/ || echo "Cache miss"
      YAML

      Performance Metrics Collection

      # Metrics collection job
collect_metrics:
  stage: .post
  image: alpine:latest
  script:
    - apk add --no-cache curl jq
    - |
      METRICS=$(cat << EOF
      {
        "pipeline_id": "$CI_PIPELINE_ID",
        "project": "$CI_PROJECT_NAME",
        "duration": $(($(date +%s) - $(date -d "$CI_PIPELINE_CREATED_AT" +%s))),
        "commit_sha": "$CI_COMMIT_SHA",
        "branch": "$CI_COMMIT_REF_NAME",
        "status": "$CI_JOB_STATUS"
      }
      EOF
      )
    - echo "$METRICS" | curl -X POST https://metrics.example.com/api/pipelines -d @-
  when: always
      YAML

      13. Troubleshooting

      Common Issues and Solutions

      Pipeline Troubleshooting Flow

      flowchart TD
    A[Pipeline Fails] --> B{Check Logs}
    B --> C[Script Error?]
    B --> D[Runner Issue?]
    B --> E[Resource Problem?]

    C --> F[Fix Code/Script]
    D --> G[Check Runner Status]
    E --> H[Increase Resources]

    G --> I[Restart Runner]
    G --> J[Check Runner Config]

    F --> K[Test Locally]
    I --> L[Retry Pipeline]
    J --> M[Update Runner]
    H --> N[Adjust Limits]
    K --> O[Commit Fix]

      Debug Configuration

      # Debug job template
.debug_template:
  variables:
    CI_DEBUG_TRACE: "true"
  before_script:
    - echo "=== Environment Debug ==="
    - env | grep CI_ | sort
    - echo "=== System Info ==="
    - uname -a
    - df -h
    - free -h
    - echo "=== Network Test ==="
    - ping -c 3 google.com
  after_script:
    - echo "=== Job completed with status: $CI_JOB_STATUS ==="

# Debug specific job
debug_build:
  extends: .debug_template
  stage: debug
  script:
    - npm config list
    - npm run build --verbose
  when: manual
  only:
    - /^debug\/.*$/
      YAML

      Common Error Patterns

      Docker Issues

      # Docker troubleshooting
docker_debug:
  stage: debug
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker info
    - docker version
  script:
    - docker build -t debug-image .
    - docker run --rm debug-image env
  when: manual
      YAML

      Permission Issues

      # Permission debugging
permission_debug:
  stage: debug
  script:
    - whoami
    - id
    - ls -la
    - pwd
    - echo "PATH: $PATH"
    - which npm
  when: manual
      YAML

      Log Analysis Patterns

      # Enhanced logging
.logging_template:
  before_script:
    - export LOG_LEVEL=debug
    - export TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
    - echo "[$TIMESTAMP] Starting job: $CI_JOB_NAME"
  after_script:
    - echo "[$TIMESTAMP] Job $CI_JOB_NAME completed with status: $CI_JOB_STATUS"
    - |
      if [ "$CI_JOB_STATUS" = "failed" ]; then
        echo "=== FAILURE ANALYSIS ==="
        echo "Pipeline: $CI_PIPELINE_URL"
        echo "Commit: $CI_COMMIT_SHA"
        echo "Branch: $CI_COMMIT_REF_NAME"
        # Send notification to monitoring system
        curl -X POST "https://alerts.example.com/webhook" \
          -H "Content-Type: application/json" \
          -d "{\"job\":\"$CI_JOB_NAME\",\"status\":\"failed\",\"pipeline\":\"$CI_PIPELINE_URL\"}"
      fi
      YAML

      14. Real-World Examples

      Complete Node.js Application Pipeline

      # .gitlab-ci.yml for Node.js application
image: node:16-alpine

stages:
  - prepare
  - build
  - test
  - security
  - package
  - deploy
  - monitor

variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm"
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

cache:
  key:
    files:
      - package-lock.json
  paths:
    - .npm/
    - cache/Cypress/
    - node_modules/

# Prepare stage
install_dependencies:
  stage: prepare
  script:
    - npm ci --cache .npm --prefer-offline
  artifacts:
    expire_in: 1 hour
    paths:
      - node_modules/

# Build stage
build_application:
  stage: build
  script:
    - npm run build
    - npm run build:docs
  artifacts:
    expire_in: 1 week
    paths:
      - dist/
      - docs/
  dependencies:
    - install_dependencies

# Test stage
unit_tests:
  stage: test
  script:
    - npm run test:unit -- --coverage
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    paths:
      - coverage/
  dependencies:
    - install_dependencies

integration_tests:
  stage: test
  services:
    - postgres:13-alpine
    - redis:6-alpine
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: ""
    REDIS_URL: redis://redis:6379
  script:
    - npm run db:migrate
    - npm run test:integration
  artifacts:
    reports:
      junit: integration-junit.xml
  dependencies:
    - install_dependencies

e2e_tests:
  stage: test
  image: cypress/browsers:node16.14.2-slim-chrome103-ff102
  services:
    - name: postgres:13-alpine
      alias: database
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: ""
  script:
    - npm start &
    - npm run wait-on http://localhost:3000
    - npm run cypress:run
  artifacts:
    when: always
    paths:
      - cypress/videos/
      - cypress/screenshots/
    reports:
      junit: cypress/results/junit.xml
  dependencies:
    - build_application

# Security stage
security_scan:
  stage: security
  script:
    - npm audit --audit-level high
    - npm run security:scan
  artifacts:
    reports:
      dependency_scanning: security-report.json
  allow_failure: true

sast_scan:
  stage: security
  image: securecodewarrior/docker-scanner
  script:
    - scan --format json --output sast-report.json src/
  artifacts:
    reports:
      sast: sast-report.json
  allow_failure: true

# Package stage
build_docker_image:
  stage: package
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
    IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $IMAGE_TAG .
    - docker tag $IMAGE_TAG $CI_REGISTRY_IMAGE:latest
    - docker push $IMAGE_TAG
    - docker push $CI_REGISTRY_IMAGE:latest
  dependencies:
    - build_application
  only:
    - main
    - develop

# Deploy stages
deploy_staging:
  stage: deploy
  image: alpine/helm:latest
  script:
    - helm upgrade --install myapp-staging ./helm-chart 
        --set image.repository=$CI_REGISTRY_IMAGE
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --set environment=staging
        --set replicaCount=2
  environment:
    name: staging
    url: https://staging.example.com
  dependencies:
    - build_docker_image
  only:
    - develop

deploy_production:
  stage: deploy
  image: alpine/helm:latest
  script:
    - helm upgrade --install myapp ./helm-chart 
        --set image.repository=$CI_REGISTRY_IMAGE
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --set environment=production
        --set replicaCount=5
  environment:
    name: production
    url: https://example.com
    deployment_tier: production
  dependencies:
    - build_docker_image
  only:
    - main
  when: manual

# Monitor stage
health_check:
  stage: monitor
  image: curlimages/curl:latest
  script:
    - sleep 30  # Wait for deployment
    - curl -f $CI_ENVIRONMENT_URL/health
    - curl -f $CI_ENVIRONMENT_URL/metrics
  dependencies:
    - deploy_staging
    - deploy_production
  when: always
      YAML

      Python Django Application Pipeline

      # Django application pipeline
image: python:3.9

services:
  - postgres:13
  - redis:6

variables:
  POSTGRES_DB: testdb
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: postgres
  POSTGRES_HOST: postgres
  REDIS_URL: redis://redis:6379/0
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip/
    - venv/

stages:
  - setup
  - test
  - quality
  - security
  - build
  - deploy

setup:
  stage: setup
  script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install --upgrade pip
    - pip install -r requirements.txt
    - pip install -r requirements-dev.txt
  artifacts:
    paths:
      - venv/
    expire_in: 1 hour

test:
  stage: test
  script:
    - source venv/bin/activate
    - python manage.py migrate
    - python manage.py collectstatic --noinput
    - coverage run --source='.' manage.py test
    - coverage xml
  coverage: '/TOTAL.+?(\d+\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
      junit: test-results.xml
  dependencies:
    - setup

code_quality:
  stage: quality
  script:
    - source venv/bin/activate
    - flake8 . --format=json --output-file=flake8-report.json
    - black --check .
    - isort --check-only .
  artifacts:
    reports:
      codequality: flake8-report.json
  dependencies:
    - setup
  allow_failure: true

security_check:
  stage: security
  script:
    - source venv/bin/activate
    - bandit -r . -f json -o bandit-report.json
    - safety check --json --output safety-report.json
  artifacts:
    reports:
      sast: bandit-report.json
  dependencies:
    - setup
  allow_failure: true

build_image:
  stage: build
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  only:
    - main
    - develop

deploy_heroku:
  stage: deploy
  image: ruby:2.7
  before_script:
    - gem install dpl
  script:
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_API_KEY
  environment:
    name: production
    url: https://$HEROKU_APP_NAME.herokuapp.com
  only:
    - main
  when: manual
      YAML

      Microservices Multi-Project Pipeline

      # Parent pipeline for microservices
stages:
  - trigger
  - verify
  - deploy

trigger_services:
  stage: trigger
  trigger:
    strategy: depend
  parallel:
    matrix:
      - SERVICE: [user-service, product-service, order-service, payment-service]
  script:
    - |
      curl -X POST \
        -F token=$CI_JOB_TOKEN \
        -F ref=main \
        -F "variables[UPSTREAM_PIPELINE_ID]=$CI_PIPELINE_ID" \
        https://gitlab.com/api/v4/projects/group%2F$SERVICE/trigger/pipeline

integration_tests:
  stage: verify
  image: postman/newman:latest
  script:
    - newman run postman_collection.json 
        --environment postman_environment.json 
        --reporters junit,cli
        --reporter-junit-export newman-results.xml
  artifacts:
    reports:
      junit: newman-results.xml
  dependencies:
    - trigger_services

deploy_gateway:
  stage: deploy
  image: alpine/helm:latest
  script:
    - helm upgrade --install api-gateway ./helm-charts/gateway
        --set services.user.image=$USER_SERVICE_IMAGE
        --set services.product.image=$PRODUCT_SERVICE_IMAGE
        --set services.order.image=$ORDER_SERVICE_IMAGE
        --set services.payment.image=$PAYMENT_SERVICE_IMAGE
  environment:
    name: production
    url: https://api.example.com
  dependencies:
    - integration_tests
  when: manual
      YAML

      Conclusion

      This comprehensive guide covers GitLab CI/CD from basic concepts to advanced enterprise patterns. Key takeaways:

      For Beginners

      • Start with simple pipelines and gradually add complexity
      • Understand the basic concepts: stages, jobs, scripts
      • Use GitLab’s built-in templates and examples
      • Practice with small projects first

      For Intermediate Users

      • Master variables and environment management
      • Implement proper testing strategies
      • Use caching and optimization techniques
      • Understand security scanning integration

      For Advanced Users

      • Design complex multi-project pipelines
      • Implement advanced deployment strategies
      • Create reusable templates and includes
      • Focus on monitoring and optimization

      Next Steps

      1. Practice: Set up pipelines for your projects
      2. Community: Join GitLab forums and communities
      3. Documentation: Stay updated with GitLab’s official docs
      4. Experimentation: Try new features and patterns
      5. Sharing: Contribute templates and share knowledge

      Additional Resources

      Remember: CI/CD is a journey of continuous improvement. Start simple, iterate, and gradually build more sophisticated pipelines as

      Discover more from Altgr Blog

      Subscribe to get the latest posts sent to your email.

      Leave a Reply

      Your email address will not be published. Required fields are marked *