HashiCorp Vault

      Ankush More

      A Comprehensive Guide to Secrets Management

      Table of Contents

      1. Introduction to HashiCorp Vault
      2. Getting Started
      3. Core Concepts
      4. Authentication Methods
      5. Secrets Engines
      6. Policies and Access Control
      7. High Availability and Scaling
      8. Advanced Features
      9. Best Practices
      10. Troubleshooting
      11. Enterprise Features
      12. Real-World Use Cases

      1. Introduction to HashiCorp Vault

      The Secrets Management Problem

      In modern infrastructure, applications and services require access to sensitive information like:

      • Database passwords and connection strings
      • API keys for third-party services
      • TLS certificates and private keys
      • Cloud provider credentials
      • Encryption keys

      Traditional approaches to secrets management create significant challenges:

      Security Risks:

      • Secrets hardcoded in application code
      • Plain text configuration files
      • Shared credentials across environments
      • No audit trail of secret access

      Operational Challenges:

      • Manual secret rotation processes
      • Inconsistent secret distribution
      • Difficulty revoking compromised credentials
      • No centralized secret lifecycle management

      What is HashiCorp Vault?

      HashiCorp Vault is an identity-based secrets and encryption management system. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Vault solves the secret sprawl problem by centralizing the storage, access, and lifecycle management of sensitive data.

      Core Value Propositions:

      • Centralized Secret Management: Single source of truth for all secrets
      • Identity-Based Access: Fine-grained access control tied to identity
      • Dynamic Secret Generation: Secrets generated on-demand with automatic expiration
      • Comprehensive Audit Trail: Every action is logged for compliance and security
      • Encryption as a Service: Encrypt/decrypt data without exposing keys

      Vault vs. Traditional Solutions

      graph TB
    subgraph "Traditional Approaches"
        A1[Environment Variables] --> B1[Limited Security]
        A2[Config Files] --> B2[Static Credentials]
        A3[Cloud Key Stores] --> B3[Cloud-Specific]
        A4[Configuration Management] --> B4[Limited Rotation]
    end

    subgraph "HashiCorp Vault"
        A5[Dynamic Secrets] --> B5[Automatic Rotation]
        A6[Identity-Based Access] --> B6[Fine-Grained Control]
        A7[Encryption as a Service] --> B7[Data Protection]
        A8[Comprehensive Auditing] --> B8[Compliance Ready]
    end

    style A5 fill:#4caf50,color:#fff
    style A6 fill:#4caf50,color:#fff
    style A7 fill:#4caf50,color:#fff
    style A8 fill:#4caf50,color:#fff

    style A1 fill:#ff5722,color:#fff
    style A2 fill:#ff5722,color:#fff
    style A3 fill:#ff5722,color:#fff
    style A4 fill:#ff5722,color:#fff

      Why Use Vault?

      graph TD
    A[Traditional Secrets Management] --> B[Scattered Secrets]
    A --> C[Static Credentials]
    A --> D[Limited Audit Trail]
    A --> E[Manual Rotation]

    F[Vault Secrets Management] --> G[Centralized Storage]
    F --> H[Dynamic Secrets]
    F --> I[Comprehensive Auditing]
    F --> J[Automated Rotation]
    F --> K[Fine-grained Access Control]

    style F fill:#326ce5,color:#fff
    style A fill:#ff6b6b,color:#fff

      Key Features

      • Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault
      • Dynamic Secrets: Vault can generate secrets on-demand for some systems
      • Data Encryption: Vault can encrypt and decrypt data without storing it
      • Leasing and Renewal: All secrets in Vault have a lease associated with them
      • Revocation: Vault has built-in support for secret revocation

      Deployment Models

      graph TB
    subgraph "Development"
        A[Dev Mode Server]
        A1[In-Memory Storage]
        A2[Auto-Unseal]
        A --> A1
        A --> A2
    end

    subgraph "Production Single Node"
        B[Vault Server]
        B1[Persistent Storage]
        B2[Manual/Auto Unseal]
        B --> B1
        B --> B2
    end

    subgraph "High Availability"
        C[Active Node]
        D[Standby Node 1]
        E[Standby Node 2]
        F[Shared Storage]
        C --> F
        D --> F
        E --> F
        C -.-> D
        C -.-> E
    end

    subgraph "Multi-Region"
        G[Primary Cluster]
        H[DR Cluster]
        I[Performance Replica]
        G --> H
        G --> I
    end

    style A fill:#ffc107,color:#000
    style B fill:#2196f3,color:#fff
    style C fill:#4caf50,color:#fff
    style G fill:#9c27b0,color:#fff

      When to Use Vault

      ✅ Use Vault When:

      • You have multiple applications/services needing secrets
      • Compliance requires audit trails and access controls
      • You need dynamic secret generation
      • Secret rotation is manual and error-prone
      • You want centralized encryption services

      ❌ Consider Alternatives When:

      • Single application with minimal secret needs
      • Cloud-native with strong cloud provider integration
      • Extremely simple use cases without compliance requirements

      Alternative Solutions Comparison

      FeatureVaultAWS Secrets ManagerAzure Key VaultK8s Secrets
      Dynamic Secrets✅❌❌❌
      Multi-Cloud✅❌❌✅
      Encryption Service✅❌✅❌
      Fine-grained Policies✅⚠️⚠️⚠️
      Audit Logging✅✅✅❌
      Self-Hosted✅❌❌✅

      2. Getting Started

      Installation

      Windows Installation

      # Using Chocolatey
choco install vault

# Using Scoop
scoop install vault

# Manual Installation
# 1. Download from https://releases.hashicorp.com/vault/
# 2. Extract to C:\vault\vault.exe
# 3. Add to PATH environment variable
$env:PATH += ";C:\vault"

# Verify installation
vault version
      PowerShell

      Linux Installation

      # Ubuntu/Debian
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vault

# CentOS/RHEL/Fedora
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install vault

# Manual Binary Installation
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
unzip vault_1.15.0_linux_amd64.zip
sudo mv vault /usr/local/bin/
      Bash

      macOS Installation

      # Using Homebrew
brew tap hashicorp/tap
brew install hashicorp/tap/vault

# Using MacPorts
sudo port install vault
      Bash

      Docker Installation

      # Run Vault in development mode
docker run --cap-add=IPC_LOCK -d --name=dev-vault -p 8200:8200 vault:latest

# Run with custom configuration
docker run --cap-add=IPC_LOCK -d --name=vault \
  -p 8200:8200 \
  -v /vault/logs:/vault/logs \
  -v /vault/file:/vault/file \
  -v /vault/config:/vault/config \
  vault:latest server

# Docker Compose for development
cat > docker-compose.yml << EOF
version: '3.8'
services:
  vault:
    image: vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      - VAULT_DEV_ROOT_TOKEN_ID=myroot
      - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
volumes:
  vault-data:
EOF

docker-compose up -d
      Bash

      Verification

      vault version
# Vault v1.15.0 (build date)
      Bash

      Environment Setup

      Development Environment

      # Windows PowerShell
$env:VAULT_ADDR="http://127.0.0.1:8200"
$env:VAULT_TOKEN="hvs.your-dev-root-token"

# Linux/macOS
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='hvs.your-dev-root-token'

# Skip TLS verification for development (NOT for production)
export VAULT_SKIP_VERIFY=true
      PowerShell

      Production Environment

      # Production environment variables
export VAULT_ADDR='https://vault.company.com:8200'
export VAULT_CACERT='/path/to/ca.pem'
export VAULT_CLIENT_CERT='/path/to/client.pem'
export VAULT_CLIENT_KEY='/path/to/client-key.pem'

# Optional: Use vault token file
export VAULT_TOKEN_FILE='/home/user/.vault-token'
      Bash

      Starting Vault in Development Mode

      # Start Vault dev server (DO NOT use in production)
vault server -dev

# Start with specific root token
vault server -dev -dev-root-token-id="myroot"

# Start with custom listen address
vault server -dev -dev-listen-address="0.0.0.0:8200"
      Bash

      First-Time User Walkthrough

      Step 1: Start Development Server

      # Terminal 1: Start Vault
vault server -dev -dev-root-token-id="myroot"

# Note the unseal key and root token from output
      Bash

      Step 2: Configure Client

      # Terminal 2: Set environment
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'

# Verify connection
vault status
      Bash

      Step 3: Basic Operations

      # Store a secret
vault kv put secret/hello foo=world excited=yes

# Retrieve a secret
vault kv get secret/hello

# Get specific field
vault kv get -field=excited secret/hello

# List secrets
vault kv list secret/

# Store JSON data
vault kv put secret/app/config @config.json

# Store from stdin
echo '{"api_key": "secret123"}' | vault kv put secret/app/api -
      Bash

      Step 4: Enable Additional Features

      # Enable userpass authentication
vault auth enable userpass

# Create a user
vault write auth/userpass/users/alice \
    password=password123 \
    policies=default

# Test user login
vault login -method=userpass username=alice
      Bash

      Basic Vault Architecture

      graph TB
    subgraph "Vault Core"
        A[HTTP API]
        B[Authentication]
        C[Authorization]
        D[Audit]
        E[Storage Backend]
    end

    subgraph "Clients"
        F[CLI]
        G[Web UI]
        H[Applications]
    end

    subgraph "Storage"
        I[File System]
        J[Consul]
        K[Database]
    end

    F --> A
    G --> A
    H --> A

    A --> B
    B --> C
    C --> D
    D --> E

    E --> I
    E --> J
    E --> K

    style A fill:#326ce5,color:#fff

      First Steps with Vault

      # Check status
vault status

# Write a secret
vault kv put secret/hello foo=world

# Read a secret
vault kv get secret/hello

# Delete a secret
vault kv delete secret/hello
      Bash

      3. Core Concepts

      Vault Architecture Deep Dive

      graph TB
    subgraph "Vault Server"
        subgraph "Core"
            A[Barrier]
            B[Storage Backend]
            C[Audit Backend]
            D[Auth Methods]
            E[Secrets Engines]
            F[Policy Store]
        end

        subgraph "API Layer"
            G[HTTP API]
            H[CLI]
            I[Web UI]
        end
    end

    subgraph "External Systems"
        J[Identity Providers]
        K[Databases]
        L[Cloud Providers]
        M[PKI Systems]
    end

    G --> A
    H --> G
    I --> G

    A --> B
    A --> C
    A --> D
    A --> E
    A --> F

    D --> J
    E --> K
    E --> L
    E --> M

    style A fill:#ff6b35,color:#fff
    style G fill:#326ce5,color:#fff

      Initialization and Unsealing

      sequenceDiagram
    participant Admin
    participant Vault
    participant KeyHolders

    Admin->>Vault: vault operator init
    Vault->>Admin: Return unseal keys + root token

    Note over Vault: Vault is SEALED

    KeyHolders->>Vault: vault operator unseal (key 1)
    KeyHolders->>Vault: vault operator unseal (key 2)
    KeyHolders->>Vault: vault operator unseal (key 3)

    Note over Vault: Vault is UNSEALED
    Vault->>Admin: Ready for operations

      Production Initialization

      # Initialize Vault (run once)
vault operator init -key-shares=5 -key-threshold=3

# Example output:
# Unseal Key 1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Unseal Key 2: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Unseal Key 3: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Unseal Key 4: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Unseal Key 5: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Initial Root Token: hvs.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Unseal Vault (need 3 of 5 keys)
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>
      Bash

      Vault Paths and Namespaces

      graph TD
    A[Vault Root] --> B[auth/]
    A --> C[secret/]
    A --> D[sys/]
    A --> E[custom-engine/]

    B --> F[userpass/]
    B --> G[ldap/]
    B --> H[aws/]

    C --> I[myapp/]
    C --> J[database/]

    I --> K[config]
    I --> L[credentials]

    style A fill:#326ce5,color:#fff
    style C fill:#4ecdc4,color:#fff
    style B fill:#45b7d1,color:#fff

      Storage Backend Comparison

      StorageAvailabilityPerformanceComplexityUse Case
      File SystemSingle NodeGoodLowDevelopment
      ConsulHighGoodMediumProduction HA
      Integrated RaftHighExcellentLowProduction HA
      DatabaseMediumGoodMediumExisting DB Infrastructure
      Cloud StorageHighGoodLowCloud-Native

      Security Model

      graph LR
    A[Request] --> B{Authentication}
    B --> C{Authorization}
    C --> D{Policy Check}
    D --> E{Audit Log}
    E --> F[Response]

    G[Identity] --> B
    H[Policies] --> D
    I[Audit Backend] --> E

    style B fill:#ff9800,color:#fff
    style C fill:#2196f3,color:#fff
    style D fill:#4caf50,color:#fff
    style E fill:#9c27b0,color:#fff

      4. Authentication Methods

      Authentication Flow

      sequenceDiagram
    participant Client
    participant Vault
    participant AuthMethod
    participant ExternalSystem

    Client->>Vault: Authentication Request
    Vault->>AuthMethod: Validate Credentials
    AuthMethod->>ExternalSystem: Verify Identity
    ExternalSystem->>AuthMethod: Identity Confirmed
    AuthMethod->>Vault: Authentication Success
    Vault->>Client: Return Vault Token

    Note over Client,Vault: Subsequent requests use Vault token

      Token Authentication

      # Create a token
vault token create -policy=default -ttl=1h

# Lookup token information
vault token lookup

# Renew token
vault token renew

# Revoke token
vault token revoke <token>
      Bash

      UserPass Authentication

      # Enable userpass auth method
vault auth enable userpass

# Create a user
vault write auth/userpass/users/john \
    password=mypassword \
    policies=admin

# Login as user
vault login -method=userpass username=john
      Bash

      LDAP Authentication

      # Enable LDAP auth method
vault auth enable ldap

# Configure LDAP
vault write auth/ldap/config \
    url="ldap://ldap.company.com" \
    userdn="ou=Users,dc=company,dc=com" \
    userattr="uid" \
    groupdn="ou=Groups,dc=company,dc=com" \
    groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
    groupattr="cn"

# Map LDAP groups to Vault policies
vault write auth/ldap/groups/admins policies=admin
vault write auth/ldap/groups/developers policies=dev
      Bash

      AWS IAM Authentication

      # Enable AWS auth method
vault auth enable aws

# Configure AWS credentials
vault write auth/aws/config/client \
    access_key=AKIAIOSFODNN7EXAMPLE \
    secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Create role for EC2 instances
vault write auth/aws/role/dev-role \
    auth_type=iam \
    bound_iam_principal_arn=arn:aws:iam::123456789012:role/MyRole \
    policies=dev \
    ttl=1h
      Bash

      Kubernetes Authentication

      # Enable Kubernetes auth method
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
    token_reviewer_jwt="<service-account-jwt>" \
    kubernetes_host="https://kubernetes.example.com:443" \
    kubernetes_ca_cert=@ca.crt

# Create role for Kubernetes service account
vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp \
    bound_service_account_namespaces=default \
    policies=myapp \
    ttl=1h
      Bash

      AppRole Authentication

      # Enable AppRole auth method
vault auth enable approle

# Create policy for application
vault policy write myapp-policy - <<EOF
path "database/creds/myapp" {
  capabilities = ["read"]
}
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

# Create AppRole
vault write auth/approle/role/myapp \
    token_policies="myapp-policy" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=24h

# Get role ID and secret ID
ROLE_ID=$(vault read -field=role_id auth/approle/role/myapp/role-id)
SECRET_ID=$(vault write -field=secret_id auth/approle/role/myapp/secret-id)

# Application authentication
vault write auth/approle/login \
    role_id="$ROLE_ID" \
    secret_id="$SECRET_ID"
      Bash

      OIDC/JWT Authentication

      # Enable OIDC auth method
vault auth enable oidc

# Configure OIDC provider
vault write auth/oidc/config \
    oidc_discovery_url="https://myco.auth0.com/" \
    oidc_client_id="myco-vault" \
    oidc_client_secret="secret" \
    default_role="demo"

# Create OIDC role
vault write auth/oidc/role/demo \
    bound_audiences="myco-vault" \
    allowed_redirect_uris="http://localhost:8200/ui/vault/auth/oidc/oidc/callback" \
    user_claim="sub" \
    policies="demo"

# Login via OIDC (opens browser)
vault auth -method=oidc
      Bash

      GitHub Authentication

      # Enable GitHub auth method
vault auth enable github

# Configure GitHub auth
vault write auth/github/config \
    organization=mycompany

# Map GitHub teams to policies
vault write auth/github/map/teams/admins \
    value=admin-policy

vault write auth/github/map/teams/developers \
    value=dev-policy

# Login with GitHub token
vault auth -method=github token=<github-token>
      Bash

      TLS Certificate Authentication

      # Enable TLS certificate auth method
vault auth enable cert

# Add trusted certificate authority
vault write auth/cert/certs/web-cert \
    display_name=web \
    policies=web,prod \
    certificate=@ca.pem \
    ttl=3600

# Login with client certificate
vault auth -method=cert \
    -client-cert=client.pem \
    -client-key=client-key.pem
      Bash

      Authentication Method Comparison

      graph TB
    subgraph "Human Users"
        A[UserPass]
        B[LDAP]
        C[OIDC]
        D[GitHub]
    end

    subgraph "Applications/Services"
        E[AppRole]
        F[AWS IAM]
        G[Kubernetes]
        H[JWT]
    end

    subgraph "Infrastructure"
        I[TLS Certificates]
        J[Cloud Instance]
    end

    A --> K[Manual Login]
    B --> L[SSO Integration]
    C --> L
    D --> L

    E --> M[Programmatic Access]
    F --> M
    G --> M
    H --> M

    I --> N[Machine Identity]
    J --> N

    style A fill:#ff6b35,color:#fff
    style E fill:#326ce5,color:#fff
    style I fill:#4ecdc4,color:#fff

      Choosing the Right Authentication Method

      Use CaseRecommended MethodWhy
      Developers/AdminsLDAP/OIDCSSO integration, existing identity system
      CI/CD PipelinesJWT/OIDCShort-lived tokens, OIDC provider integration
      ApplicationsAppRoleSecure, machine-to-machine authentication
      MicroservicesKubernetesNative K8s integration, service account based
      Cloud VMsAWS/Azure/GCPInstance identity, no credential management
      Legacy SystemsTLS CertificatesPKI-based, mutual TLS authentication

      5. Secrets Engines

      Secrets Engine Architecture

      graph TB
    A[Vault Core] --> B[Key/Value v2]
    A --> C[Database]
    A --> D[AWS]
    A --> E[PKI]
    A --> F[Transit]
    A --> G[SSH]
    A --> H[Consul]
    A --> I[RabbitMQ]

    B --> J[Static Secrets]
    C --> K[Dynamic DB Credentials]
    D --> L[Dynamic AWS Credentials]
    E --> M[Certificates]
    F --> N[Encryption as a Service]
    G --> O[SSH Keys]
    H --> P[Consul Tokens]
    I --> Q[RabbitMQ Credentials]

    style A fill:#326ce5,color:#fff
    style B fill:#4ecdc4,color:#fff
    style C fill:#ff6b35,color:#fff

      Key/Value Secrets Engine

      # Enable KV v2 engine
vault secrets enable -path=secret kv-v2

# Write secret with metadata
vault kv put secret/myapp/config \
    username=dbuser \
    password=dbpass123 \
    url=https://api.example.com

# Read secret
vault kv get secret/myapp/config

# Get specific version
vault kv get -version=1 secret/myapp/config

# List secrets
vault kv list secret/myapp/

# Delete secret (soft delete)
vault kv delete secret/myapp/config

# Permanently delete
vault kv destroy -versions=1,2 secret/myapp/config
      Bash

      Database Secrets Engine

      # Enable database secrets engine
vault secrets enable database

# Configure MySQL connection
vault write database/config/my-mysql-database \
    plugin_name=mysql-database-plugin \
    connection_url="{{username}}:{{password}}@tcp(localhost:3306)/" \
    allowed_roles="my-role" \
    username="vault-user" \
    password="vault-password"

# Create role for dynamic credentials
vault write database/roles/my-role \
    db_name=my-mysql-database \
    creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" \
    default_ttl="1h" \
    max_ttl="24h"

# Generate dynamic credentials
vault read database/creds/my-role
      Bash

      AWS Secrets Engine

      # Enable AWS secrets engine
vault secrets enable aws

# Configure AWS credentials
vault write aws/config/root \
    access_key=AKIAIOSFODNN7EXAMPLE \
    secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
    region=us-east-1

# Create role for dynamic AWS credentials
vault write aws/roles/my-role \
    credential_type=iam_user \
    policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

# Generate AWS credentials
vault read aws/creds/my-role
      Bash

      PKI Secrets Engine

      # Enable PKI secrets engine
vault secrets enable pki

# Configure PKI engine
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
vault write pki/root/generate/internal \
    common_name=example.com \
    ttl=87600h

# Configure URLs
vault write pki/config/urls \
    issuing_certificates="http://vault.example.com:8200/v1/pki/ca" \
    crl_distribution_points="http://vault.example.com:8200/v1/pki/crl"

# Create role
vault write pki/roles/example-dot-com \
    allowed_domains=example.com \
    allow_subdomains=true \
    max_ttl=72h

# Generate certificate
vault write pki/issue/example-dot-com \
    common_name=test.example.com
      Bash

      SSH Secrets Engine

      # Enable SSH secrets engine
vault secrets enable ssh

# Configure SSH CA
vault write ssh/config/ca generate_signing_key=true

# Create role for OTP
vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=ubuntu \
    cidr_list=0.0.0.0/0

# Create role for signed certificates
vault write ssh/roles/my-role \
    key_type=ca \
    ttl=30m \
    allow_user_certificates=true \
    allowed_users="*" \
    default_extensions=permit-pty=""

# Generate OTP for SSH access
vault write ssh/creds/otp_key_role ip=192.168.1.100

# Sign SSH public key
vault write -field=signed_key ssh/sign/my-role \
    public_key=@$HOME/.ssh/id_rsa.pub
      Bash

      GCP Secrets Engine

      # Enable GCP secrets engine
vault secrets enable gcp

# Configure GCP credentials
vault write gcp/config \
    credentials=@my-key.json

# Create roleset for service account
vault write gcp/roleset/my-roleset \
    project="my-project" \
    bindings=-<<EOF
resource "//cloudresourcemanager.googleapis.com/projects/my-project" {
  roles = ["roles/viewer"]
}
EOF

# Generate service account key
vault read gcp/key/my-roleset

# Generate access token
vault read gcp/token/my-roleset
      Bash

      Consul Secrets Engine

      # Enable Consul secrets engine
vault secrets enable consul

# Configure Consul connection
vault write consul/config/access \
    address=127.0.0.1:8500 \
    token=consul-master-token

# Create role
vault write consul/roles/my-role \
    policy=@consul-policy.hcl \
    ttl=1h \
    max_ttl=24h

# Generate Consul token
vault read consul/creds/my-role
      Bash

      RabbitMQ Secrets Engine

      # Enable RabbitMQ secrets engine
vault secrets enable rabbitmq

# Configure connection
vault write rabbitmq/config/connection \
    connection_uri="http://localhost:15672" \
    username="admin" \
    password="admin"

# Create role
vault write rabbitmq/roles/production \
    vhosts='{"/":{"write": ".*", "read": ".*"}}'

# Generate credentials
vault read rabbitmq/creds/production
      Bash

      Nomad Secrets Engine

      # Enable Nomad secrets engine
vault secrets enable nomad

# Configure Nomad connection
vault write nomad/config/access \
    address=http://127.0.0.1:4646 \
    token=nomad-management-token

# Create role
vault write nomad/role/monitoring \
    policies=readonly \
    type=client

# Generate Nomad token
vault read nomad/creds/monitoring
      Bash

      Transit Secrets Engine (Encryption as a Service)

      # Enable transit secrets engine
vault secrets enable transit

# Create encryption key
vault write transit/keys/my-key type=aes256-gcm96

# Encrypt data
vault write transit/encrypt/my-key \
    plaintext=$(echo "my secret data" | base64)

# Decrypt data
vault write transit/decrypt/my-key \
    ciphertext=vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==

# Rotate key
vault write transit/keys/my-key/rotate
      Bash

      Secrets Engine Lifecycle Management

      # List all secrets engines
vault secrets list

# Enable secrets engine at custom path
vault secrets enable -path=myapp-db database

# Tune secrets engine parameters
vault secrets tune -default-lease-ttl=1h -max-lease-ttl=24h myapp-db/

# Move secrets engine to new path
vault secrets move myapp-db/ database-myapp/

# Disable secrets engine
vault secrets disable database-myapp/

# Check secrets engine configuration
vault read sys/mounts/myapp-db/tune
      Bash

      Secrets Engine Performance Considerations

      graph TB
    A[Performance Factors] --> B[Secrets Engine Type]
    A --> C[Storage Backend]
    A --> D[Network Latency]
    A --> E[Concurrent Requests]

    B --> F[Static: Fast reads]
    B --> G[Dynamic: External calls]

    C --> H[In-Memory: Fastest]
    C --> I[Local Disk: Fast]
    C --> J[Network Storage: Slower]

    D --> K[Local: <1ms]
    D --> L[Regional: 10-50ms]
    D --> M[Global: 100ms+]

    E --> N[Rate Limiting]
    E --> O[Connection Pooling]

    style A fill:#326ce5,color:#fff
    style F fill:#4caf50,color:#fff
    style G fill:#ff9800,color:#fff

      Dynamic Secrets Lifecycle

      sequenceDiagram
    participant App
    participant Vault
    participant Database

    App->>Vault: Request DB credentials
    Vault->>Database: CREATE USER with permissions
    Database->>Vault: User created successfully
    Vault->>App: Return username/password + lease

    Note over App,Database: Application uses credentials

    App->>Vault: Renew lease (optional)
    Vault->>App: Lease renewed

    Note over Vault: Lease expires or revoked

    Vault->>Database: DROP USER
    Database->>Vault: User deleted

      6. Policies and Access Control

      Policy Architecture

      graph TB
    A[Vault Token] --> B[Attached Policies]
    B --> C[Policy 1]
    B --> D[Policy 2]
    B --> E[Policy N]

    C --> F[Path Permissions]
    C --> G[Capabilities]
    C --> H[Conditions]

    F --> I[secret/myapp/*]
    G --> J[read, write, delete]
    H --> K[time, IP, MFA]

    style A fill:#326ce5,color:#fff
    style B fill:#4ecdc4,color:#fff

      HCL Policy Syntax

      # Example policy file: myapp-policy.hcl

# Allow read access to secret path
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

# Allow CRUD operations on specific secret
path "secret/data/myapp/config" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Allow reading dynamic database credentials
path "database/creds/myapp-role" {
  capabilities = ["read"]
}

# Deny access to admin secrets
path "secret/data/admin/*" {
  capabilities = ["deny"]
}

# Allow token renewal
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Conditional access based on time
path "secret/data/business-hours/*" {
  capabilities = ["read"]
  allowed_parameters = {
    "version" = []
  }

  # Only allow access during business hours
  condition = "time.hour >= 9 && time.hour <= 17"
}
      HCL

      Policy Management

      # Write policy to Vault
vault policy write myapp-policy myapp-policy.hcl

# List policies
vault policy list

# Read policy
vault policy read myapp-policy

# Delete policy
vault policy delete myapp-policy

# Create token with specific policies
vault token create -policy=myapp-policy -ttl=1h
      Bash

      Advanced Policy Features

      # Template policies with identity information
path "secret/data/users/{{identity.entity.name}}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Control based on groups
path "secret/data/team/{{identity.groups.names.team1}}/*" {
  capabilities = ["read"]
}

# Parameter constraints
path "transit/encrypt/{{identity.entity.name}}" {
  capabilities = ["update"]
  allowed_parameters = {
    "plaintext" = []
    "context" = []
  }
  denied_parameters = {
    "nonce" = []
  }
}

# Required parameters
path "database/creds/readonly" {
  capabilities = ["read"]
  required_parameters = ["common_name"]
}
      Bash

      Policy Capabilities

      graph LR
    A[Policy Capabilities] --> B[create]
    A --> C[read]
    A --> D[update]
    A --> E[delete]
    A --> F[list]
    A --> G[sudo]
    A --> H[deny]

    B --> I[Write new data]
    C --> J[Read existing data]
    D --> K[Modify existing data]
    E --> L[Delete data]
    F --> M[List keys/paths]
    G --> N[Admin operations]
    H --> O[Explicit deny]

    style A fill:#326ce5,color:#fff
    style H fill:#ff6b35,color:#fff

      Entity and Group Management

      # Create entity
vault write identity/entity name="john-doe" policies="dev-policy"

# Create group
vault write identity/group name="developers" \
    policies="dev-policy" \
    member_entity_ids="<entity-id-1>,<entity-id-2>"

# Create external group
vault write identity/group name="ldap-developers" \
    type="external" \
    policies="dev-policy"

# Map external group to auth method
vault write identity/group-alias name="CN=developers,OU=Groups,DC=company,DC=com" \
    mount_accessor="<ldap-accessor>" \
    canonical_id="<group-id>"
      Bash

      7. High Availability and Scaling

      Vault Clustering Architecture

      graph TB
    subgraph "Load Balancer"
        A[HAProxy/ALB]
    end

    subgraph "Vault Cluster"
        B[Vault Node 1 - Active]
        C[Vault Node 2 - Standby]
        D[Vault Node 3 - Standby]
    end

    subgraph "Storage Backend"
        E[Consul Cluster]
        F[Consul Node 1]
        G[Consul Node 2]
        H[Consul Node 3]
    end

    subgraph "Clients"
        I[Applications]
        J[CLI]
        K[Web UI]
    end

    I --> A
    J --> A
    K --> A

    A --> B
    A --> C
    A --> D

    B -.-> C
    C -.-> D
    B -.-> D

    B --> E
    C --> E
    D --> E

    E --> F
    E --> G
    E --> H

    style B fill:#4caf50,color:#fff
    style C fill:#ff9800,color:#fff
    style D fill:#ff9800,color:#fff

      Vault Configuration for HA

      # vault.hcl - High Availability Configuration

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

cluster_addr = "https://vault-1.example.com:8201"
api_addr     = "https://vault-1.example.com:8200"

ui = true

# Enterprise license (if using Vault Enterprise)
license_path = "/etc/vault.d/vault.hclic"
      HCL

      Integrated Storage (Raft)

      # vault-raft.hcl - Integrated Storage Configuration

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-3.example.com:8200"
  }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/vault.crt"
  tls_key_file  = "/etc/vault.d/vault.key"
}

cluster_addr = "https://vault-1.example.com:8201"
api_addr     = "https://vault-1.example.com:8200"

ui = true
      HCL

      Raft Cluster Management

      # Initialize first node
vault operator init

# Join additional nodes to cluster
vault operator raft join https://vault-1.example.com:8200

# List cluster members
vault operator raft list-peers

# Check cluster status
vault status

# Remove node from cluster
vault operator raft remove-peer vault-2

# Take snapshot
vault operator raft snapshot save backup.snap

# Restore from snapshot
vault operator raft snapshot restore backup.snap
      Bash

      Disaster Recovery Setup

      sequenceDiagram
    participant Primary as Primary Cluster
    participant DR as DR Cluster
    participant Admin

    Admin->>Primary: Enable DR replication
    Primary->>DR: Initialize replication
    Primary->>DR: Sync encryption keys

    loop Continuous Replication
        Primary->>DR: Replicate data
    end

    Note over Primary: Disaster occurs

    Admin->>DR: Promote to primary
    DR->>Admin: Generate operation token
    Admin->>DR: Complete promotion

    Note over DR: Now serving traffic

      Performance Replication

      # Enable performance replication on primary
vault write sys/replication/performance/primary/enable

# Generate secondary token
vault write sys/replication/performance/primary/secondary-token \
    id="secondary-1"

# On secondary cluster, enable replication
vault write sys/replication/performance/secondary/enable \
    token="<token-from-primary>"

# Check replication status
vault read sys/replication/status
      Bash

      Auto-Unseal with Cloud KMS

      # AWS KMS Auto-Unseal
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "12345678-1234-1234-1234-123456789012"
  endpoint   = "https://kms.us-east-1.amazonaws.com"
}

# Azure Key Vault Auto-Unseal
seal "azurekeyvault" {
  tenant_id      = "46646709-b63e-4747-be42-516edeaf1e14"
  client_id      = "03dc33fc-16d9-4b77-8152-3ec568f8af6e"
  client_secret  = "DUJDS3..."
  vault_name     = "hc-vault"
  key_name       = "vault_key"
}

# Google Cloud KMS Auto-Unseal
seal "gcpckms" {
  project     = "vault-helm-dev-246514"
  region      = "global"
  key_ring    = "vault-helm-unseal-kr"
  crypto_key  = "vault-helm-unseal-key"
}
      HCL

      8. Advanced Features {#advanced-features}

      Vault Agent

      # vault-agent.hcl
pid_file = "/tmp/vault-agent.pid"

vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method "aws" {
    mount_path = "auth/aws"
    config = {
      type = "iam"
      role = "my-vault-role"
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}

template {
  source      = "/etc/vault/templates/config.ctmpl"
  destination = "/etc/myapp/config.json"
  command     = "systemctl reload myapp"
}

cache {
  use_auto_auth_token = true
}
      HCL

      Vault Agent Template

      {{/* config.ctmpl */}}
{
  "database": {
    {{- with secret "database/creds/myapp" }}
    "username": "{{ .Data.username }}",
    "password": "{{ .Data.password }}",
    {{- end }}
    "host": "db.example.com",
    "port": 5432
  },
  "api": {
    {{- with secret "secret/data/myapp/api" }}
    "key": "{{ .Data.data.api_key }}",
    "url": "{{ .Data.data.api_url }}"
    {{- end }}
  }
}
      HCL

      Vault Secrets Operator for Kubernetes

      # VaultAuth CRD
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: default
  namespace: default
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: myapp
    serviceAccount: default
---
# VaultStaticSecret CRD
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-secret
  namespace: default
spec:
  type: kv-v2
  mount: secret
  path: myapp/config
  destination:
    name: myapp-secret
    create: true
  refreshAfter: 30s
  vaultAuthRef: default
---
# VaultDynamicSecret CRD
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: vault-db-secret
  namespace: default
spec:
  mount: database
  path: creds/myapp
  destination:
    name: myapp-db-secret
    create: true
  renewalPercent: 67
  vaultAuthRef: default
      YAML

      Transform Secrets Engine (Enterprise)

      # Enable transform secrets engine
vault secrets enable transform

# Create transformation template
vault write transform/template/ccn-fpe \
    type=regex \
    pattern='(\d{4})-(\d{4})-(\d{4})-(\d{4})' \
    alphabet=builtin/numeric

# Create transformation
vault write transform/transformation/ccn \
    type=fpe \
    template=ccn-fpe \
    tweak_source=supplied \
    allowed_roles=payments

# Create role
vault write transform/role/payments \
    transformations=ccn

# Encode data
vault write transform/encode/payments \
    value="1111-2222-3333-4444" \
    transformation=ccn \
    tweak="user123"

# Decode data
vault write transform/decode/payments \
    value="encoded-value" \
    transformation=ccn \
    tweak="user123"
      Bash

      Control Groups (Enterprise)

      # Control group policy
path "secret/data/super-secret/*" {
  capabilities = ["read"]
  control_group = {
    factor "approvers" {
      identity {
        group_names = ["managers"]
        approvals = 2
      }
    }

    factor "time" {
      identity {
        group_names = ["security"]
        approvals = 1
      }
    }

    max_ttl = "8h"
  }
}
      HCL

      Sentinel Policies (Enterprise)

      # Sentinel policy example
import "time"
import "strings"

# Only allow secrets to be read during business hours
business_hours = rule {
  time.now.hour >= 9 and time.now.hour <= 17
}

# Only allow access from specific IP ranges
allowed_cidrs = [
  "10.0.0.0/8",
  "192.168.0.0/16"
]

valid_cidr = rule {
  request.connection.remote_addr in allowed_cidrs
}

# Combine rules
main = rule {
  business_hours and valid_cidr
}
      HCL

      8.5. Operational Excellence

      Monitoring and Observability

      # Enable Prometheus metrics
vault read sys/metrics?format=prometheus

# Key metrics to monitor
vault read sys/health
vault read sys/leader
vault read sys/seal-status
      Bash

      Essential Vault Metrics

      graph TB
    A[Vault Metrics] --> B[Health Metrics]
    A --> C[Performance Metrics]
    A --> D[Security Metrics]
    A --> E[Storage Metrics]

    B --> B1[vault_core_unsealed]
    B --> B2[vault_core_active]
    B --> B3[vault_core_standby]

    C --> C1[vault_runtime_alloc_bytes]
    C --> C2[vault_barrier_get]
    C --> C3[vault_barrier_put]

    D --> D1[vault_audit_log_request]
    D --> D2[vault_token_creation]
    D --> D3[vault_policy_get_policy]

    E --> E1[vault_wal_loadWAL]
    E --> E2[vault_merkle_flushDirty]
    E --> E3[vault_storage_operation]

    style A fill:#326ce5,color:#fff
    style B fill:#4caf50,color:#fff
    style D fill:#ff9800,color:#fff

      Backup and Recovery Strategy

      #!/bin/bash
# vault-backup.sh - Comprehensive backup script

# Set variables
VAULT_ADDR="https://vault.example.com:8200"
BACKUP_DIR="/backup/vault"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=30

# Create backup directory
mkdir -p ${BACKUP_DIR}/${DATE}

# 1. Integrated storage snapshot (Raft)
if vault operator raft list-peers &>/dev/null; then
    echo "Taking Raft snapshot..."
    vault operator raft snapshot save ${BACKUP_DIR}/${DATE}/vault-snapshot-${DATE}.snap
fi

# 2. Export policies
echo "Backing up policies..."
vault policy list | grep -v "^root$" | while read policy; do
    vault policy read "$policy" > ${BACKUP_DIR}/${DATE}/policy-${policy}.hcl
done

# 3. Export auth methods configuration
echo "Backing up auth methods..."
vault auth list -format=json > ${BACKUP_DIR}/${DATE}/auth-methods.json

# 4. Export secrets engines configuration  
echo "Backing up secrets engines..."
vault secrets list -format=json > ${BACKUP_DIR}/${DATE}/secrets-engines.json

# 5. Export audit devices
echo "Backing up audit configuration..."
vault audit list -format=json > ${BACKUP_DIR}/${DATE}/audit-devices.json

# 6. Compress backup
tar -czf ${BACKUP_DIR}/vault-backup-${DATE}.tar.gz -C ${BACKUP_DIR} ${DATE}
rm -rf ${BACKUP_DIR}/${DATE}

# 7. Cleanup old backups
find ${BACKUP_DIR} -name "vault-backup-*.tar.gz" -mtime +${RETENTION_DAYS} -delete

echo "Backup completed: vault-backup-${DATE}.tar.gz"
      Bash

      Disaster Recovery Procedures

      graph TD
    A[Disaster Detected] --> B{Primary Cluster Available?}
    B -->|Yes| C[Temporary Issue - Monitor]
    B -->|No| D[Activate DR Procedures]

    D --> E[Promote DR Cluster]
    E --> F[Update DNS/Load Balancer]
    F --> G[Verify Application Connectivity]
    G --> H[Monitor Performance]

    H --> I{Primary Recovery Ready?}
    I -->|No| J[Continue on DR]
    I -->|Yes| K[Plan Failback]

    K --> L[Sync Data if Needed]
    L --> M[Update DNS Back]
    M --> N[Verify Primary Operation]

    style A fill:#ff5722,color:#fff
    style E fill:#ff9800,color:#fff
    style G fill:#4caf50,color:#fff

      Upgrade Procedures

      # Rolling upgrade procedure for HA cluster

# 1. Pre-upgrade checks
vault status
vault operator raft list-peers
vault read sys/health

# 2. Backup before upgrade
./vault-backup.sh

# 3. Upgrade standby nodes first
# Stop Vault on standby node
systemctl stop vault

# Update binary
wget https://releases.hashicorp.com/vault/1.16.0/vault_1.16.0_linux_amd64.zip
unzip vault_1.16.0_linux_amd64.zip
sudo cp vault /usr/local/bin/

# Start Vault
systemctl start vault

# Verify node rejoined cluster
vault operator raft list-peers

# 4. Step down active node and upgrade
vault operator step-down

# Repeat upgrade process on former active node

# 5. Post-upgrade verification
vault status
vault read sys/health
vault auth list
vault secrets list
      Bash

      Performance Tuning

      Configuration Optimization

      # performance-tuned-vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  # Performance tuning
  max_entry_size = 1048576
  autopilot_cleanup_dead_servers = true
  autopilot_last_contact_threshold = "10s"
  autopilot_max_trailing_logs = 1000
}

listener "tcp" {
  address = "0.0.0.0:8200"

  # TLS configuration
  tls_cert_file = "/etc/vault/vault.crt"
  tls_key_file = "/etc/vault/vault.key"

  # Performance settings
  tls_min_version = "tls12"
  tls_cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"

  # Connection settings
  tls_client_ca_file = "/etc/vault/ca.crt"
  http_idle_timeout = "5m"
  http_read_header_timeout = "10s"
  http_read_timeout = "30s"
  http_write_timeout = "30s"
}

# Performance parameters
default_lease_ttl = "768h"  # 32 days
max_lease_ttl = "8760h"     # 365 days
disable_mlock = false       # Keep enabled for security
raw_storage_endpoint = true # For debugging only

# Cluster performance
cluster_addr = "https://vault-1.internal:8201"
api_addr = "https://vault-1.internal:8200"

# Logging
log_level = "INFO"
log_format = "json"
      HCL

      9. Best Practices

      Security Best Practices

      graph TB
    A[Vault Security Best Practices] --> B[Access Control]
    A --> C[Network Security]
    A --> D[Audit & Monitoring]
    A --> E[Secret Management]
    A --> F[Operations]

    B --> B1[Principle of Least Privilege]
    B --> B2[Role-Based Access]
    B --> B3[Regular Policy Review]

    C --> C1[TLS Everywhere]
    C --> C2[Network Segmentation]
    C --> C3[Firewall Rules]

    D --> D1[Enable Audit Logs]
    D --> D2[Monitor Access Patterns]
    D --> D3[Alert on Anomalies]

    E --> E1[Short TTLs]
    E --> E2[Regular Rotation]
    E --> E3[Dynamic Secrets]

    F --> F1[Auto-Unseal]
    F --> F2[Regular Backups]
    F --> F3[Disaster Recovery]

    style A fill:#326ce5,color:#fff

      Production Configuration Best Practices

      # production-vault.hcl - Security-hardened configuration
storage "consul" {
  address = "consul.internal:8500"
  path    = "vault/"

  # Use TLS for Consul communication
  scheme = "https"
  tls_ca_file = "/etc/vault/consul-ca.pem"
  tls_cert_file = "/etc/vault/consul-client.pem"
  tls_key_file = "/etc/vault/consul-client-key.pem"
}

listener "tcp" {
  address = "0.0.0.0:8200"

  # Enable TLS
  tls_cert_file = "/etc/vault/vault.crt"
  tls_key_file = "/etc/vault/vault.key"
  tls_min_version = "tls12"

  # Security headers
  tls_cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"

  # Client certificate authentication (optional)
  tls_require_and_verify_client_cert = true
  tls_client_ca_file = "/etc/vault/client-ca.pem"
}

# Auto-unseal with cloud KMS
seal "awskms" {
  region = "us-west-2"
  kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
}

# Enable multiple audit devices for redundancy
audit_device "file" {
  path = "/vault/logs/audit.log"
  log_raw = false
  format = "json"
}

audit_device "syslog" {
  facility = "AUTH"
  tag = "vault"
}

# Disable UI in production (optional)
ui = false

# Performance tuning
default_lease_ttl = "768h"    # 32 days
max_lease_ttl = "8760h"       # 365 days

# Cluster configuration
cluster_addr = "https://vault-node1.internal:8201"
api_addr = "https://vault.example.com:8200"

# Security settings
disable_mlock = false         # Keep enabled for security
disable_cache = false         # Keep cache enabled for performance

# Logging
log_level = "INFO"
log_format = "json"
log_file = "/vault/logs/vault.log"
log_rotate_duration = "24h"
log_rotate_max_files = 30
      HCL

      Policy Design Patterns

      # 1. Environment-based policies
path "secret/data/{{identity.entity.metadata.environment}}/*" {
  capabilities = ["read"]
}

# 2. Application-specific policies  
path "secret/data/apps/{{identity.entity.metadata.app_name}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}

# 3. Team-based access
path "secret/data/teams/{{identity.groups.names.primary}}/*" {
  capabilities = ["read", "list"]
}

# 4. Time-based access
path "secret/data/business-hours/*" {
  capabilities = ["read"]
  condition = "time.hour >= 9 && time.hour <= 17 && time.weekday >= 1 && time.weekday <= 5"
}

# 5. Emergency access pattern
path "secret/data/emergency/*" {
  capabilities = ["read"]
  control_group = {
    factor "emergency_approvers" {
      identity {
        group_names = ["emergency-response"]
        approvals = 2
      }
    }
  }
}

# 6. Read-only pattern with metadata access
path "secret/metadata/readonly/*" {
  capabilities = ["list"]
}

path "secret/data/readonly/*" {
  capabilities = ["read"]
}

# 7. Self-service pattern
path "secret/data/users/{{identity.entity.name}}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
      Bash

      Monitoring and Alerting Best Practices

      # prometheus-vault-config.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'vault'
    static_configs:
      - targets: ['vault:8200']
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    bearer_token: 'vault-monitoring-token'
    scheme: https
    tls_config:
      ca_file: /etc/ssl/vault-ca.pem

# Critical alerts for Vault
rule_files:
  - "vault-alerts.yml"

---
# vault-alerts.yml
groups:
  - name: vault-critical
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault instance {{ $labels.instance }} is sealed"
          description: "Vault has been sealed and is not serving requests"

      - alert: VaultDown
        expr: up{job="vault"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Vault instance {{ $labels.instance }} is down"

      - alert: VaultHighMemoryUsage
        expr: vault_runtime_alloc_bytes / vault_runtime_sys_bytes > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Vault memory usage is high"

      - alert: VaultTokenCountHigh
        expr: vault_token_count_by_auth > 10000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High number of active tokens in Vault"

      - alert: VaultRequestFailureRate
        expr: rate(vault_audit_log_request_failure[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High request failure rate in Vault"

  - name: vault-storage
    rules:
      - alert: VaultStorageErrors
        expr: rate(vault_storage_operation{operation="put",error="true"}[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault storage errors detected"

      - alert: VaultLeadershipChange
        expr: changes(vault_core_active[5m]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Vault leadership has changed"
      YAML

      Development vs Production Differences

      AspectDevelopmentProduction
      UnsealingAuto-unseal (dev mode)Manual or auto-unseal with KMS
      StorageIn-memoryPersistent (Consul/Raft/Database)
      TLSDisabledRequired everywhere
      AuditOptionalMultiple devices required
      PoliciesPermissiveLeast privilege
      MonitoringBasicComprehensive
      BackupsNot requiredDaily automated
      HASingle nodeMulti-node cluster
      Root TokenLong-livedEmergency use only

      Security Checklist

      Pre-Production Security Review

      • TLS enabled on all connections
      • Auto-unseal configured with cloud KMS
      • Root token secured and access limited
      • Comprehensive audit logging enabled
      • Network segmentation implemented
      • Regular backup procedures tested
      • Monitoring and alerting configured
      • Disaster recovery plan documented
      • Security policies reviewed and approved
      • Penetration testing completed

      Operational Security Checks

      • Regular policy reviews (quarterly)
      • Token usage audits (monthly)
      • Access pattern analysis (weekly)
      • Security patch updates (as available)
      • Backup restoration tests (monthly)
      • Disaster recovery drills (quarterly)
      • Security training for operators (annually)

      Common Anti-Patterns to Avoid

      graph TB
    A[Vault Anti-Patterns] --> B[Long-lived Root Tokens]
    A --> C[Overly Permissive Policies]
    A --> D[Disabled Audit Logging]
    A --> E[Single Point of Failure]
    A --> F[Secrets in Plain Text]

    B --> B1[Use short-lived tokens]
    B --> B2[Generate emergency tokens only]

    C --> C1[Apply least privilege]
    C --> C2[Use templated policies]

    D --> D1[Enable multiple audit devices]
    D --> D2[Monitor audit logs]

    E --> E1[Deploy HA clusters]
    E --> E2[Use multiple regions]

    F --> F1[Encrypt all secrets]
    F --> F2[Use dynamic secrets]

    style A fill:#ff5722,color:#fff
    style B fill:#ff5722,color:#fff
    style C fill:#ff5722,color:#fff
    style D fill:#ff5722,color:#fff
    style E fill:#ff5722,color:#fff
    style F fill:#ff5722,color:#fff

C --> C1[TLS Everywhere]
C --> C2[Network Segmentation]
C --> C3[Firewall Rules]

D --> D1[Enable Audit Logs]
D --> D2[Monitor Access Patterns]
D --> D3[Alert on Anomalies]

E --> E1[Short TTLs]
E --> E2[Regular Rotation]
E --> E3[Dynamic Secrets]

F --> F1[Auto-Unseal]
F --> F2[Regular Backups]
F --> F3[Disaster Recovery]

style A fill:#326ce5,color:#fff

      Production Configuration

      # production-vault.hcl
storage "consul" {
  address = "consul.internal:8500"
  path    = "vault/"

  # Use TLS for Consul communication
  scheme = "https"
  tls_ca_file = "/etc/vault/consul-ca.pem"
  tls_cert_file = "/etc/vault/consul-client.pem"
  tls_key_file = "/etc/vault/consul-client-key.pem"
}

listener "tcp" {
  address = "0.0.0.0:8200"

  # Enable TLS
  tls_cert_file = "/etc/vault/vault.crt"
  tls_key_file = "/etc/vault/vault.key"
  tls_min_version = "tls12"

  # Security headers
  tls_cipher_suites = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
}

# Auto-unseal with cloud KMS
seal "awskms" {
  region = "us-west-2"
  kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
}

# Enable auditing
audit_device "file" {
  path = "/vault/logs/audit.log"
  log_raw = false
}

# Disable UI in production (optional)
ui = false

# Performance tuning
default_lease_ttl = "768h"
max_lease_ttl = "8760h"

# Cluster configuration
cluster_addr = "https://vault-node1.internal:8201"
api_addr = "https://vault.example.com:8200"

# Disable mlock if running in containers
disable_mlock = true

# Log level
log_level = "INFO"
      HCL

      Policy Design Patterns

      # 1. Environment-based policies
path "secret/data/{{identity.entity.metadata.environment}}/*" {
  capabilities = ["read"]
}

# 2. Application-specific policies
path "secret/data/apps/{{identity.entity.metadata.app_name}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}

# 3. Team-based access
path "secret/data/teams/{{identity.groups.names.primary}}/*" {
  capabilities = ["read", "list"]
}

# 4. Time-based access
path "secret/data/business-hours/*" {
  capabilities = ["read"]
  condition = "time.hour >= 9 && time.hour <= 17 && time.weekday >= 1 && time.weekday <= 5"
}

# 5. Emergency access pattern
path "secret/data/emergency/*" {
  capabilities = ["read"]
  control_group = {
    factor "emergency_approvers" {
      identity {
        group_names = ["emergency-response"]
        approvals = 2
      }
    }
  }
}
      Bash

      Monitoring and Alerting

      # Prometheus metrics scraping
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'vault'
    static_configs:
      - targets: ['vault:8200']
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    bearer_token: 'vault-monitoring-token'

# Example alerts
groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault is sealed"

      - alert: VaultHighTokenUsage
        expr: vault_token_count_by_auth > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High number of active tokens"

      - alert: VaultLeadershipChange
        expr: changes(vault_core_active[5m]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Vault leadership has changed"
      YAML

      Backup and Recovery

      #!/bin/bash
# vault-backup.sh

# Set variables
VAULT_ADDR="https://vault.example.com:8200"
BACKUP_DIR="/backup/vault"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p ${BACKUP_DIR}/${DATE}

# Integrated storage snapshot
if vault operator raft list-peers &>/dev/null; then
    echo "Taking Raft snapshot..."
    vault operator raft snapshot save ${BACKUP_DIR}/${DATE}/vault-snapshot-${DATE}.snap
fi

# Export policies
echo "Backing up policies..."
vault policy list | grep -v "^root$" | while read policy; do
    vault policy read "$policy" > ${BACKUP_DIR}/${DATE}/policy-${policy}.hcl
done

# Export auth methods
echo "Backing up auth methods..."
vault auth list -format=json > ${BACKUP_DIR}/${DATE}/auth-methods.json

# Export secrets engines
echo "Backing up secrets engines..."
vault secrets list -format=json > ${BACKUP_DIR}/${DATE}/secrets-engines.json

# Compress backup
tar -czf ${BACKUP_DIR}/vault-backup-${DATE}.tar.gz -C ${BACKUP_DIR} ${DATE}
rm -rf ${BACKUP_DIR}/${DATE}

echo "Backup completed: vault-backup-${DATE}.tar.gz"
      Bash

      10. Troubleshooting

      Common Issues and Solutions

      graph TD
    A[Vault Issues] --> B[Startup Problems]
    A --> C[Authentication Issues]
    A --> D[Permission Denied]
    A --> E[Performance Issues]
    A --> F[Network Problems]

    B --> B1[Check Config File]
    B --> B2[Verify Storage Backend]
    B --> B3[Check Permissions]

    C --> C1[Token Expired]
    C --> C2[Wrong Auth Method]
    C --> C3[Invalid Credentials]

    D --> D1[Check Policies]
    D --> D2[Verify Token Capabilities]
    D --> D3[Review Audit Logs]

    E --> E1[Storage Backend Issues]
    E --> E2[Network Latency]
    E --> E3[Resource Constraints]

    F --> F1[Firewall Rules]
    F --> F2[TLS Certificate Issues]
    F --> F3[DNS Resolution]

    style A fill:#ff6b35,color:#fff

      Troubleshooting Matrix

      SymptomLikely CauseInvestigation StepsSolution
      Vault won’t startConfig file errorCheck syntax, validate pathsFix configuration
      “Connection refused”Network/firewallTest connectivity, check portsConfigure network
      “Permission denied”Policy issueReview policies, check tokenUpdate policies
      “Invalid token”Expired/revoked tokenCheck token statusRenew or recreate
      Slow responsesPerformance issueCheck storage backend, networkTune performance
      Sealed vaultManual seal/restartCheck seal statusUnseal vault

      Diagnostic Commands

      # Check Vault status and basic health
vault status
vault read sys/health

# Verify connectivity and authentication
vault auth list
vault token lookup

# Debug storage and clustering
vault read sys/storage/raft/configuration
vault operator raft list-peers

# Performance and metrics
vault read sys/metrics
vault read sys/pprof/goroutine

# Audit and logging analysis
tail -f /vault/logs/audit.log | jq
journalctl -u vault -f

# Policy and permission debugging
vault policy read my-policy
vault token capabilities secret/myapp/config

# Test authentication methods
vault auth -method=userpass username=test
vault auth -method=aws role=my-role

# Storage backend debugging
vault read sys/raw/logical/<mount>/config
vault list sys/raw/logical/
      YAML

      Advanced Troubleshooting

      Debug Mode Startup

      # Start with maximum verbosity
vault server -config=vault.hcl -log-level=TRACE

# Enable specific debugging
export VAULT_LOG_LEVEL=DEBUG
export VAULT_LOG_FORMAT=json

# Debug specific subsystems
vault server -config=vault.hcl -log-level=DEBUG -dev-plugin-dir=/plugins
      Bash

      Network Debugging

      # Test TLS connectivity
openssl s_client -connect vault.example.com:8200 -servername vault.example.com

# Verify certificate chain
openssl verify -CAfile ca.pem vault.pem

# Check DNS resolution
nslookup vault.example.com
dig vault.example.com

# Test network connectivity
telnet vault.example.com 8200
nc -zv vault.example.com 8200
      Bash

      Log Analysis

      # Parse Vault audit logs for patterns
jq '.type' /vault/logs/audit.log | sort | uniq -c

# Find failed authentication attempts
jq 'select(.error != null and .type == "request")' /vault/logs/audit.log

# Monitor token usage patterns
jq 'select(.auth.token_type != null) | .auth.display_name' /vault/logs/audit.log

# Track secret access patterns
jq 'select(.request.path | startswith("secret/")) | .request.path' /vault/logs/audit.log

# Performance monitoring from logs
jq 'select(.request.path == "sys/metrics")' /vault/logs/audit.log

# Policy violations
jq 'select(.error != null and (.error | contains("permission denied")))' /vault/logs/audit.log

# Token creation analysis
jq 'select(.request.path | contains("auth/token/create"))' /vault/logs/audit.log

# Failed seal/unseal operations
jq 'select(.request.path | contains("sys/seal") or contains("sys/unseal"))' /vault/logs/audit.log
      Bash

      Performance Troubleshooting

      Memory Issues

      # Check memory usage
vault read sys/metrics | grep alloc

# Monitor garbage collection
vault read sys/pprof/heap

# Check for memory leaks
vault read sys/pprof/allocs
      Bash

      Storage Performance

      # Raft performance metrics
vault read sys/storage/raft/autopilot/state

# Storage operation latency
vault read sys/metrics | grep storage

# Database connection issues (if using DB storage)
vault read database/config/connection
      Bash

      Recovery Procedures

      Emergency Recovery Steps

      # 1. Recovery from sealed state
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

# 2. Recovery from corrupted storage (Raft)
# Stop all Vault instances
systemctl stop vault

# Restore from snapshot
vault operator raft snapshot restore backup.snap

# Start Vault and verify
systemctl start vault
vault status

# 3. Root token generation (emergency)
vault operator generate-root -init
vault operator generate-root -nonce=<nonce> <unseal-key>

# 4. Revoking all tokens (emergency)
vault write sys/leases/revoke-prefix prefix=auth/

# 5. Force leadership change
vault operator step-down

# 6. Remove problematic node from cluster
vault operator raft remove-peer <node-id>
      Bash

      Data Recovery

      # Export critical data before recovery
vault policy list > policies-backup.txt
vault auth list -format=json > auth-methods-backup.json
vault secrets list -format=json > secrets-engines-backup.json

# Restore specific secrets engine
vault secrets enable -path=recovered-secrets kv-v2

# Import data from another Vault instance
vault kv put recovered-secrets/app/config @original-config.json
      Bash

      Common Error Messages and Solutions

      “Error reading from server”

      # Possible causes: Network issues, TLS problems, wrong address
export VAULT_ADDR=https://correct-vault-address:8200
export VAULT_CACERT=/path/to/ca.pem
vault status
      Bash

      “Token lookup failed: permission denied”

      # Token may be expired or invalid
vault token lookup  # Check if token is valid
vault login -method=userpass username=myuser  # Re-authenticate
      Bash

      “No such host”

      # DNS resolution issues
nslookup vault.example.com
# Add to /etc/hosts if needed
echo "192.168.1.100 vault.example.com" >> /etc/hosts
      Bash

      “Connection refused”

      # Vault not running or wrong port
systemctl status vault
netstat -tlnp | grep 8200
      Bash

      “Operation not permitted”

      # Policy restrictions
vault policy read my-policy
vault token capabilities secret/path/to/resource
      Bash

      Automated Health Checks

      #!/bin/bash
# vault-health-check.sh

VAULT_ADDR=${VAULT_ADDR:-"https://vault.example.com:8200"}
HEALTH_ENDPOINT="$VAULT_ADDR/v1/sys/health"

# Check if Vault is responding
if curl -s -k "$HEALTH_ENDPOINT" > /dev/null; then
    echo "✅ Vault is responding"
else
    echo "❌ Vault is not responding"
    exit 1
fi

# Check seal status
SEALED=$(vault status -format=json | jq -r '.sealed')
if [ "$SEALED" = "false" ]; then
    echo "✅ Vault is unsealed"
else
    echo "❌ Vault is sealed"
    exit 1
fi

# Check cluster health
if vault operator raft list-peers > /dev/null 2>&1; then
    echo "✅ Raft cluster is healthy"
else
    echo "⚠️  Raft cluster may have issues"
fi

# Check authentication
if vault token lookup > /dev/null 2>&1; then
    echo "✅ Authentication working"
else
    echo "❌ Authentication failed"
    exit 1
fi

echo "🎉 All health checks passed"
      Bash

      11. Enterprise Features

      Vault Enterprise Overview

      graph TB
    A[Vault Enterprise] --> B[Governance & Policy]
    A --> C[Multi-tenancy]
    A --> D[Replication]
    A --> E[Advanced Security]

    B --> B1[Sentinel Policies]
    B --> B2[Control Groups]
    B --> B3[Advanced Auditing]

    C --> C1[Namespaces]
    C --> C2[Resource Quotas]
    C --> C3[Mount Filters]

    D --> D1[Performance Replication]
    D --> D2[Disaster Recovery]
    D --> D3[Cross-Cluster Replication]

    E --> E1[HSM Integration]
    E --> E2[Transform Engine]
    E --> E3[KMIP Engine]

    style A fill:#326ce5,color:#fff
    style B fill:#4ecdc4,color:#fff

      Namespaces

      # Create namespace
vault namespace create development

# List namespaces
vault namespace list

# Set namespace context
export VAULT_NAMESPACE=development

# Create sub-namespace
vault namespace create -namespace=development team-alpha

# Create namespace-specific policy
vault policy write -namespace=development app-policy - <<EOF
path "secret/*" {
  capabilities = ["read", "list"]
}
EOF

# Enable auth method in namespace
vault auth enable -namespace=development userpass
      Bash

      Resource Quotas

      # Set lease count quota
vault write sys/quotas/lease-count/dev-quota \
    max_leases=1000 \
    path="auth/userpass/"

# Set rate limit quota
vault write sys/quotas/rate-limit/global \
    rate=1000 \
    interval=1m

# View quota usage
vault read sys/quotas/lease-count/dev-quota

# Delete quota
vault delete sys/quotas/lease-count/dev-quota
      Bash

      HSM Integration

      # vault-hsm.hcl
seal "pkcs11" {
  lib            = "/usr/lib/libCryptoki2_64.so"
  slot           = "0"
  pin            = "AAAA-BBBB-CCCC-DDDD"
  key_label      = "vault-hsm-key"
  hmac_key_label = "vault-hsm-hmac-key"

  # Optional: specify mechanism
  mechanism      = "0x1085"  # CKM_AES_GCM
}

# Alternative: AWS CloudHSM
seal "awskms" {
  region         = "us-west-2"
  kms_key_id     = "alias/vault-key"

  # For CloudHSM cluster
  endpoint       = "https://cloudhsm.us-west-2.amazonaws.com"
}
      Bash

      KMIP Secrets Engine

      # Enable KMIP secrets engine
vault secrets enable kmip

# Configure KMIP
vault write kmip/config \
    listen_addrs=0.0.0.0:5696 \
    server_hostnames=vault.example.com \
    tls_ca_key_type=rsa \
    tls_ca_key_bits=2048

# Create KMIP scope
vault write kmip/scope/my-service

# Create KMIP role
vault write kmip/scope/my-service/role/admin \
    operation_all=true

# Generate client certificate
vault write kmip/scope/my-service/role/admin/credential/generate \
    format=pem
      Bash

      Advanced Auditing

      # Enable multiple audit devices
audit_device "file" {
  path = "/vault/logs/audit.log"

  # Options for enterprise features
  log_raw = false
  hmac_accessor = false
  elide_list_responses = true
  format = "json"

  # Custom audit headers
  prefix = "vault-prod"
}

audit_device "syslog" {
  facility = "AUTH"
  tag = "vault"

  # Socket audit device
  # address = "unix:///var/log/vault/audit.sock"
}

# Sentinel policy for audit
audit_policy = <<EOF
import "audit"

# Require audit for sensitive paths
sensitive_paths = [
  "secret/production/",
  "database/creds/",
  "pki/issue/"
]

main = rule {
  any audit.devices as device {
    device.enabled
  }
}
EOF
      Bash

      12. Real-World Use Cases

      Use Case 1: Application Secret Management

      sequenceDiagram
    participant App
    participant VaultAgent
    participant Vault
    participant DB

    App->>VaultAgent: Start with role ID
    VaultAgent->>Vault: Authenticate with AppRole
    Vault->>VaultAgent: Return token

    VaultAgent->>Vault: Request DB credentials
    Vault->>DB: Create dynamic user
    DB->>Vault: User created
    Vault->>VaultAgent: Return credentials + lease

    VaultAgent->>App: Write config file with creds
    App->>App: Reload configuration

    Note over VaultAgent,Vault: Automatic lease renewal

    VaultAgent->>Vault: Renew lease
    Vault->>VaultAgent: Lease renewed

    Note over Vault: Lease expires
    Vault->>DB: Revoke user

      Implementation: AppRole Authentication

      # Enable AppRole auth method
vault auth enable approle

# Create policy for application
vault policy write myapp-policy - <<EOF
path "database/creds/myapp" {
  capabilities = ["read"]
}

path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

# Create AppRole
vault write auth/approle/role/myapp \
    token_policies="myapp-policy" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=24h

# Get role ID and secret ID
ROLE_ID=$(vault read -field=role_id auth/approle/role/myapp/role-id)
SECRET_ID=$(vault write -field=secret_id auth/approle/role/myapp/secret-id)

# Application authentication
vault write auth/approle/login \
    role_id="$ROLE_ID" \
    secret_id="$SECRET_ID"
      Bash

      Use Case 2: CI/CD Pipeline Integration

      # .github/workflows/deploy.yml
name: Deploy Application

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Import Secrets
        uses: hashicorp/vault-action@v2
        with:
          url: https://vault.example.com:8200
          method: jwt
          role: github-actions
          secrets: |
            secret/data/production/database username | DB_USER ;
            secret/data/production/database password | DB_PASS ;
            secret/data/production/api key | API_KEY

      - name: Deploy Application
        run: |
          # Use secrets in deployment
          ./deploy.sh
        env:
          DATABASE_URL: "postgresql://${{ env.DB_USER }}:${{ env.DB_PASS }}@db.example.com/myapp"
          API_KEY: ${{ env.API_KEY }}
      YAML

      Vault Configuration for CI/CD

      # Configure JWT auth for GitHub Actions
vault auth enable jwt

vault write auth/jwt/config \
    bound_issuer="https://token.actions.githubusercontent.com" \
    oidc_discovery_url="https://token.actions.githubusercontent.com"

# Create role for GitHub Actions
vault write auth/jwt/role/github-actions \
    bound_audiences="https://github.com/myorg" \
    bound_claims='{
      "repository": "myorg/myapp"
    }' \
    user_claim="actor" \
    role_type="jwt" \
    policies="ci-cd-policy" \
    ttl=15m
      Bash

      Appendix A: CLI Quick Reference

      Essential Commands

      # Server Operations
vault server -config=vault.hcl          # Start server
vault operator init                     # Initialize Vault
vault operator unseal                   # Unseal Vault
vault operator seal                     # Seal Vault
vault status                           # Check status

# Authentication
vault auth list                        # List auth methods
vault auth enable userpass             # Enable auth method
vault login -method=userpass           # Login
vault token lookup                     # Check token info
vault token renew                      # Renew token

# Secrets Management
vault kv put secret/app key=value      # Store secret
vault kv get secret/app                # Read secret
vault kv delete secret/app             # Delete secret
vault kv list secret/                  # List secrets

# Policy Management
vault policy write name policy.hcl     # Create policy
vault policy read name                 # Read policy
vault policy list                      # List policies
vault policy delete name               # Delete policy

# Secrets Engines
vault secrets enable database          # Enable engine
vault secrets list                     # List engines
vault secrets disable database         # Disable engine
vault secrets tune -ttl=1h database    # Tune engine
      Bash

      Environment Variables

      # Core Configuration
export VAULT_ADDR="https://vault.example.com:8200"
export VAULT_TOKEN="hvs.your-token-here"
export VAULT_NAMESPACE="development"

# TLS Configuration
export VAULT_CACERT="/path/to/ca.pem"
export VAULT_CLIENT_CERT="/path/to/client.pem"
export VAULT_CLIENT_KEY="/path/to/client-key.pem"
export VAULT_SKIP_VERIFY="false"

# Advanced Options
export VAULT_FORMAT="json"
export VAULT_CLI_NO_COLOR="1"
export VAULT_MAX_RETRIES="3"
export VAULT_RATE_LIMIT="10"
      Bash

      Appendix B: Configuration Templates

      Development Configuration

      # dev-vault.hcl
ui = true
disable_mlock = true

storage "file" {
  path = "/tmp/vault-data"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
      Bash

      Production Configuration

      # prod-vault.hcl
ui = false
disable_mlock = false

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"

  retry_join {
    leader_api_addr = "https://vault-2.example.com:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/vault.crt"
  tls_key_file  = "/etc/vault/vault.key"
  tls_min_version = "tls12"
}

seal "awskms" {
  region     = "us-west-2"
  kms_key_id = "alias/vault-unseal-key"
}

api_addr = "https://vault-1.example.com:8200"
cluster_addr = "https://vault-1.example.com:8201"

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}
      Bash

      Appendix C: Policy Examples

      Application Policy Template

      # Application-specific policy template
path "secret/data/{{identity.entity.metadata.app_name}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}

path "database/creds/{{identity.entity.metadata.app_name}}" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}
      Bash

      Team-Based Policy

      # Team-based access control
path "secret/data/teams/{{identity.groups.names.primary}}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/metadata/teams/{{identity.groups.names.primary}}/*" {
  capabilities = ["list"]
}

# Shared resources
path "secret/data/shared/*" {
  capabilities = ["read", "list"]
}
      Bash

      Conclusion

      This comprehensive guide covers HashiCorp Vault from basic concepts to advanced enterprise features. Key takeaways:

      1. Start Simple: Begin with basic KV storage and gradually adopt advanced features
      2. Security First: Always use TLS, proper authentication, and least-privilege policies
      3. Plan for Scale: Design your Vault architecture with high availability in mind
      4. Automate Everything: Use Vault Agent, operators, and CI/CD integration
      5. Monitor Actively: Implement comprehensive logging, metrics, and alerting
      1. Week 1-2: Complete installation and basic operations
      2. Week 3-4: Master authentication methods and policies
      3. Week 5-6: Implement secrets engines for your use cases
      4. Week 7-8: Set up high availability and monitoring
      5. Week 9-10: Explore advanced features and enterprise capabilities
      6. Week 11-12: Practice troubleshooting and operations

      Additional Resources

      Community and Support

      • Slack: HashiCorp Community Slack
      • Reddit: r/hashicorp
      • Stack Overflow: Questions tagged with hashicorp-vault
      • YouTube: HashiCorp official channel
      • Conferences: HashiConf, local HashiCorp User Groups

      This comprehensive guide provides a solid foundation for mastering HashiCorp Vault. Continue learning through hands-on practice, community engagement, and staying updated with the latest features and best practices.

      Use Case 3: Database Credential Rotation

      graph LR
    A[Vault] --> B[Database Engine]
    B --> C[PostgreSQL]
    B --> D[MySQL]
    B --> E[MongoDB]

    F[Application 1] --> A
    G[Application 2] --> A
    H[Application 3] --> A

    A --> I[Dynamic Credentials]
    I --> J[Auto Rotation]
    I --> K[Lease Management]
    I --> L[Audit Trail]

    style A fill:#326ce5,color:#fff
    style I fill:#4ecdc4,color:#fff

      Database Secrets Configuration

      # Configure PostgreSQL connection
vault write database/config/postgresql \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/myapp?sslmode=require" \
    allowed_roles="readonly,readwrite" \
    username="vault-admin" \
    password="vault-password"

# Create readonly role
vault write database/roles/readonly \
    db_name=postgresql \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# Create readwrite role
vault write database/roles/readwrite \
    db_name=postgresql \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    rotation_statements="ALTER ROLE \"{{name}}\" WITH PASSWORD '{{password}}';" \
    default_ttl="1h" \
    max_ttl="24h"
      Bash

      Use Case 4: PKI Certificate Management

      graph TB
    A[Vault PKI Root CA] --> B[Intermediate CA 1]
    A --> C[Intermediate CA 2]

    B --> D[Web Server Certs]
    B --> E[Client Certs]

    C --> F[Service Mesh Certs]
    C --> G[API Gateway Certs]

    H[Cert Manager] --> B
    I[Applications] --> C
    J[Load Balancers] --> B

    style A fill:#ff6b35,color:#fff
    style B fill:#326ce5,color:#fff
    style C fill:#326ce5,color:#fff

      PKI Setup and Certificate Issuance

      # Enable PKI engine for root CA
vault secrets enable -path=pki-root pki
vault secrets tune -max-lease-ttl=87600h pki-root

# Generate root CA
vault write pki-root/root/generate/internal \
    common_name="Example Corp Root CA" \
    ttl=87600h

# Enable intermediate CA
vault secrets enable -path=pki-int pki
vault secrets tune -max-lease-ttl=43800h pki-int

# Generate intermediate CSR
vault write -format=json pki-int/intermediate/generate/internal \
    common_name="Example Corp Intermediate CA" \
    | jq -r '.data.csr' > pki_intermediate.csr

# Sign intermediate certificate
vault write -format=json pki-root/root/sign-intermediate \
    csr=@pki_intermediate.csr \
    format=pem_bundle ttl="43800h" \
    | jq -r '.data.certificate' > intermediate.cert.pem

# Set intermediate certificate
vault write pki-int/intermediate/set-signed \
    certificate=@intermediate.cert.pem

# Create role for web servers
vault write pki-int/roles/web-server \
    allowed_domains="example.com" \
    allow_subdomains=true \
    max_ttl="720h"

# Issue certificate
vault write pki-int/issue/web-server \
    common_name="app.example.com" \
    ttl="24h"
      Bash

      Use Case 5: Kubernetes Integration

      # vault-k8s-auth.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
---
# Application deployment with Vault integration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          export API_KEY="{{ .Data.data.api_key }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          {{- end }}
    spec:
      serviceAccountName: myapp
      containers:
      - name: myapp
        image: myapp:latest
        command: ["/bin/sh"]
        args: ["-c", "source /vault/secrets/config && ./app"]
      YAML

      Conclusion

      This comprehensive guide covers HashiCorp Vault from basic concepts to advanced enterprise features. Key takeaways:

      1. Start Simple: Begin with basic KV storage and gradually adopt advanced features
      2. Security First: Always use TLS, proper authentication, and least-privilege policies
      3. Plan for Scale: Design your Vault architecture with high availability in mind
      4. Automate Everything: Use Vault Agent, operators, and CI/CD integration
      5. Monitor Actively: Implement comprehensive logging, metrics, and alerting

      Next Steps

      1. Set up a development environment
      2. Practice with different authentication methods
      3. Implement dynamic secrets for your databases
      4. Explore enterprise features if applicable
      5. Build automation around Vault operations

      Remember: Vault is a powerful tool that requires careful planning and implementation. Always test thoroughly in non-production environments before deploying to production.

      Discover more from Altgr Blog

      Subscribe to get the latest posts sent to your email.

      Related Posts

      Leave a Reply

      Your email address will not be published. Required fields are marked *