PABCD Initiative Documentation Hub pabcd_initiative codexclaw cli-jaw

dev-devops

Source: skills/dev-devops/SKILL.md

Container builds, deploy pipelines, K8s, IaC, SRE, edge/serverless, ML infra.

왜 별도 모듈인가
인프라 작업은 단일 실수가 프로덕션 전체를 다운시킬 수 있다. unpinned base image, root로 실행되는 컨테이너, 시크릿이 노출된 CI 로그. 이 스킬은 컨테이너 빌드부터 배포 전략, Kubernetes, IaC, SRE까지 인프라 전 영역의 STRICT 규칙을 한곳에 모아서 에이전트가 인프라 작업 시 안전하게 움직이게 한다.
LLM 단독 사용 시 발생하는 문제

LLM의 학습 데이터에는 2022-2023년의 Dockerfile과 CI 설정이 압도적으로 많다. 그래서 FROM node:latest, USER root, ARG로 시크릿 전달 같은 안티패턴을 자연스럽게 생성한다. 이건 학습 데이터의 분포 문제다 — 나쁜 Dockerfile이 좋은 Dockerfile보다 인터넷에 훨씬 많다. 또한 OIDC/Trusted Publishing은 2024-2025년에 주류가 되었기 때문에 학습 데이터에 반영이 부족하다. LLM은 기본적으로 NPM_TOKEN을 먼저 요구하는데, 이건 이미 열등한 폴백 경로다. Kubernetes도 마찬가지다 — Ingress 예제가 학습 데이터에 압도적으로 많아서 Gateway API v1.6+를 자발적으로 쓰지 않는다.

이 스킬이 해결하는 실제 문제

OIDC/Trusted Publishing 우선 원칙

2025-2026년 npm과 PyPI가 Trusted Publishing(OIDC)을 기본 지원하면서, 장기 토큰(NPM_TOKEN, PYPI_TOKEN)은 보안상 열등한 폴백이 되었다. 에이전트는 관성적으로 토큰을 먼저 요청하는데, package-release.md의 Auth Decision Table은 OIDC를 먼저 시도하고 토큰은 OIDC가 불가능할 때만 쓰게 한다.

FROM node:latest의 재현 불가능 빌드

에이전트가 Dockerfile을 만들 때 FROM node:latest를 쓰면 다음 주에 빌드가 깨진다. section 1.1은 버전+SHA256 다이제스트 고정을 STRICT 규칙으로 강제한다.

컨테이너 이미지 서명과 SBOM의 의무화

2026년 Sigstore/Cosign 기반 이미지 서명이 표준이 되었고, SBOM 생성(Syft/Docker Scout)이 컴플라이언스 요구사항이다. section 1.2는 스캔 -> SBOM -> 서명 -> 다이제스트 프로모트를 하나의 파이프라인으로 묶는다.

Kubernetes Gateway API v1.6+ 전환

Ingress는 deprecated되고 Gateway API가 표준이 되었다. 에이전트의 학습 데이터에는 Ingress 예제가 더 많아서 구형 패턴을 생성한다. kubernetes.md는 Gateway API v1.6+ 기준으로 작성되었다.

Triggers: Dockerfile, K8s, CI/CD, Terraform, IaC, release, deploy, SRE

Key Concepts

  • Dockerfile Rules (multi-stage, non-root, pinned)
  • Image Security & Supply Chain
  • CI/CD Pipeline Design
  • Kubernetes (Gateway API v1.6+)
  • Infrastructure as Code (OpenTofu/Pulumi)
  • SRE Foundations (SLO/SLI/Error Budget)

Reference Documents

DocumentDescription
CI/CD & Deploy — Pipeline & Delivery PatternsGHA 재사용 워크플로, 배포 전략, 롤백, GitOps
Cross-platform Release — CI Matrix vs Local OS Proof
Docker — Container Build & Security Patterns멀티스테이지 빌드, distroless, Docker Scout/Trivy, SBOM/Cosign
Edge & Serverless — Edge Computing Patterns엣지 요청 셰이핑, Cloudflare Workers, Vercel Edge
Homebrew — Formula & Cask Distribution
Infrastructure as Code — OpenTofu, Terraform, PulumiOpenTofu/Terraform 모듈, Pulumi, 상태 암호화, 블래스트 반경 격리
Kubernetes — Deployment & Orchestration PatternsGateway API, Kustomize, HPA/VPA, Helm, ArgoCD GitOps
ML Infrastructure — GPU Clusters, Model Registry, ServingGPU 클러스터, 모델 레지스트리, 스케일링, 엣지 추론
Package Release — Trusted Publishing & Registry Authnpm/PyPI 신뢰 퍼블리싱(OIDC), 레지스트리 인증 모델
Platform Engineering — DevOps Capability Routing
SRE Foundations — SLO, Incident Response, PostmortemSLO/SLI/에러 예산, 번 레이트 알러팅, 인시던트 대응

Sections Overview

SectionSummary
Container BuildsDockerfile 규칙, 멀티스테이지, non-root, 의존성 우선 복사.
Image SecuritySBOM 생성, 취약점 스캔, 서명, 다이제스트 기반 프로모션.
CI/CD Pipeline재사용 워크플로, 배포 전략(blue-green/canary), 롤백 절차.
KubernetesGateway API, Kustomize 오버레이, 오토스케일링, GitOps.
SRE FoundationsSLO/SLI 정의, 에러 예산, 인시던트 대응, 사후 분석.

Academic References

PaperYearRelevance
SoK: Software Supply Chain Attacks and Countermeasures
arXiv:2307.07529
2023SSC 공격 분류와 SLSA/Sigstore/SBOM 대응책 매핑.
Systematically Identifying Security Smells in IaC Scripts
arXiv:1907.07159
2021Puppet/Ansible/Chef의 하드코딩 시크릿, 암호화 누락 등 안티패턴 식별.

Official Guides

Full Specification

Show full SKILL.md content

Dev-DevOps — Production Infrastructure & Delivery

Build reliable, secure, and automated infrastructure and delivery pipelines.

This skill has modular references for specialized guidance — read the relevant ones before coding.

> C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

Severity mapping: CRITICAL/HIGH ⇒ STRICT; MEDIUM ⇒ DEFAULT (aligned with dev §0.2).

Modular References

FileWhen to ReadWhat It Covers
references/docker.mdContainer build/deployMulti-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign
references/package-release.mdPackage publishing / release authnpm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table
references/cross-platform-release.mdCross-platform release proofCI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries
references/homebrew.mdHomebrew distributionFormula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof
references/platform-engineering.mdPlatform / DORA / provider routingDORA capabilities, platform guardrails, provider table rows, SLSA handoff
references/kubernetes.mdK8s deploymentGateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps
references/ci-cd-deploy.mdDeploy pipelineGHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery
references/iac.mdInfrastructure codeOpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation
references/sre-foundations.mdOperations/incidentsSLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem
references/edge-serverless.mdEdge/serverless workEdge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage
references/ml-infra.mdML infrastructureGPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns

Read package-release.md for package publishing, registry auth, npm/PyPI

trusted publishing, Bun-to-npm release decisions, and downstream package

channels. Read cross-platform-release.md when a release claim depends on

OS-local behavior that CI may not prove. Read homebrew.md for Formula/Cask

distribution work. Read platform-engineering.md for broader DevOps capability

refresh, DORA, provider routing, and platform guardrails. Read docker.md + ci-cd-deploy.md first for containerized deploy workflows.

For K8s-specific work, add kubernetes.md. For SRE/on-call, add sre-foundations.md.

When release, registry-auth, provider-doc, service-status, image/platform

version, or package-manager behavior depends on current external evidence, read

the active search skill and follow its source-fetch and evidence-status rules

instead of relying on stale memory or copied snippets.

---

§1 Container Builds

§1.1 Dockerfile Rules (STRICT)

RuleDetail
Multi-stageSeparate build and runtime stages; final image has no build tools
Base imagePin version + SHA256 digest: node:22-slim@sha256:abc...
Minimal runtime basePrefer gcr.io/distroless/* (Debian-based, keyless-signed) — no shell/package manager; OR Chainguard/Wolfi images when nightly rebuilds, SBOM attestations, patch SLAs, or compliance matter
Non-rootUSER nonroot:nonroot (distroless) or create dedicated user
Dependency-first copyCOPY package.json bun.lock ./ → install → COPY . . for layer caching
BuildKit secretsRUN --mount=type=secret,id=token ... — never use ARG for secrets
.dockerignore.git, node_modules, .env*, *.log, dist/, coverage/, __pycache__/

For canonical Dockerfile templates, read references/docker.md §1.

§1.2 Image Security & Supply-Chain Baseline (STRICT)

CRITICAL/HIGH findings → block push. No exceptions. The 2026 baseline is ONE workflow,

not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) →

vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based

promote (§2.4). Read references/docker.md §4 for scan/SBOM/sign command examples, and

../dev-security/references/supply-chain-sbom.md for deeper SBOM/signing policy.

§1.3 Anti-Patterns

BannedSymptomFix
FROM node:latestIrreproducible buildsPinned version + digest
USER root in final stageAttack surfaceNon-root user
COPY . . as first instructionCache invalidation on every changeDependency files first
ARG SECRET=xxxExposed in image historyBuildKit --mount=type=secret
No scan before pushVulnerable images in prodTrivy/Scout CI gate
apt-get install without cleanupBloated image--no-install-recommends && rm -rf /var/lib/apt/lists/*

---

§2 Deploy Pipeline

§2.1 Pipeline Stages (DEFAULT)

[dev-testing §5]  lint → typecheck → test → contract → e2e
[dev-devops]      build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod

§2.2 GHA Reusable Workflows (DEFAULT)

# .github/workflows/ci.yml (caller)
jobs:
  build:
    uses: org/templates/.github/workflows/build-test.yml@v2
    with:
      service: payments
    secrets: inherit
RuleDetail
workflow_callCentral CI template, max 10-level nesting
PermissionsCaller cannot escalate; downgrade only
Environmentenvironment: production + required reviewers + prevent self-review
PromoteDigest-based (image@sha256:...), never mutable tags

§2.3 Deploy Strategies (DEFAULT)

StrategyToolWhenRisk
Rolling updateK8s DeploymentStateless, low riskLow
Blue-greenArgo Rollouts blueGreen:Instant rollback, no DB migrationMedium
CanaryArgo Rollouts canary: steps:Traffic % control, metric-based promoteMedium
ProgressiveFlaggerAuto analysis + rollback, A/B testingMedium-High
Feature flagLaunchDarkly / UnleashCode-level gradual rolloutLow

§2.4 Rollback Rules (STRICT)

  • Every deployment must be rollback-capable within 5 minutes
  • Digest-based promote only — mutable tags are banned
  • DB migrations: forward-only + backward-compatible (expand-contract pattern)
  • Post-rollback: automatic Slack/PagerDuty notification

§2.5 Secret Management (STRICT)

SourceUsage
GHA Secrets / Vault / AWS SMCI pipeline secrets
External Secrets OperatorK8s → Vault/AWS SM sync
.env filesNever committed — generated in CI
Rotation90-day cycle or immediate on incident

§2.6 GitOps (DEFAULT)

  • Actions = CI, ArgoCD = CD — separation of concerns
  • Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles
  • Self-heal: ArgoCD auto-reverts drift
  • Environment protection: GitHub Environments for prod approval gate

---

§3 Kubernetes Basics

§3.1 Minimum Viable K8s (DEFAULT)

ResourcePurpose
DeploymentPod template + replica management
ServiceInternal networking
HTTPRoute (Gateway API)External traffic routing — not Ingress
ResourceQuotaRequest/limit enforcement
ProbesLiveness + readiness + startup
NamespaceEnvironment isolation (dev/staging/prod)

§3.2 Gateway API (v1.6+, verified 2026-07-02)

Gateway API is the successor for new routing while Ingress remains GA but feature-frozen

(not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role

separation: platform team owns GatewayClass + Gateway, app team owns HTTPRoute.

Progressive delivery pairs directly with it: Flagger supports Gateway API HTTPRoute canaries.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-route
spec:
  parentRefs:
    - name: shared-gateway
  hostnames: ["payments.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: payments-svc
          port: 8080

§3.3 Scaling (DEFAULT)

MechanismScopeWatch
HPACPU/memory + custom metricsDon't combine with VPA on same metric
VPARequest auto-tuningUse Off mode for recommendations only
PDBDisruption budgetminAvailable: 50% or maxUnavailable: 1

§3.4 Anti-Patterns

BannedFix
Ingress (new projects)Gateway API HTTPRoute
No resource limitsAlways set requests + limits
image: app:latestSHA digest or pinned SemVer
Single replica in prodMinimum 2 + PDB
Secrets in ConfigMapK8s Secret + External Secrets Operator
Annotation-based routingGateway API native fields

---

§4 Infrastructure as Code

§4.1 OpenTofu/Terraform Rules (DEFAULT)

RuleDetail
StateRemote backend required (S3+DynamoDB / TF Cloud / OpenTofu)
EncryptionOpenTofu native state/plan encryption via KMS
Blast radiusSeparate state per app/layer/env
ModulesPurpose-built (vpc, iam, ecs-service), typed I/O
Applyplan → PR review → apply; auto-apply staging only
VersionsPin provider + module versions explicitly

§4.2 Tool Selection (HEURISTIC)

ToolBest ForStatus (verified 2026-07-02)
OpenTofu (HCL)Open-source/licensing-neutral IaC default (MPL-2.0, Linux Foundation; v1.12.x)✅ Recommended for OSS neutrality
Terraform / HCP Terraform (BSL)Vendor support or HashiCorp platform integration required✅ Active (BSL license — check competitive-use terms)
Pulumi (TS/Python)Teams preferring programming languages✅ Active
AWS CDKAWS-only infrastructure✅ Active (AWS only)
CDKTF❌ Deprecated 2025-12-10; repo archived/read-only, no further fixes

§4.3 Anti-Patterns

BannedFix
Local state fileRemote backend required
Manual console changesAll changes via code
Hardcoded valuesVariables + tfvars
Monolithic main.tfModular decomposition
CDKTF (new projects)OpenTofu or Pulumi
Unpinned provider versionsExplicit version constraints

---

§5 SRE Foundations

§5.1 SLO/SLI (DEFAULT)

SLIMeasurementTypical SLO
AvailabilitySuccess requests / total99.9% (28-day rolling)
Latencyp50/p95/p99 response timep99 < 500ms
Error rate5xx / total< 0.1%
FreshnessData update delay< 5min (pipelines)

SLIs measure user experience, not infrastructure metrics. "CPU is fine ≠ users are fine."

Error budget = 1 − SLO (99.9% → 0.1% budget).

DORA 2025 (verified 2026-07-02): AI acts as an *amplifier* — returns depend on the

underlying sociotechnical system. For AI-agent-heavy delivery, invest in golden paths,

guardrails, observability, provenance, and review gates, not just faster merges.

§5.2 Error Budget Policy (DEFAULT)

Budget StateAction
Normal (>50%)Continue releases, routine monitoring
Accelerated burn (20–50%)Heightened alerts, slow releases, reliability triage
Exhausted (≤0%)Feature freeze — security/bugfix only; VP exception required

Single incident consuming >20% → mandatory postmortem.

Two consecutive window misses → architecture review.

§5.3 Incident Response (DEFAULT)

  1. DetectTriage (S1/S2/S3) → StabilizeFixPostmortem
  2. Roles: IC, Primary Responder, Comms Lead, Scribe
  3. Mitigation first, diagnosis second during active incidents
  4. All S1/S2 → mandatory blameless postmortem within 5 business days
  5. Status updates: S1 every 15min, S2 every 30min

§5.4 Runbook Template (HEURISTIC)

## [Service] — [Symptom]
### Diagnosis
1. Logs: `kubectl logs -l app=<name> --tail=100`
2. Metrics: Grafana → [dashboard URL]
3. Dependencies: `curl -s http://<dep>/health | jq .`
### Emergency Mitigation
1. Rollback: `argocd app rollback <app>`
2. Traffic block: ...
### Root Fix
1. ...
### Escalation
- Owner: @team-sre
- PagerDuty: [policy]

§5.5 Anti-Patterns

BannedFix
Infrastructure-only SLIsUser-experience-based SLIs
SLO without consequencesError budget policy with freeze gate
Too many SLIs (>5 per service)2-4 meaningful SLIs
Page on every deviationBurn-rate multi-window alerting
Blame individualsBlameless postmortem, system improvement
No error budget policyDefine 3-stage policy (normal/accelerated/exhausted)

---

§6 Cross-References

TopicCanonical OwnerWhat dev-devops defers Test strategy & CI test stagesdev-testing §5Test pyramid, coverage gates
Backend observability code patternsdev-backend observability.mdOTel SDK setup, structured logging
Security hardening (app-layer)dev-securityOWASP, auth, input validation
SBOM/signing depthdev-security references/supply-chain-sbom.mdSupply-chain evidence policy beyond image scan gates
Architecture module boundariesdev-architectureCoupling taxonomy, barrel discipline
Scaffolding conventionsdev-scaffoldingFile naming, project structure
Frontend build/bundledev-frontendVite/webpack config, SSR

dev-devops owns: container builds, image security, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra.

dev-backend owns: application-layer observability code, API design, health check implementation.

Overlap: observability alerting rules (dev-devops §5) ↔ observability code instrumentation (dev-backend observability.md). Cross-ref both.

Pre-flight Checklist

Before submitting infrastructure changes:

  • [ ] Dockerfile is multi-stage with distroless/slim final image and non-root user
  • [ ] Image scanned (Trivy/Scout) with CRITICAL/HIGH gate — no unresolved findings
  • [ ] SBOM generated and attestation signed (Cosign) for production images
  • [ ] Deploy pipeline uses digest-based promotion, never mutable tags
  • [ ] K8s manifests have resource requests/limits, probes, PDB, and use Gateway API (not Ingress)
  • [ ] IaC uses remote state, pinned versions, and modular decomposition
  • [ ] Secrets are managed through Vault/AWS SM/GHA Secrets — no .env commits, no ARG secrets
  • [ ] SLO/SLI defined with error budget policy and burn-rate alerting
  • [ ] Rollback plan documented and tested — <5min rollback capability confirmed
  • [ ] Runbook exists for critical failure scenarios