dev-debugging

Dev-Debugging — Systematic Root Cause Analysis

This skill is the thinking process for fixing bugs. It activates by change surface for errors and diagnoses, and enforces a structured

5-phase methodology for every technical issue — test failures, runtime errors,

build failures, performance regressions, integration bugs.

Boundary: This skill covers how to reason about bugs. For test harness,

reproduction frameworks, and verification tooling, see dev-testing. For

domain-specific context (API errors, hydration issues, query performance),

consult dev-backend or dev-frontend.

> C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev-debugging = root cause methodology (the thinking)
dev-testing   = test harness for reproducing/verifying (the tooling)
dev §2        = summary pointer to this skill (the overview)

---

Core Principle

Check if the problem is structural before debugging code.

Complete root cause investigation before proposing any fix.

If Phase 1 is not done, keep investigating.

---

When to Activate

• Test failures, runtime errors, build failures, performance regressions
• Integration issues (API, database, third-party), CI pipeline failures
• Especially: when under time pressure or when "just one quick fix" seems obvious — that's when methodology matters most

---

The Phases

Phase 0: Is This a Bug or a Design Problem?

Before debugging code, ask: "Could this be a structural/design issue rather

than a code bug?" Patching symptoms of architectural debt creates an endless

stream of "bugs" that are really design consequences.

Decision Tree — escalate to architecture review if any apply:

If structural: escalate to architecture review. Do not patch the symptom —

the patch creates the next bug.

Symptom vs Root Cause Fix:

If none of the above apply — proceed to Phase 1 (it's a code bug, not a

design problem).

---

Phase 1: Root Cause Investigation

Feedback loop gate: For UI, browser, TUI, visual, streaming, or agent-output bugs,

first create a red-capable loop that can fail before the fix: screenshot/assertion,

recorded terminal bytes, Playwright visual check, log fixture, or a manual repro script

with explicit pass/fail evidence. Do not patch from screenshots alone when a repeatable

probe can be built in reasonable time.

Complete these before attempting any fix:

1. Read the full error — stack trace, line numbers, error code, surrounding

context. Do not skim. The answer is often in the error message itself.

1. Reproduce consistently — exact steps to trigger the bug. If intermittent,

document frequency, conditions, and environment state. A bug you cannot

reproduce is a bug you cannot verify as fixed.

1. Check recent changes — run git log --oneline -10 and git diff. Check

new dependencies, config changes, environment variables. Bugs correlate with

recent changes most of the time.

1. Trace data flow — where does the bad value originate? Trace backward from

the failure point through the call stack until you find the source. Follow the

full causal chain from trigger → boundary → bad state → failure. Removing the

visible symptom is not a fix unless the defect that creates the bad state is gone.

1. Instrument component boundaries — for multi-layer systems (API → service →

database, CI → build → deploy), log input/output at each boundary BEFORE

proposing fixes.

1. Trace-first for distributed/async/agent failures (DEFAULT) — capture the

evidence trail before hypothesizing: request IDs, OpenTelemetry spans/logs,

Playwright traces/videos, and exact agent tool transcripts. For order-dependent,

native, concurrency, or intermittent failures that logs cannot explain, use

time-travel/replay debugging (Microsoft TTD on Windows, rr on Linux) — see

references/tool-guides.md.

For EACH component boundary:
  - Log what data enters the component
  - Log what data exits the component
  - Verify environment/config propagation
Run once → analyze evidence → identify failing layer → investigate THAT layer

Work through these steps; skip only if clearly irrelevant to the problem at hand.

Phase 2: Pattern Analysis

1. Find working examples — similar working code in the same codebase. If it

worked before, use git bisect to find the breaking commit (see

references/tool-guides.md).

1. Compare systematically — list every difference between working and broken

code. No matter how small. Resist assuming "that can't matter."

1. Read reference docs completely — official documentation for the library,

API, or framework involved. Don't skim — read the full relevant section.

1. Check known issues — GitHub Issues, changelogs, migration guides. Someone

may have hit the same bug. Search with the exact error message.

When the bug depends on third-party library/API/framework behavior, current

error workarounds, upstream issues, changelogs, or migration guides, read the

active search skill and follow its source-fetch and evidence-status rules

before treating external material as proof.

Phase 3: Hypothesis and Testing

1. List competing hypotheses first — write at least three plausible root-cause

hypotheses before investigating any single one. Include the evidence that would

support or reject each hypothesis. If fewer than three are plausible, state why.

1. State the leading hypothesis explicitly — "X is the root cause because

evidence Y shows Z." If you can't articulate it clearly, you don't understand

it yet.

1. Design a test to disprove — falsification is stronger than confirmation.

What would you expect to see if your hypothesis is wrong?

1. Test one variable — smallest possible change, one variable at a time.

Never fix multiple things at once.

1. If it fails → move to another listed hypothesis. Revert the failed change and

start from clean state. Stacking fixes obscures the root cause.

1. Keep the rejection record — preserve rejected hypotheses and the evidence

that rejected them. The final report must include them, not just the winning cause.

1. Admit ignorance — "I don't understand X" is a valid finding. Research

further rather than guessing. Record the open question explicitly.

Phase 4: Implementation

1. Write a failing test first — the test reproduces the bug. It should fail

before the fix. Use dev-testing for TDD patterns and test harness setup.

1. Make the minimal fix — address the root cause, not symptoms. One logical

change only.

1. Verify: the test passes, no regressions (run the full test suite:

npm test / pytest / equivalent).

1. Check for similar patterns — does the same bug class exist elsewhere in

the codebase? Search for it. Fix all instances, not just the one you found.

1. Document — final report and commit message explain root cause AND fix,

including rejected hypotheses and rejection evidence. Not "fixed bug"

but "fix: race condition in session middleware caused by missing await on

Redis write."

---

Red Flags — Return to Phase 1

If you catch yourself doing any of these, pause — root cause investigation

was likely skipped.

Repeated Failure Rule: After repeated failed fix attempts, pause entirely.

Each fix revealing a new problem in a different place is a sign of

architectural issues, not simple bugs. Discuss with the user before

attempting more fixes.

---

Slop Debugging Patterns

Slop debugging is spray-and-pray: guess, patch, pray, repeat.

---

Concrete Debugging Scenarios

Scenario A: API Returns 500

Root cause pattern: Missing input validation lets undefined values propagate into business logic. Instrument controller/service/repository boundaries to find where the bad value enters. Compare with a working endpoint that validates input with a schema. Fix: add schema validation at the entry point, write a test that sends invalid input and expects 400.

Worked example:

curl -i -X POST http://localhost:3000/api/orders \
  -H 'content-type: application/json' \
  -d '{"sku":"book-1"}'

Observed failure:

HTTP/1.1 500 Internal Server Error
TypeError: Cannot read properties of undefined (reading 'toFixed')
    at calculateTotal (src/orders/service.ts:42:21)
    at createOrder (src/orders/controller.ts:27:18)

Competing hypotheses before narrowing:

1. Request validation allows missing quantity.
2. Controller mapping drops quantity before service call.
3. Repository returns an order row with quantity = null.

Boundary instrumentation:

DEBUG=orders:* npm run dev
curl -s -X POST http://localhost:3000/api/orders \
  -H 'content-type: application/json' \
  -d '{"sku":"book-1"}' | jq .

Sample log output:

orders:controller input {"sku":"book-1"}
orders:controller mapped {"sku":"book-1"}
orders:service input {"sku":"book-1"}
orders:repository skipped insert due service error

Rejections: repository-null is rejected because the repository is never reached.

Controller-drop is rejected because controller input already lacks quantity.

Root cause: entry validation accepts a payload missing a required domain field.

Fix at the entry boundary: schema rejects missing quantity; regression test posts

the same payload and expects HTTP 400 with a stable error.code.

Scenario B: React Hydration Mismatch

Root cause pattern: Server renders a value (e.g., date, locale string) that differs from client-side rendering due to environment differences (UTC vs. local timezone). Compare with components that defer environment-dependent rendering to useEffect. Fix: move environment-dependent formatting into a client component.

Scenario C: N+1 Query Performance

Root cause pattern: List endpoint lazy-loads related records per item (1 query + N queries). Enable query logging to count queries, then compare with an endpoint that uses eager loading. Fix: add include/joinedload, write a test asserting bounded query count.

Scenario D: Flaky Test (Intermittent Failure)

Root cause pattern: Test passes in isolation but fails in suite due to shared mutable state (database rows, global variables, uncleared mocks). Compare with stable tests that use transaction rollback in beforeEach/afterEach. Fix: add proper test isolation, then search for other tests missing cleanup.

---

When to Escalate vs When to Keep Digging

Keep Digging When:

• You have untested hypotheses from Phase 2
• You haven't read the full error message or stack trace
• You haven't checked recent changes (git log, git diff)
• You haven't found working comparison code yet
• The bug is in YOUR code (not a third-party library)
• You still have untested approaches to try

Escalate When:

• Repeated fix attempts failed — likely architectural; needs human judgment
• Undocumented library behavior — file an issue upstream, work around it
• Environment-specific — requires access you don't have (prod DB, cloud IAM)
• Security-sensitive — don't debug auth/crypto/payment alone; flag for human review
• Multi-team dependency — bug is in another team's service or API contract
• Stalled: if investigation stalls, reassess approach

How to Escalate Well

Don't just say "I'm stuck." Provide: symptom (exact error), **reproduction

steps, evidence gathered (logs, traces, bisect results), hypotheses

tested (including rejected hypotheses and rejection evidence), remaining hypotheses** (untested),

and a recommendation for next steps.

---

Post-Mortem Discipline

After resolving any bug that:

• Was user/customer-impacting
• Took >1 hour to diagnose
• Involved 3+ failed fix attempts (per postmortem-template.md)
• Revealed a systemic issue (same bug class exists elsewhere)

Fill out references/postmortem-template.md and include it in the PR or commit.

The goal is learning, not blame. Every postmortem must produce at least one

action item that prevents the same class of bug from recurring.

---

Modular References

---

Integration with Other Skills

---

Security-Sensitive Bugs

For security-sensitive bugs (auth bypass, data leak, injection), follow the incident response in dev-security/SKILL.md before applying a fix.

---

Compact Summary

When context is limited, preserve: (1) Phase 0 — is it a bug or a design problem?,

(2) Core principle — no fixes without root cause,

(3) phases 0-4 — architecture check → investigate → analyze → hypothesize → implement,

(4) Repeated Failure Rule — after repeated failures, reassess, (5) one variable at a time,

(6) evidence over intuition, (7) failing test first.