Release Engineering — The Pipeline as the Single Source of Truth

In plain terms — The deployment became the authoritative record of what shipped, updating the tracking ticket automatically instead of asking people to reconcile two lists. If that paperwork update failed, the release still continued—a tracking hiccup should not stop the package. Like a UPS scan updating the tracking page instantly, the system supported a shift from releases every two weeks to daily releases with under one percent failure.

This page documents the release-engineering platform Harsh built and operated at Auction.com across roughly seven years (2019→2026): the Jenkins pipeline as the single driver of release state, the two-layer library-and-DSL design that made it reproducible, and the trunk-based daily-release cadence the platform eventually enabled. It is drawn from two bodies of work — the release orchestration that tied pipeline execution to an issue tracker's release lifecycle, and the longer arc of the shared library and job-DSL platform that carried it — with the hard-won friction kept as seasoning.

1. The pipeline as the single source of release truth

There were two sources of truth for "what is being released": the CI/CD pipeline (Jenkins) and the release object in an external issue tracker, whose stages ran Planned → Deploying → Deployed-to-CI → Deployed-to-QA → Released. Humans kept them in sync by hand — error-prone, and release status was never trustworthy at a glance.

The redesign made one authority the writer and the other the mirror: pipeline execution drives release state. A set of stage-driven Jenkinsfiles — one per environment and flow — covers CI, QA, PROD, PROD-DR, on-demand release, mobile release, cleanup, and release-request flows: roughly nine distinct pipelines. A release-steps module in the shared library talks to the tracker's release API (GraphQL) to resolve a pipeline, list releases and their stages, fetch a build, create or advance a release, mark it released, attach issues, and publish release notes.

Pipeline stages map 1:1 to release stages. As a deploy pipeline clears a stage, it advances the corresponding release stage — a successful CI deploy moves the release to "Deployed to CI," and so on up to "Released." Two release modes coexist: a scheduled Standard release and an On-Demand release kicked off by a submit-release-request job and dispatcher.

flowchart LR
  subgraph pipeline["Jenkins pipeline stage · writer"]
    checkout["checkout / build"]
    deployCI["deploy -> CI"]
    deployQA["deploy -> QA"]
    deployPROD["deploy -> PROD"]
  end

  subgraph tracker["tracker release stage · mirror"]
    planned[Planned]
    deployedCI[Deployed-to-CI]
    deployedQA[Deployed-to-QA]
    released[Released]
  end

  checkout --> planned
  deployCI --> deployedCI
  deployQA --> deployedQA
  deployPROD --> released

The rejected alternative — tracker drives pipeline — was considered and turned down: the build is the ground truth of what actually deployed, so it should be the writer. The tracker only reflects it.

2. Lifecycle sync: best-effort, never fails a deploy

Release bookkeeping is important, but it must never block shipping. Every tracker call is wrapped in try/catch, logs a warning, and continues: deploy reliability outranks sync fidelity. A sync failure is a paper cut, not a blocked release. Hard-failing the deploy on a sync error was explicitly rejected as a design.

Three real bugs shaped the integration, and each one left a constraint behind it:

• A lock around release creation. When jobs overlapped, stray "New release" objects appeared after a release was already done — an idempotency gap. The fix put a lock around release creation and routed the final "Released" state through an explicit markReleased transition instead of creating a placeholder.
• Defensive reads against an API shape. The tracker's stages field is a GraphQL connection (read via .nodes), not a flat list — a subtle bug that stalled stage advancement until the query read the connection correctly. Attaching issues to a release hit the same class of bug (a paginated connection read as a flat list).
• Overlapping releases. Multiple releases can be in flight at once (with -01 / -02 / -03 suffixes), so the system has to reason about which pipeline run owns which release object. A verified behavior matrix was maintained for the overlap cases and the specific sync bugs already diagnosed.

Code-level toggles act as operational escape hatches: skip publishing release notes, or skip creating new placeholder releases, for edge cases.

Reusable pattern: syncing a pipeline to an external stateful system needs idempotency + locking + defensive reads — and every external call wrapped so a sync failure never fails the deploy. The external system is a mirror, not a gate.

3. The two-layer platform: how (library) + what (DSL)

Underneath the release orchestration sits a two-layer design that carried the whole org's CI for seven years:

• A shared library of reusable Groovy pipeline "vars" — the steps every service's Jenkinsfile calls. Pipelines for Java, Node, mobile (App Store / Google Play / Firebase), Heroku, MuleSoft, and library-only builds, in single-branch and multi-branch variants. This is where how we build, test, publish, and deploy lives, once, for the whole org.
• A job-DSL repo that declares the Jenkins jobs themselves as code — a seed job generates them — so job configuration is version-controlled and reproducible rather than clicked into the UI.

Reusable pattern: put the "how" (steps) in a shared library and the "what jobs exist" (topology) in DSL. Two axes, two repos, both as code.

4. Seven years visible in the history

The platform's shape changed repeatedly as the org, the cloud, and the tooling changed, and each change is legible in the commit history:

• 2019–2021 — foundation. The shared library was established; early secrets handling moved off a legacy tool onto AWS Secrets Manager.
• 2022–2023 — the Bitbucket-Cloud migration era. Heavy DSL churn tagged "BB Cloud Migration"; pipelines and job definitions adjusted for the new SCM host, plus a large batch of one-off data and migration jobs generated via DSL.
• 2023–2024 — dynamic configuration. "Define branches dynamically for each job type," "configure seed-job as pipeline" — a move from hardcoded job specs toward parameterized, generated ones.
• 2024 — the GitHub migration. DSL and library updated for the move off Bitbucket to GitHub — the second full SCM migration the platform survived (Bitbucket Server → Bitbucket Cloud → GitHub) without being abandoned.
• 2025 — arm64 / Graviton migration + a Java pipeline refactor. The default build agent flipped to arm64, a Java pipeline was refactored, and a hotfix workflow was added that deploys hotfixes straight to QA.
• 2026 — ECR consolidation + release-lifecycle integration. Docker build/publish moved off an external artifact registry onto the cloud-native container registry (ECR), and the release-lifecycle sync described above landed as a series of roughly fifteen commits.

5. Decisions and their tradeoffs

Each major choice was a tradeoff made under real constraints:

• Jobs-as-code via a seed job over UI-configured jobs — reproducibility and review, at the cost of a DSL learning curve and a regeneration step.
• One shared library over per-team copy-paste — consistency and a single place to fix a CVE or add a gate, at the cost of blast radius: a bad library change touches every build. That blast radius is exactly why the defensive patterns below exist.
• Dynamic branch and job definition over static enumeration — scales to many services without hand-editing each job.
• Hotfixes deploy directly to QA — a deliberate workflow carve-out so urgent fixes skip the normal promotion path, guarded by branch-name conditions (which themselves needed several fixes — the history shows the iteration).

6. Friction worth keeping (war stories, as seasoning)

The platform's design was shaped as much by what broke as by what was planned. A few of the harder lessons, kept short:

• The arm64 migration was gated by the least portable tool, not by our own code. The default build agent flipped to arm64 in April 2025, was disabled a week later ("needs more work to execute on arm64") because a security / SAST scan step in the pipeline didn't run on arm64, and only completed roughly eleven months later. The build itself ran fine on arm64 early; one third-party scanner held the whole migration for the better part of a year. Sequence an arch migration around the laggard, and be willing to disable-and-revisit rather than block everything.
• Registry migrations ripple into every build variant and into caching. The move off the external artifact registry onto ECR touched PR, prototype, and release flows. An ECR remote-cache change for preview images landed, was reverted the same day, and re-landed differently — the classic "the optimization broke something, back it out, ship the correct version." Migrate the happy path first; treat caching as a separate, revertible change.
• Shared mutable state on a build agent plus concurrency equals corruption. Concurrent pipeline steps logging into ECR could corrupt the shared AWS credentials file on the agent. The first fix covered some call sites; the real fix was finding and guarding all of them. The bug isn't "one bad login," it's "an unguarded write path you didn't enumerate." Grep for every call site, not the one in the stack trace.
• Release-sync bugs are a class, not a one-off. Stages that won't advance past a checkpoint, stray "New release" objects, issues that don't link — all traced back to idempotency, locking, or defensive-read gaps. Each one reinforced the "best-effort, never fails the deploy" contract.
• Death by a thousand cuts in CI. Stale git refs breaking checkouts (fixed with a prune-stale-branch checkout option) and hotfix / stale-branch auto-cleanup workflows — small individually, collectively the bulk of the operational papercuts.

7. The cadence this platform enabled

The cumulative effect of the platform above is the release cadence it made cheap: a move from bi-weekly to daily releases with a <1% rollback rate, on an in-house Jenkins + GitHub Actions toolchain (replacing Harness, a six-figure-per-year SaaS suite) running 10K+ monthly pipeline executions. GitOps delivery via Argo CD and the trunk-based workflows sit on top of this pipeline layer; this page is the part of the stack that made "a release is a build, and the build is the truth" operationally real.

8. Build vs. buy: migrating off Harness

The in-house platform above replaced Harness, a commercial CI/CD and release-orchestration product Auction.com had been paying roughly $200K a year to license. Harsh worked directly with Harness as the incumbent delivery platform and helped lead the transition away from it.

The interesting part is not the swap; it's the evaluation that preceded it. Running a commercial release-orchestration platform and then rebuilding its capabilities in-house forces you to be explicit about what you were actually buying:

• What the license was really paying for. Pipeline orchestration, release governance, and a managed control plane — each of which had to be accounted for in the replacement, not quietly dropped. The shared-library + job-DSL design in §3 is the answer to "who provides the opinionated structure once the vendor no longer does."
• Where a vendor platform earns its price, and where it stops. A managed product is worth real money when it absorbs operational burden you'd otherwise carry. The calculus changes when your delivery patterns are specific enough that you're configuring around the product as often as you're using it, and when you already operate the underlying compute.
• Continuity is the hard requirement. This was a live, high-volume delivery system: 10K+ pipeline executions a month, with teams shipping continuously throughout. A migration that stopped the line would have cost more than the license. The stage-driven pipeline contract in §1 is what made cutover possible incrementally rather than as a big-bang switch.
• Cost reduction that doesn't quietly bill engineers instead. Removing a $200K line item is only a win if it doesn't reappear as developer friction or reliability regression. The cadence numbers in §7 — bi-weekly to daily at <1% rollback — are the check on that.

To be explicit about the framing: this was a build-vs-buy decision driven by Auction.com's economics and platform requirements, not a verdict on Harness as a product. Plenty of orgs are correct to buy exactly this category of tool.

What this does and doesn't demonstrate. It shows hands-on familiarity with a commercial CI/CD platform, a build-vs-buy evaluation carried through to execution, migration planning for a high-volume production delivery system, and leading engineering teams from a vendor platform onto an internally operated one. It does not make Harsh a Harness administrator or a specialist in its feature surface — he does not claim a specific tenure with the product, nor depth in Harness Services, Connectors, Templates, or Triggers. "Helped lead" is the accurate verb; this was not a solo effort.

9. Honest gaps

Impact numbers are not in the source material: how many services the shared library serves, builds per day, mean pipeline duration over time, and any before/after "releases mislabeled per month" or "manual minutes saved" figure. A headline version of this story ("maintaining CI for a large org for seven years") would need those pulled from Jenkins. They are not in the material this page was distilled from, and nothing here invents them.

Sources

This page was distilled from Harsh's commit history and session logs across the Jenkins shared library, the job-DSL repo, and the release-orchestration work — cross-checked against the in-repo PRD and resume. No internal hostnames, credential identifiers, ticket IDs, or proprietary source code appear; the pipelines and library are described, not copied. Every architectural claim above traces to that material.