Skip to content
Nyl
OrchestrationRoadmapIn development

Roadmap

Orchestration grows in milestones, each with an exit criterion that says when it is done. This page tracks them. The direction explains where orchestration is going and why, and the design contracts specify each part. Everything not marked done is planned and does not exist in Nyl yet.

Orchestration39 of 69 acceptance items done

Next step

Section titled “Next step”

Milestones

Section titled “Milestones”

A milestone is in progress once one of its items is done, and done only when its exit criterion is met. The line between milestones fills as the milestone above it progresses.

  1. M1

    Unit, input, state, and promotion contract

    Done

    The written contract for units, inputs, state, and promotion, proven by walkthrough scenarios.

    6 acceptance items
    • Done: Define Release input declarations, DeploymentTarget bindings, types, and lock semantics for fromGit (contract).
    • Done: Define Unit, Environment, and PromotionPath schemas; typed references; output admission; and artifact descriptors (core, promotion).
    • Done: Define desired/observed ref layout, receipt freshness, identity fences, leases, deletion, and recovery invariants.
    • Done: Define the driver trait, capabilities, and the command unit's process contract (core, infrastructure units).
    • Done: Specify CLI effects, revision selection, partial plans, and exit categories.
    • Done: Write the exit-criterion walkthroughs as scenario files for the M3 harness.

    Done when: The contract explains a successful dependency wave, an unavailable upstream output, effects without a receipt, competing runners, a promotion with stale source evidence, and deletion, without hidden state or ambiguous command semantics. Met by the walkthroughs and their scenarios.

    6 of 6 done
  2. M2

    Release inputs without orchestration

    Done

    Typed Release inputs and DeploymentTarget bindings, without any orchestration.

    11 acceptance items
    • Done: Implement Release inputs and DeploymentTarget bindings for value, fromFile, fromGit, and fromPublication, with type validation and defaults.
    • Done: Support carryFileFromWorktree for fromPublication, excluding carried files from the dirty check.
    • Done: Extend nyl update source-locks to refresh fromGit locks, with a --target filter.
    • Done: Apply target bindings in direct commands and add --input/--inputs; a target that selects no group containing the Release fails unless --application-group or --defaults-only is given.
    • Done: Require unique generated Argo CD names across every pair of targets whose instances resolve to the same cluster and namespace, including implicit per-target instances.
    • Done: Record resolved inputs in the dependency recorder, render-cache key, and the existing ownership-index inputs map as digests under reserved @-prefixed keys, keeping index format version 2.
    • Done: Reject fromUnit and fromPromotion bindings outside orchestration with an actionable message.
    • Done: Move binding declarations, validation, and resolution into one generic resolver in nyl_core::bindings: typed declarations, exactly one source per binding, JSON Pointer selection, effective-value precedence, digests and provenance, and aggregated issues, with each binding kind served by a provider trait. Release input sources become providers in nyl-render, so M3's unit values and variables, PromotionPath selectors, and nyl get reuse the same resolver instead of growing their own: it resolves declared slots and references at any depth of a document, records provenance per JSON Pointer, and lists blocked references by location.
    • Done: Document the feature and regenerate resource schemas.
    • Done: Expand template values (${ … }) in fields rendered later than the structural pass, starting with applicationNameTemplate, which requires at least one expression and no longer expands the {% raw %} form.
    • Done: Extract nyl-core and nyl-render from the current crate without behavior change, per the implementation architecture. The nyl package keeps the CLI role, and no crate is published to crates.io.

    Done when: A target renders Releases from static, locked external, and same-branch publication state inputs, committed or carried, through render-tree and publish-tree; a concurrent state push makes publication fail rather than interleave; projects without inputs that the previous release rendered produce byte-identical output, except where the two listed validation changes now reject them: a {% raw %} applicationNameTemplate, and duplicate generated names across implicit per-target Argo CD instances.

    11 of 11 doneNeedsM1
  3. M3

    Orchestration core with a constrained command unit

    Done

    Environments, state in Git, and dependency-ordered runs of command units.

    14 acceptance items
    • Done: Implement environments, YAML state files with published schemas, nyl state init/move/forget (including moves between branches and custom refs), leases and run checkpoints under a configurable coordination ref prefix, validated against GitHub, GitLab, and Forgejo for custom refs and mixed atomic pushes, and one transition commit per operation with a machine-readable summary, pushed atomically with the lease check.
    • Done: Implement nyl state delete, including decommissioning declared environments.
    • Done: Support optional dedicated SSH keys and HTTPS tokens, configured only when needed, for state pushes and publish-tree. Remote Git operations run the git command line through nyl-git, so by default they authenticate as the user's git does (SSH config and keys, agent, credential helpers); [[git.credentials]] in nyl.toml names a key file, or a key or token in an environment variable, per repository URL prefix, for runners without those. State pushes use the same transport, including atomic multi-ref pushes with per-ref leases; libgit2 handles only local repository access.
    • Done: Implement plan, reconcile, status, verify, recover, teardown (including --hold), hold, and resume for a dependency graph of command units with fromUnit references, including --local runs next to remote state on the clone's own state branches, state without a remote, and state move --from local when a remote is added.
    • Done: Implement cross-environment references and multi-environment runs by repeated -e, a label selector, or an EnvironmentGroup with shared defaults, ordered by those references, each environment with its own lease and transition commit.
    • Done: Implement approvals bound to the desired document, with recorded approval sources and identity (verified, asserted, local), the GitHub approver lookup and the [approval] lookup hook, and the common env credential admission.
    • Done: Implement pinned source worktrees, reachability checks against protected refs, and content-based execution keys.
    • Done: Prove stale-evidence blocking, competing runners, and recovery after an effect succeeds but receipt publication fails.
    • Done: Move the business rules the nyl package still holds into the crates that own them before orchestration builds on them: ownership-index input hashing (hash_inputs), the worktree source state with its carry exclusion (source_state), and publication-base verification (verify_publication_base). The command line then only parses arguments, wires crates, and renders reports.
    • Done: Replace the remote fields of ApplicationGroupSource and fromGit's GitInputSource with the one GitSource type of the implementation architecture, which command-unit sources then use too. Both already share one commit-lock rule.
    • Done: Start nyl-state, nyl-orchestration, and nyl-drivers as separate crates with the scenario harness, property tests, and crash injection.
    • Done: Expose a pure, versioned graph of literal unit addresses and dependency edges before upstream values exist, shared by local execution and future CI exporters. Prove separate unit-scoped invocations through the unit-jobs walkthrough without changing environment-wide lease or ownership semantics.
    • Done: Add the test-clock Cargo feature and the test-only fake kinds.
    • Done: Add the two-tier harness for the reference scenarios: in-process with fakes, and through the real binary under the tier2 Cargo feature.

    Done when: Two dependent command units reconcile locally and in CI with identical results; repeat execution is a no-op; interruption has a demonstrated recovery path; the M3 part of the reference scenarios and every walkthrough scenario marked M3 pass.

    14 of 14 doneNeedsM1
  4. M4

    Container image and Terraform units

    Done

    Container image builds and Terraform or OpenTofu configurations as units.

    8 acceptance items
    • Done: Add the OciImage kind over docker buildx, recording digest references and a ContainerImage artifact, with the registryAuth helper and named build contexts.
    • Done: Add nyl build for kinds with the build capability, starting with OciImage: builds from the current worktree, forwards driver options after --, and writes no state.
    • Done: Support nyl reconcile --worktree for one environment, using an immutable working-file snapshot throughout preflight and execution, retained in state and run history for fresh-clone recovery and teardown.
    • Done: Add the Terraform and OpenTofu kinds over one implementation, with native state and locking, declared output admission, change digests, plan approval (bind: plan), verification, and teardown.
    • Done: Support a local-module Terraform configuration pinned to a source commit and path.
    • Done: Demonstrate Terraform-to-Terraform output references.
    • Done: Check out only what a unit reads when its kind declares its inputs: OciImage and Terraform/OpenTofu units get a sparse worktree of their source paths and the nyl.toml directory, so their cost in a large monorepo follows the unit, not the repository. A Command unit can read anything in the repository, so it keeps a full worktree unless it declares the paths it reads.
    • Done: Add the exec driver capability and nyl exec for held units, with the deployed or --worktree spec and source snapshot, ownership-scope checks, recorded spec differences, approval of the first execution afterwards, and refusal exit 125; implement it for OpenTofu and Terraform, with their missing-state guard.

    Done when: A network configuration's outputs feed a dependent configuration; an image build records a digest; unchanged inputs plan no change; the M4 part of the reference scenarios passes with the real tools in the required tier2 CI job.

    8 of 8 doneNeedsM3
  5. M5

    Images and Terraform outputs into Kubernetes releases

    Planned

    Kubernetes publications that consume image digests and Terraform outputs.

    8 acceptance items
    • Not done: Add the Kubernetes publication unit over render-tree/publish-tree.
    • Not done: Resolve fromUnit Release input bindings into an explicit pinned input snapshot for rendering.
    • Not done: Distinguish published from attested evidence and named attestations for dependents.
    • Not done: Add the publication unit's observe mode: read generated Argo CD Applications' last successful sync and health, match them to publication commits by Application directory tree, record the observations, and attest accepted and healthy.
    • Not done: Document an end-to-end local/CI example.
    • Not done: Declare the build capability for KubernetesPublication: nyl build renders the tree, --diff compares it with the published tree, and --push publishes only targets no environment owns.
    • Not done: Support inline DeploymentTargets on KubernetesPublication, its two-phase teardown with the teardownWait strategies, ownership-index owner fencing, and the teardown readiness check with its warnings.
    • Not done: Add EnvironmentTemplates for preview environments: instances through nyl state init --template, teardown --all, state delete, shared state refs through state.path, cross-environment references to declared environments, sliding expiry, and maxInstances.

    Done when: One reconcile builds an image, applies Terraform, and publishes Kubernetes manifests consuming both; the same Releases still render with static inputs in a target that does not use orchestration; all three reference scenarios, including preview closure and expiry, pass in both tiers.

    0 of 8 doneNeedsM2M4
  6. M6

    Promotion paths

    Planned

    Explicit promotion of proven source commits and values between environments.

    10 acceptance items
    • Not done: Implement PromotionPath, PromotionRecord, and nyl promote with evidence checks and an optional pull-request change gate.
    • Not done: Promote dev's source commit and image digest to prod from one recorded dev state, with record: state and record: source, the whole-source evidence rule, recorded evidence for older states, artifact verification, and rollback through --revision.
    • Not done: Support several paths into one environment: source.fromPromotion.paths, path-less bindings, the per-value supersede check with --supersede, and the hotfix pattern with a fifth reference scenario.
    • Not done: Add nyl get states, nyl get promotion-candidates, and nyl promote --dry-run.
    • Not done: Promote values only, such as an image digest and a Terraform source commit, across differently named units and inputs, each from the state its consumer runs.
    • Not done: Promote from a non-orchestrated target's published inputs, including a carried state file, with values verified against the recorded digests.
    • Not done: Promote into a target outside every environment: to: {target}, the record in the target's spec.promotions, fromPromotion resolution in tree and direct commands, and the ownership index's applied sequence.
    • Not done: Implement attestations: driver-declared names, spec.attestations on units and environments, nyl attest for environments and targets, the published | attested evidence levels with a path's per-unit attestations, blocking on failures with --ignore-attestation, and maxAttestationAge.
    • Not done: Gate promotion on fresh recorded attestations for both PromotionPath sources, sourcing each value from the publication its Application runs.
    • Not done: Show promotion lineage in status.

    Done when: Prod runs exactly the source commit and image digest dev proved, with auditable lineage; stale, missing, or mixed source evidence blocks promotion; a hotfix reaches prod through its own path, and a dev promotion without the fix is refused until the fix is merged or superseded; a value-only path promotes each value from what its consuming Application runs.

    0 of 10 doneNeedsM5
  7. M7

    Continuous operation and scope decision

    Planned

    Evaluate continuous operation and decide what comes after the initial scope.

    6 acceptance items
    • Not done: Evaluate a scheduled or long-running runner and its observation cadence using M3–M6 evidence. The drift-repair policy is settled in M3 part 5: converging units repair recorded drift automatically by default.
    • Not done: Add an opt-in nyl reconcile --verify: the run verifies its current units under its own lease, then repairs what drifted by driftRepair, all in one transition commit, so a scheduled job needs one command and no lease gap between nyl verify and nyl reconcile. It reuses the verify path; without the flag, reconcile does not verify.
    • Not done: Decide which command-unit uses warrant typed drivers, and whether to add the plugin driver protocol.
    • Not done: Support minimum healthy durations and flapping guards over recorded observations.
    • Not done: Evaluate application-level checks as promotion evidence.
    • Not done: Record the selected direction and constraints in ROADMAP.md.
    0 of 6 doneNeedsM6
  8. M8

    Provider-neutral CI execution and first GitLab adapter

    Planned

    Each unit as its own CI job, starting with GitLab, with Nyl keeping the decisions.

    6 acceptance items
    • Not done: Implement nyl ci export gitlab for one environment, with reconcile and read-only plan operations, consuming M3's graph and typed reports. Keep CI runner configuration separate from unit semantics.
    • Not done: Specify provider-neutral graph/manifest types, capability negotiation, normalized diagnostics, and a conformance suite shared by GitLab and an in-memory test adapter. Provider IDs and scheduler syntax never enter unit identity, receipt formats, or the execution key. Unsupported capabilities fail explicitly or use a documented safe fallback.
    • Not done: Generate stable, collision-checked job IDs and needs edges, a pinned public graph manifest, per-unit reports, and an always-run read-only summary. Include current producers and unresolved consumers, and a lifecycle node: reconcile --leaving after the unit jobs for reconciliation, and a read-only lifecycle report for planning, including empty selections, preserving the caller's --fail-on-leaving policy without state writes.
    • Not done: Bind jobs to the admitted source, project, environment, operation, and adapter configuration. Implement opt-in deployment admission: a versioned record on state.admissionRef with forward-only generations chosen by the environment's source rules, idempotent re-dispatch, authorized rollback, and normalized authorization evidence. Under the environment lease, superseded worker attempts skip before intent writes, imports, or effects; state restore carries the record forward. Provider pipeline serialization is an optional optimization, not the fence.
    • Not done: Preserve waiting/failure/uncertainty categories and receipt-based retries; revalidate approvals and never pass authoritative outputs through CI artifacts.
    • Not done: Cover the CI acceptance scenarios with fake drivers and local Git repositories, plus a GitLab configuration and integration check. Demonstrate the generated jobs with M4's real drivers.

    Done when: A GitLab child pipeline displays separate producer and consumer jobs, executes at a pinned source, explains no-ops and blockers, and preserves the local core's freshness, lease, approval, and recovery rules across retries and cancellation. Independent unit jobs are observably serialized within one environment; no distributed-run guarantee is made. Retrying an old worker after a newer admission has no deployment effects, and a newly authorized rollback remains possible. Removing a unit shows a lifecycle job that destroys nothing without --allow-teardown, and removing all units still produces a read-only MR lifecycle report. The in-memory adapter proves the same semantic graph and admission behavior without GitLab types or services.

    0 of 6 doneNeedsM3M4
  9. M9

    Domain modeling for units and Releases

    Planned

    Team-owned, typed declarations that units implement and Releases read, the fourth layer of Nyl.

    Open the M9 roadmap, which tracks its 11 slices.