Pipelines
Mozaic reads pipeline configuration from the repository revision being built, validates a closed YAML schema, records a SHA-256 plan identity, and schedules bounded jobs against an immutable source bundle.
Commit the build definition with the code.
Prerequisites
- An
ActiveGit repository in a Mozaic project. - For execution, an enabled runner control plane and an eligible runner. Registration management and source inspection remain available when execution is disabled.
- An explicitly registered UTF-8
.ymlor.yamlfile in the revision you want to run. Its repository-relative path can be anywhere in the repository. - Container images pinned to complete lowercase SHA-256 digests. Mutable tags such as
:latestare rejected. - The claim for a manual run. Pull-request authors can run the current check for their own proposal; other users need Run pipelines.
- Enough concurrent-job capacity and service credits for the resolved resource request.
Open Project › Pipelines, or use the Pipelines tab in a repository, to manage registered pipelines separately from run history. The repository tab keeps its navigation and limits loaded run history to that repository; Project Pipelines combines project history. Select a reference, then click a pipeline name to inspect its file at that exact revision. Use Load older runs to continue through history, including pages with no matching repository runs. A committed YAML file is not automatically a pipeline: only files you explicitly add are eligible for scheduling.
Add an existing pipeline file
Commit and push the YAML file with your normal source workflow. Choose Add pipeline, select its repository, enter a display name and its repository-relative path, then save the registration. Registration requires Manage source. It does not create a source commit, run the pipeline, or activate other YAML files. Registration management remains available while execution is disabled.
Choose only the pipelines you want
You can register .mozaic/build.yml and .mozaic/deploy.yml while leaving .mozaic/temporary.yml inactive. A file such as random/folder/whatever.yml is equally valid. Use safe relative paths with forward slashes; absolute paths, traversal components, and backslashes are rejected. A repository can have up to 20 active registrations, with at most 100 across a project and one registration per repository path.
Each registered file declares its own triggers and jobs. Repository events use the declared triggers; authorized users can manually run any registered definition. The manual trigger remains valid for existing definitions. Remove a registration to stop future selection and automatic scheduling; existing run history and the repository file are retained. Previously used .mozaic/pipeline.yml files must also be explicitly registered. No historical events are replayed when a file is added.
Automation can remove a registration with DELETE /api/v1/organizations/{organization_id}/repositories/{repository_id}/pipelines/{pipeline_id} using a tenant PAT with Provision source. The issuer must still hold source-management authority in the organization and the repository's project; the registration must belong to that exact repository. Success returns 204 and retains audit evidence. Existing runs, resolved plans, attempts and artifacts remain unchanged.
When replacing a pilot pipeline, wait for its run to become terminal, remove its old registration, register the full replacement definition, and verify the catalog before the next push. Multiple active definitions or both enabled push and trusted_branch triggers can create separate runs and duplicate cost.
Pipeline configuration is code. Editing the file on the default branch does not alter a run for an older revision, and a manual caller cannot substitute different YAML in the scheduling request.
Monitor runs from automation
An organization API token with the separately selected Authorize pipeline triggers delegation can read GET /api/v1/organizations/{organization_id}/projects/{project_id}/pipelines and its /{run_id} detail on the matching organization host. The issuer must still be able to view that project and hold current pipeline-execution authority for it. Expiry, revocation, membership and role changes apply to every read.
History uses bounded pages and an opaque cursor tied to the same token, issuer, organization and project. Run detail includes the exact revision, canonical plan, attempts and normalized result metadata. Authorized readers can also inspect the run's target-reported deployment receipts through the separate run API. This read delegation does not grant log streams, artifact downloads, cancellation, manual scheduling or deployment execution or reporting authority. Keep the bearer in a private credential helper; never extract browser cookies to monitor a run.
Start with the shipped .NET sample.
This valid schema-version-1 definition is derived from samples/dotnet/.mozaic/pipeline.yml. The complete image digest is intentional. Adapt the solution, project, cache key, and artifact paths to your repository. Authorized users can run any registered definition manually. Commit the dependency lock files required by locked restore, and ensure any SDK selection in global.json matches the pinned image.
version: 1
triggers: [pull_request, manual, tag]
jobs:
verify:
image: mcr.microsoft.com/dotnet/sdk@sha256:0d99ebf1ccad15037b5c98c9fe8979d6b8dbac23812feae2277a0b277b5c5ffa
resources:
cpu_millis: 2000
memory_mib: 4096
scratch_mib: 10240
pids: 512
log_mib: 100
network:
mode: http_https
allow_hosts: [api.nuget.org]
timeout_minutes: 30
steps:
- name: Restore locked dependencies
run: HOME=/workspace/.home DOTNET_CLI_HOME=/workspace/.dotnet NUGET_PACKAGES=/workspace/.nuget/packages dotnet restore Mozaic.Sample.slnx --locked-mode
- name: Build release binaries
run: HOME=/workspace/.home DOTNET_CLI_HOME=/workspace/.dotnet NUGET_PACKAGES=/workspace/.nuget/packages dotnet build Mozaic.Sample.slnx --configuration Release --no-restore
- name: Run tests
run: HOME=/workspace/.home DOTNET_CLI_HOME=/workspace/.dotnet NUGET_PACKAGES=/workspace/.nuget/packages dotnet test Mozaic.Sample.slnx --configuration Release --no-build --no-restore --logger "trx;LogFileName=tests.trx" --results-directory artifacts/test-results
- name: Pack NuGet artifacts
run: HOME=/workspace/.home DOTNET_CLI_HOME=/workspace/.dotnet NUGET_PACKAGES=/workspace/.nuget/packages dotnet pack src/Mozaic.Sample/Mozaic.Sample.csproj --configuration Release --no-build --no-restore --output artifacts/packages
artifacts:
- name: nuget-packages
paths: [artifacts/packages/*.nupkg, artifacts/packages/*.snupkg]
caches:
- key: dotnet10-lock-v1
paths: [.nuget/packages]
test_results:
- path: artifacts/test-results/tests.trx
format: trx
- Add the file.Create the
.mozaicdirectory at repository root and commitpipeline.yml. - Adapt paths.Every artifact, cache, test-result, and publication path is workspace-root relative. Absolute paths, backslashes, and
.or..components are rejected. - Keep restore deterministic.The sample uses locked restore, then builds and tests without restoring again.
- Push and register the file.Use Add pipeline to register its path, then schedule a manual run or send a new enabled repository event. Mozaic compiles the exact committed bytes of the registered file.
Compose a bounded job graph.
version- Must be
1. Unknown fields are rejected rather than ignored. triggers- One or more of
push,pull_request,manual,trusted_branch, andtag. Duplicates are normalized. jobs- A map of job IDs to job definitions. IDs contain at most 63 ASCII letters, digits, hyphens, or underscores and begin with a letter or digit.
needs- Job IDs that must succeed first. Unknown, duplicate, self, and cyclic dependencies are rejected. Jobs without dependencies can run concurrently when capacity permits. Dependencies order execution; they do not transfer files or build outputs.
image- A container image pinned with
@sha256:followed by 64 lowercase hexadecimal characters. services- Up to three named digest-pinned service images per job, with an optional bounded command array.
steps: run- A non-empty shell command of at most 16 KiB, with an optional display name.
steps: uses- An installation-allowlisted, digest-pinned
oci://action and bounded string inputs. Arbitrary actions are not fetched. network- Defaults to
none. Usehttp_httpsplus explicitallow_hostspublic DNS names for HTTP(S). Optionalallow_connectentries pair one public DNS host or canonical public IP literal with one port for proxy CONNECT; private and reserved addresses remain denied. artifacts- Named path sets retained as downloadable run artifacts.
caches- Keyed path sets restored only from suitable immutable successful attempts with matching trust.
test_results- Currently supports normalized Visual Studio TRX files with
format: trx. deployment- Optional nonsecret intent metadata: an enrolled
target_id, anenvironmentkey, a transitive prerequisiterelease_job, and 1-8 exact OCIimage_repositories. It participates in the plan hash and grants no deployment authority. See target reporting.
Workspaces, shells, and .NET settings
Each job starts from the exact source revision in its own workspace at /workspace. Ordered steps in that job share workspace files, but execute in separate containers. A dependent job must restore and build its own inputs: needs does not copy a previous job's artifacts. For example, let independent unit_tests and smoke_test jobs verify the code, then give the packaging job needs: [unit_tests, smoke_test] and its own restore, build, and pack steps.
Each run uses /bin/sh with set -eu, beginning at workspace root. Use POSIX shell syntax, or explicitly invoke another shell available in the image. Files under /workspace persist between steps; shell variables, export, cd, and files elsewhere do not. A failed command stops its step and prevents later execution steps from running.
The job runs as a non-root user with a read-only container filesystem. Point .NET's writable state into the workspace: the example sets HOME, DOTNET_CLI_HOME, and NUGET_PACKAGES on every command. For longer scripts, commit a shell setup file and source it at the start of every step that needs those variables. Declaring a cache path alone does not configure NuGet to use it. Restore must also succeed with an empty cache; change cache keys when SDK or dependency locks change.
There are no YAML env, if, matrix, job-output, or per-job trigger fields. All jobs belong to every enabled trigger. The public variable MOZAIC_PIPELINE_TRUSTED is true or false. Jobs also receive MOZAIC_ATTEMPT_ID, MOZAIC_REPOSITORY_ID, and the exact opaque revision value in MOZAIC_REVISION. Branch, tag and run-number variables are not injected. Keep package versions in committed project properties or a version file.
${{ ... }} expressions are not supported. YAML anchors, aliases, merge keys, custom tags, multiple documents, tab indentation, and excessively deep indentation are also rejected.
Use protected files only in an explicitly authorized ordinary job.
Job names are arbitrary. An ordinary downstream run command can invoke a target through SSH and wait for its exit status. On an operator-configured runner, secret_files: {ssh_key: target_invoke} requests a named private file at /run/mozaic/secrets/ssh_key. The value in YAML is a reference, never a credential. Trusted execution alone does not grant that reference: an operator must provision a stable policy for the exact organization, repository, authored job key and allowed secret references on that runner. Only server-authenticated Trusted jobs from governed trusted branches or tags may use it. Each execution keeps its immutable revision, resolved plan and scoped lease. Missing or revoked authorization fails closed. This bounded facility has no general secrets-management UI.
Protected-file jobs support ordinary run steps without service containers, artifacts, caches, test-result exports or publication actions. Their command output is withheld and their writable workspace is never exported; normal exit status still determines success or failure. Put build, tests and publication in a prerequisite job. Only the named read-only file is delivered, with owner-only permissions, and its temporary engine copies are removed. Treat code that receives a credential as trusted: it can deliberately send that credential to an allowed destination.
any-job-name:
image: <approved-tools-image-pinned-by-digest>
needs: [make_artifacts]
secret_files: {ssh_key: target_invoke}
network:
mode: http_https
allow_connect: [{host: deploy.example.com, port: 22}]
steps:
- run: sh ci/deploy.sh --target helsinki --manifest deploy/production.json --release-job make_artifacts --attempt "$MOZAIC_ATTEMPT_ID" --wait
The versioned command can use the supplied SSH CONNECT helper through HTTP_PROXY, a dedicated invoke-only key and pinned known-host keys. Its manifest selects the exact host and port; direct networking remains unavailable. The target keeps its own evidence/runtime credential locally. Never place that credential or a workstation root key in the job.
The source example at samples/ordinary-ssh-deployment/ includes a stateless HTTP application and a complete test → native OCI build/publication → ordinary SSH deployment pipeline. Separate operator provisioning installs the target command, private registry/evidence credentials, isolated account, pinned host key, ingress and recovery state before jobs are enabled. The example manifest reads /run/mozaic/secrets/ssh_key. Its sample hosts, UUIDs and synthetic host-key pin must be replaced with authenticated, reviewed inputs. The same scoped policy applies to later Trusted revisions without a new per-revision grant. The target still checks the protected source ref and exact successful upstream publications for the active attempt. Obsolete policies containing revision or plan_hash fail closed; an authorized operator must migrate them during a drained runner upgrade, preserving scope, allowed references, private backups and file ownership/mode, and replacing the policy atomically. This facility is not a general pipeline secret vault.
The downstream command waits for the target's running-digest, local health and public HTTPS checks and fails the ordinary job on a rejected release, disconnect, timeout or failed readiness. Protected command output remains withheld. An enrolled reporter can retain typed deployment observations through the API described below; authorized readers inspect those receipts in the run's Deployment evidence section. OCI publication receipts, ordinary job exit status and target-reported outcomes remain separate evidence. Preserve target-local audit and recovery evidence. The stateless example has no application database backup or migration path; use the separately reviewed stateful executor for durable application data. A complete source example does not establish installed-runner or live-deployment acceptance.
While the downstream job is actively leased, the target can request exact upstream receipts with GET /api/v1/organizations/{organization_id}/deployment-runtime-secrets/{target}/attempt-evidence?requesting_attempt_id=...&release_job=make_artifacts&image_repositories=.... The target credential fixes the repository and source ref. Evidence requires successful declared ancestors in the current execution generation and actual publication receipts from the selected release job. It excludes the active downstream job and unrelated jobs. Missing, pending or stale evidence returns unavailable; the command waits within a bound and fails on timeout. Existing latest-release polling endpoints still require a fully successful pipeline. Neither path substitutes a moving image tag for an exact published digest.
For a retained failed activation, the separate machine-only recovery-evidence endpoint revalidates its exact successful upstream publications in the current execution generation. It reports the failed pipeline status and grants no retry authority. Recovery requires a supported operator procedure with fresh local authority bound to the pending release, backup generation and current evidence. Ordinary exact activation still requires an active leased attempt; latest-release polling still requires a successful pipeline.
The generic stateless helper instead uses the target's private organization-scoped evidence reader with GET /api/v1/organizations/{organization_id}/deployment-evidence:attempt, bound to the exact requesting attempt, repository, protected ref, upstream job and image path. Both paths consume authenticated immutable publication evidence; neither grants target execution authority merely because a pipeline succeeded.
Bind target observations to an exact deployment attempt.
The source supports bounded target enrollment through the API and target-reported deployment receipts in the pipeline run UI. An ordinary SSH job keeps its authored command and arbitrary job name. Add matching intent metadata to that job after an owner has enrolled the target:
needs: [make_artifacts]
deployment:
target_id: 11111111-1111-4111-8111-111111111111
environment: production
release_job: make_artifacts
image_repositories: [applications/example/web]
Replace the example UUID and image path with the enrolled binding. The release job must be a direct or transitive prerequisite that publishes the exact image paths. Metadata changes the canonical plan hash retained in execution evidence. Protected-file access uses the existing stable organization, repository, job-key and secret-reference policy for Trusted execution; the target still verifies its protected ref and exact attempt evidence. Omitting metadata preserves existing definitions. It does not provision a host, select a new executor or grant production access.
- Enroll the exact target.An organization owner uses the matching tenant HTTPS origin, a cookie session, fresh MFA, CSRF and an
Idempotency-KeywithPOST /api/v1/organizations/{organization_id}/deployment-targets. The immutable binding names the environment, repository, exact protected source ref, upstream publishing job, 1-8 image paths and 1-8 named health checks. A different scope requires a new target. - Issue a separate reporting credential.Use
POST /api/v1/organizations/{organization_id}/deployment-targets/{target_id}/reporting-credentialswith a future UTCexpires_atand the same management protections. At most eight active credentials exist per target; each expires within 365 days. Install the returned token in a private target-local file. Runtime-secret and renewal credentials do not grant reporting authority, and the job receives no report token. - Require admission before host changes.The target durably journals an operation UUID and requesting attempt UUID, then posts them to
.../deployment-targets/{target_id}/operationsusing its dedicatedAuthorization: Bearer mozaic_report_...credential. Cookies, PATs and mixed authentication are rejected on machine writes. The server derives the exact run, attempt lease, execution generation, revision, plan and coherent immutable upstream publications. The target compares this frozen intent with its verified evidence and local policy before preparing or replacing containers. - Journal observations before delivery.Post bounded typed events to
.../operations/{operation_id}/eventswith a durable event UUID and previousexpected_sequence. The original admission credential owns that operation; another credential for the target cannot append or replay it. Exact payload replay returns retained evidence without repeating a host mutation. Success requires every intended digest and every enrolled health check to match. - Read the run's receipts.Open Project › Pipelines › run › Deployment. The run keeps execution, package publication and target-reported deployment outcomes separate. Receipts show the frozen target/environment, exact requesting attempt and execution generation, repository/ref, full revision and plan hash, producing attempts, publication IDs and immutable component digests. The typed chronology preserves sequence, UTC receipt time, stage, reason, and bounded component and health observations. Next receipts page follows a bounded continuation; the count describes only the returned page. Refresh checks current access again. Current authorized project pipeline readers can also use
GET /api/v1/organizations/{organization_id}/projects/{project_id}/pipelines/{run_id}/deployments?limit=50and the returned UUID cursor. A tenant cookie or currently authorized pipeline-monitoring PAT can read this bounded history. It includes frozen intent and typed events across retained attempts, withevidence_kind: target_reported; it contains no tokens, host addresses or runtime values. Target enrollment and credential management remain separate API operations; this reader does not issue credentials or grant target execution authority.
Stages distinguish preparation, application, verification, success, failure, recovery, failed recovery and interruption. Recovered means the target reported recovery after a deployment failure; it does not establish that the intended release is installed. Interrupted is an explicit terminal target statement. A run with no receipt shows an unavailable deployment outcome and no manufactured intent or chronology, even when its job and publication succeeded. Checking access, denied access and temporary errors hide previously displayed receipt details. There are at most 32 events per operation. A terminal statement is immutable. An unfinished operation becomes unknown after lease loss, cancellation, supersession or loss of its original credential, issuer, source or placement authority. That status makes no claim about the running container or rollback. New exact attempts can supersede unknown history; an alternate credential cannot take over the old operation. Owners can revoke a reporting credential or permanently revoke the target through their respective /revoke management endpoints with fresh MFA, CSRF and idempotency.
Configured reporting fails closed on admission before new host preparation or replacement. Pending target-local recovery must remain available during an API outage or credential expiry or revocation; retain its private journal when a recovery observation cannot be accepted centrally. A receipt authenticates the enrolled target's statement, not independent installation truth. Neither SSH exit status nor successful OCI publication proves deployment success. The reviewed sample and operator contract in deploy/deployment-reporting.md cover private credential installation and durable replay. Installed-helper, SSH/Podman, health and recovery acceptance remain separate operational gates. Full deployment environments, approvals, preview allocation and a general secret vault are outside this capability.
Enable only the entry points you use.
| Trigger | When it is recorded | Run trust |
|---|---|---|
pull_request | Opening, synchronizing, or explicitly verifying a current pull-request snapshot. | external_restricted |
manual | A user with Run pipelines selects an exact repository reference in the Pipelines page. | external_restricted |
push | An accepted branch update whose actor has Manage source and Run pipelines. | external_restricted |
trusted_branch | An accepted update to the repository's configured default branch under server-side authorization. | trusted |
tag | An accepted tag update whose actor also has Manage packages. | trusted |
Automatic repository triggers require both a matching YAML declaration and the corresponding accepted product event and current governance claims. Manual scheduling requires the project-scoped Run pipelines claim, independently of YAML triggers. Client input cannot mark a manual or pull-request job trusted.
An explicitly configured repository can accept governed fast-forward default-branch pushes without a separate pull-request approval and build and test each accepted revision through its registered trusted-branch definition. Default-branch history and deletion protection remain enforced. For a repository that releases from master, first select its existing refs/heads/master branch through the default-branch API; a push to master then qualifies for trusted_branch. Other branches retain restricted push semantics. Deployment still requires exact successful trusted upstream output and separate target authorization. Legacy latest-release polling waits for a fully successful run; an actively leased downstream ordinary job can instead request its declared upstream publication receipts. Importing a repository or manually running its pipeline does not supply a trusted native push event.
If the same default-branch update enables both push and trusted_branch, each enabled trigger is recorded independently and can produce its own run.
Schedule an exact repository reference.
- Select an Active repository.Archived and provisioning repositories are not offered for manual scheduling.
- Select a branch or tag.The picker shows the full reference and the exact resolved revision.
- Select a pipeline and schedule it.Choose any valid registered definition. Manual scheduling requires the project-scoped Run pipelines claim, regardless of the definition's triggers. The server reloads that exact file from the selected revision, freezes registry slugs to immutable IDs, and hashes the canonical plan with its definition identity.
- Open the receipt.A successful response shows the plan SHA-256, source-bundle SHA-256, run ID, and immutable attempt IDs.
Success state
The new run appears as Queued in project Pipelines with trigger Manual, the selected ref and revision, and trust external_restricted. Repeating the same in-flight browser command is idempotent.
Manual runs can publish their declared container and package registry outputs. In addition to Run pipelines, you need the required package-management permissions for the publication targets. Missing publication permission returns an authorization error when scheduling. Mozaic binds publication to the exact run and declared outputs, then checks current permissions and target ownership again before publishing. The run keeps its manual provenance and restricted execution policies for secrets, connections and caches. Use a runner version that supports authorized manual publication; older versions remain available for ordinary jobs.
Browse runs across the organization.
Open Pipelines from the organization navigation to discover runs across currently authorized projects. Search matches literal run, project, repository, definition, source-ref, revision, and trigger-label text. Project, trigger, and current-status filters run on the server before cursor pagination. The browser reader requires a signed-in organization member; it does not add anonymous access or personal access token permissions, and it remains readable when pipeline execution is disabled.
Each result identifies its project, repository, definition, exact source ref and revision. Status, completed-job counts, and start/completion times describe the current execution generation. A failed earlier attempt does not classify an admitted retry as failed. Open the exact run for actual jobs, logs, retained outputs, and earlier attempt generations; discovery does not manufacture job evidence, publication provenance, or deployment outcomes.
Filters and page cursors remain in the URL through reload. Back to organization pipelines restores that exact authorized discovery page from run detail, including after a direct run reload. Project filter options load in bounded pages with an explicit Load more project options action; selecting no project still searches all authorized projects on the server. Page counts describe only returned runs, with a continuation when more history is available. There is no aggregate run total or frozen history snapshot: current status and access can change between reads.
Opening or reopening a run checks current access before cached evidence appears. A current authorization failure clears protected run content. Unavailable durable storage is shown as an error, rather than as an empty organization.
Verify the current proposal snapshot.
- Open or synchronize the proposal.Each registered definition that enables
pull_requestschedules its own check. The legacy file retainsverify; other paths have distinct check identities shown with their runs. - Select a registered pipeline and use Run pipeline when needed.The selected file must declare
pull_requestin the exact source snapshot. The author can schedule their own check. A different member requires Run pipelines. After scheduling, Open scheduled run opens the exact returned run for that proposal and snapshot. The action disappears when the current proposal or snapshot changes. Use Back to pull request to return, then Back to pull requests to restore the validated filtered list when that navigation context is available. A direct link without list context falls back to the repository's pull requests. - Review the binding.The check records the snapshot ID, exact source revision, plan hash, attempt ID, conclusion, and trust.
- Rerun on new evidence.Synchronizing creates a new snapshot and revokes the old grant. Rerunning a check in the same snapshot replaces its own earlier run and grant without cancelling sibling pipeline checks.
A pull-request execution grant lasts 30 minutes. A required check counts only when it belongs to the current snapshot, is trusted according to the branch policy path, and concludes Passed.
Configure required checks in the repository's branch policy settings using the check identity displayed in the Pipelines registration catalog. The default policy requires verify, which is supplied only by a registered .mozaic/pipeline.yml. For other paths, replace or extend that requirement with their displayed identities. Registering a file does not change branch policies; renaming a registration leaves its check identity unchanged.
Read status at both run and attempt level.
The run page groups jobs and immutable attempt generations beside their ordered live or final logs. Select an attempt to inspect its completion details, declared plan steps and retained log downloads. Step status and timing are not recorded individually. Tests can be filtered by name, suite and outcome, with more cases loaded into view on demand; totals include every retained attempt, including retries. Artifacts and explicit package publication receipts preserve their exact attempt and content identity. Inspect the source and canonical plan for the immutable binding. The Published OCI content section shows explicit publication receipts across retained attempt generations. Each receipt identifies its exact run, job, attempt and generation, registry, native repository, manifest digest, media type and size. Tags record assignments at publication time; current mutable tags may point to different content. Expand Publication and source archive evidence to inspect the source OCI layout archive SHA-256 and size separately from the native manifest identity, plus any retained artifact binding. A successful job or retained archive alone does not establish publication. The Deployment section shows target-reported receipts when available; execution or registry publication does not establish a deployed release.
Open source revision follows the recorded repository identity and immutable Git commit. A PR snapshot identifier remains evidence and is not used to guess a proposal link. Published NuGet receipts offer Open published version only when the server independently authorizes the exact current catalog destination, binding its current project, registry, normalized package ID and immutable version. Pending, reserved, failed, incomplete or inaccessible targets offer no action; the historical publication receipt remains readable. A missing current target never falls back to the run origin. An OCI receipt offers Open current catalog version when the server resolves a currently authorized exact manifest in Artifacts. This current destination is separate from the immutable publication evidence; it does not add a repository identity to the historical receipt or follow a mutable tag. Missing or inaccessible catalog content leaves the receipt readable without a link.
- Run states
queued,running,succeeded,failed,indeterminate, orcancelled.- Attempt states
- The same progression plus
leased, which means a runner holds the fenced attempt but has not yet reported it running. - Immutable binding
- Repository revision, definition path, canonical plan, plan SHA-256, trust class, ref, trigger, requester, and optional pull-request snapshot/check.
- Logs
- Per-attempt compressed job logs exposed as downloadable result objects.
- Artifacts
- Declared artifact archives with logical name, byte size, and SHA-256. Cache archives are intentionally not downloadable through the user result endpoint.
- Tests
- TRX cases normalized to suite, name, class, outcome, duration, and optional error message.
- Packages
- Explicit NuGet feed publication records, including feed, entry path, digest, kind, state, normalized package identity, and failure reason.
- OCI publication
- Committed native manifest receipts bound to the exact producing attempt. Manifest digests and source archive checksums identify different content; publication-time tags are historical mutable references. No receipt means publication is not established, and the receipt grants no registry pull or deployment authority.
Wait for concurrent-job capacity
An accepted trusted-branch or tag revision stays in the dispatch queue while another run occupies the organization's concurrent-job allowance. Mozaic retries admission automatically with bounded backoff and dispatches the same accepted revision when capacity is released. This wait does not require another push, a higher quota or fresh MFA. A run appears in history after admission succeeds. During upgrade, eligible historical accepted revisions rejected by the previous quota classification receive one audited recovery attempt; normal admission checks still apply. Invalid resource requests, insufficient credits and other permanent admission failures remain refusals.
Cancel a run
The requester can cancel their own queued or running run. Another user needs Run pipelines. Cancellation is a request: already running work may take time to observe it, and the final result stays in history.
Indeterminate is different from failed
Failed means the attempt produced a definite unsuccessful result. Indeterminate means infrastructure lost certainty after execution may have begun. Mozaic does not silently retry such an attempt because doing so could repeat side effects.
The original requester can deliberately retry a whole terminal failed or indeterminate trusted-branch or tag run through POST /api/v1/organizations/{organization_id}/projects/{project_id}/pipelines/{run_id}/retry. Supply an Idempotency-Key, the current expected_generation, acknowledge_possible_side_effects: true and an audit reason of 8–512 non-control characters. A selected run_pipelines personal access token requires current organization and project execution authority; browser requests retain CSRF. Routine retry needs no fresh MFA. Current source management authority remains required; tag runs and package-publishing plans also need package management authority.
Retry preserves the original run, exact revision and resolved plan, appends new attempts for every job and retains earlier results. Current repository, registration, trust, retained source, publication targets and funding are checked again. Inspect the receipt's state: only enqueued admits execution; rejected records a durable refusal. After a lost response or temporary failure, repeat the exact key and body to resume the prepared request. Reserved funding is recovered through durable reconciliation if enqueue cannot finish. A run permits at most 32 retry intents, including prepared and rejected requests; exact existing-key receipts remain replayable at the limit. Manual or restricted runs, successful or explicitly cancelled runs, active work and another requester's run are excluded. No partial-job retry or retry button is included in this initial API workflow.
Resource transfers change current containment without rewriting registered pipeline identities or recorded execution context. Current project discovery and new manual or trusted-ref scheduling use the destination project and its current authorization. Existing run history, review checks, attempts and publication provenance retain their accepted project. A retry whose original context no longer matches current repository or publication containment is rejected; moving a resource back does not revive old generation authority. Schedule a fresh destination run after reviewing its current targets and grants. Historical check links use the independently authorized recorded run project; they never redirect the run to the repository current project.
Any project member can view project run history and download logs or declared artifacts. Public run history is a smaller read-only view and exists only when the organization and project are public.
Make feed publication an explicit terminal step.
Declaring a .nupkg as an artifact makes it downloadable from the run; it does not publish it. Add the canonical publish_in_package_registry action after all executable steps:
- name: Publish packages
publish_in_package_registry:
registry:
slug: releases
source:
adapter: nuget
paths:
- artifacts/packages/*.nupkg
- artifacts/packages/*.snupkg
allow_untrusted: false
- Select exactly one feed by
idorslug. The scheduler freezes a slug to an immutable feed ID before execution. - Declare 1-64 safe workspace-root NuGet glob patterns. Every pattern must contain
.nupkgor.snupkg. - No
runorusesstep may follow the first registry-publication step. allow_untrusteddefaults to false. For NuGet, external contribution runs execute the build and retain artifacts, then skip publication with a warning. Skipping publication does not itself fail the job. Authorized manual publication uses the initiating user's package permissions and does not require this override. Keep it false for release feeds.
One definition can enable triggers: [pull_request, manual, tag] and keep allow_untrusted: false: pull-request runs verify and pack without publishing; an authorized manual run can publish its declared outputs, and an accepted tag runs the same jobs with trusted publication. Selecting a tag in the manual picker retains manual provenance and restricted execution trust, with separately authorized publication. There is no tag-name filter, so every accepted tag can trigger a release. Commit a new package version before creating and pushing a release tag.
After a supported repository or NuGet/OCI identity transfer, review registered definitions and publication selectors in the current destination before scheduling. Native UUID registry endpoints remain stable, while current project grants and generation-bound runner publication authority are checked again. Historical publication receipts keep their original identities and execution project; current catalog navigation is authorized separately.
Create the target NuGet feed in the same project before scheduling any run, including a restricted run: its selector is resolved before execution. Native pipeline publication uses runner-bound credentials; do not create a personal feed token or call dotnet nuget push for this step.
A manual NuGet declaration selects a feed before the package identity is parsed. This feed-wide authorization excludes package IDs with an explicit retained ownership override; it cannot supply that package's captured ownership generation. OCI declarations name their repository before scheduling and use that exact native target's permissions and generation.
The older NuGet publish_in_feed spelling remains accepted and compiles to the same immutable plan. OCI's publish_in_registry alias behaves the same way. Adapter sources whose fenced runner publisher is not installed fail validation before a job is scheduled.
See Packages for feed creation, token pushes, symbols, lifecycle, and public-feed gates.
Design inside the operated safety envelope.
| Boundary | Current project pipeline limit |
|---|---|
| Definition | 1-262,144 UTF-8 bytes; schema version 1; at most 64 spaces of indentation. |
| Jobs | 1-20 per pipeline. |
| Services | 0-3 per job. |
| Steps | 1-50 per job; run command at most 16 KiB. |
| CPU | Ordinary worker: 1-2,000 millicores per job; default 2,000. |
| Memory | Ordinary worker: 1-4,096 MiB per job; default 4,096 MiB. |
| Scratch | Ordinary worker: 1-10,240 MiB per job; default 10,240 MiB. |
| Processes | Ordinary worker: 1-512 PIDs per job; default 512. |
| Logs | 1-256 MiB per job; default 100 MiB. |
| Timeout | 1-30 minutes per job; default 30. |
| Network hosts | At most 100 explicit public DNS names; no IP literals, localhost, .local, or .internal. |
An operator may prepare a dedicated, single-slot Fedora worker with a fixed disk workspace. Jobs requesting more than 10,240 MiB scratch must select both host-profile: fedora-44-cloud-v1 and workspace-storage: bounded-disk-v1 under runs_on.labels. With the ADR-0120 server update installed, the installation compiler permits those individually selected jobs to request up to 8,000 millicores, 16,384 MiB memory, 102,400 MiB scratch and 120 minutes. The existing 512-PID compilation limit and organization entitlements remain unchanged. Jobs without both selectors retain the ordinary 2,000-millicore, 4,096-MiB, 30-minute limits. An approved runner configuration remains the execution ceiling; a smaller configuration cannot execute a larger compiled request. Older servers still reject CPU, memory and timeout requests above the ordinary limits even when the worker configuration is larger. Each job must declare the entire configured disk envelope; smaller scratch requests fail. Selectors request placement and do not authorize or configure a worker: execution still requires an authorized matching applied configuration and an independently verified disk. The workspace uses preallocated disk, while job temporary files have a separate 64 MiB limit. Put large temporary build files under /workspace. Availability depends on operator acceptance, the disposable cloud acceptance gate and configured capacity; these limits do not enlarge an ordinary worker or the native Dockerfile builder.
- Jobs execute in ephemeral, rootless isolation with bounded CPU, memory, scratch, PIDs, log size, time, and network policy.
- The operator-owned organization can use an explicitly approved trusted worker on an existing shared Debian host. This separate profile accepts only that exact organization's trusted runs and retains rootless execution and resource limits; it does not claim per-container AppArmor confinement or accept external-restricted contributions. It is not a fallback that another tenant can select. Operators must verify the actual host before admitting jobs and revisit isolation before opening that organization to untrusted code.
- Container and action images are digest-pinned; unknown fields and unsupported dynamic expressions fail closed.
- Restricted jobs do not receive protected secrets, privileged mode, host mounts, deployment authority, or implicit package-publication authority.
- The operated static OCI deployment pilot uses a separate operator approval and deployment account. Only successful accepted trusted-branch publication evidence qualifies; the active ordinary-job path checks its successful declared prerequisites and exact receipts, while legacy latest-release polling waits for the whole run. A pipeline run does not itself grant production access.
- A separate operated adapter is limited to approved Website, JodelCity, Bantapop, UTF and TeamSpeak targets. An organization owner uses fresh MFA to provision or rotate a dedicated target credential and supported runtime inputs. Existing private values staged on the deployment host can be activated through a five-minute public approval URL: the owner reviews exact target, source, version and key names and explicitly approves a one-time upload and 30-day credential. This connects the host initially; it is not approval for every release. Values and credentials stay off the browser page; the host saves the credential privately. Arrange owner-authorized replacement before expiry, or separately enroll the host for ongoing renewal: the owner reviews a five-minute metadata request and explicitly selects initially unchecked until-revoked consent with fresh MFA. Renewal binds the exact organization, current placement, fixed target, immutable source and approving owner; activation and publication do not imply this consent. Each new read credential lasts 30 days and rotates seven days before expiry. The host keeps the grant and rotation journal private, verifies and atomically installs the candidate before confirmation, and retires only that grant’s previous read credential. Revoking a grant or its completed request also revokes all derived credentials; unrelated manual credentials remain separate. At most two unrevoked grants and eight active read credentials share each target. Owner settings show grant metadata and revoked history, never secret values or tokens. A configured renewal failure blocks new promotion while healthy accepted containers and separately authorized offline recovery remain available. Renewal does not clear a failed-release fence or authorize a new recovery attempt. Expiry, revocation or loss of issuer authorization stops retrieval. If a completion response is lost, that page offers fresh-MFA credential revocation and shows the current version for a new request. Deployment polls use the dedicated credential for release evidence bound to the configured source repository's main branch; personal onboarding tokens are not required for ongoing deployment. All images must come from one accepted trusted run. Runtime secrets remain separate from jobs, with controlled backups and recovery fencing before startup. The source includes this bounded renewal capability; production enablement still requires installed-controller and actual target-host HTTPS acceptance plus separate fresh-MFA owner enrollment. The selected real desktop Chromium owner journey is not full browser-matrix or production-controller acceptance. General deployment environments and a general-purpose secret vault remain outside this adapter.
- TeamSpeak runtime promotion, automatic resume and recovery are currently fenced in the protected controller source pending review and live acceptance of its anchored startup transaction. TeamSpeak polls can reconcile credential renewal and observe publication without starting the engine. Operators must retain any pending migration fence and use the separate protected recovery procedure only after that transaction is accepted.
- Jodel's fixed protected adapter has a separate first-install transaction requiring operator installation and actual host acceptance. Normal polling can renew credentials and observe publications, but cannot start, update or recover Jodel. Private runtime acceptance precedes public TLS and WebSocket cutover. Automatic deployment of later main pushes requires its separate accepted update transaction.
- Source staging verifies the Git bundle and referenced LFS objects against the selected repository revision and canonical plan.
- Artifact and cache paths are validated before scheduling; archive extraction is separately bounded and traversal-safe.
Resolve common failures.
Configure and inspect runner hosts
Installation operators manage hosts under Admin → Runners. The overview shows loaded-host capacity and recorded problems; open a host or group for Overview, Job activity or Hosts, Configuration, History, and Manage. The host overview links directly to Revoke enrollment; Manage lists lifecycle actions with their purpose. Supply an audit reason of at least eight characters and complete fresh MFA if prompted. Hard stop requires the exact displayed confirmation. The result shows the returned command state and command ID; pending, accepted, or running does not mean completion. Refresh details and verify the expected changes before continuing. When creating a runner or group, compare server-size cards by CPU, memory, and disk, then set parallel jobs per runner. The newest compatible known runner software is selected automatically. Open the older-software list only when you need a previous build; available version and build dates identify choices instead of hashes. For organization-owned capacity, search for the organization by name instead of copying its UUID. Mozaic reads provider hardware and calculates suggested job limits with room reserved for the host. The software selection retains its exact image digest, so you do not need to copy a hash. Reviewed public images can provide verified build metadata. Build dates and dates when a configuration was added are labeled separately; unavailable metadata uses an explicit known-configuration fallback. The catalog is bounded and does not promise every registry release. Creating a runner pins the selected software; later releases do not silently upgrade it. When editing an existing runner or group, use the guided hardware, workload, and software choices to review a change. The current configuration remains in place until you choose and save a new revision; catalog refresh does not silently upgrade the software or recalculate your draft. Provider facts and suggested per-job limits remain advisory, and host preflight determines actual capacity. If catalog choices are unavailable, retry or use the manual and advanced settings with reviewed inputs. Image hashes and catalog diagnostics are available in technical details. A group shares one size and policy; its runner count is separate from jobs per runner. Review the capacity summary and choose whether to provision hosts immediately. Advanced settings allow deliberate resource and execution-policy overrides. If setup data is unavailable or no compatible software is known yet, retry or use Configure hardware and image manually with reviewed inputs. Actual host preflight still controls execution capacity. In manual configuration, a blank Cloud server size uses the deployment default. Changing size drains and reprovisions hosts through the existing replacement workflow. The runner software bundle and pipeline job images remain separate settings.
Budgeted cloud capacity requires an explicitly reviewed operator policy. Its physical minimum defaults to zero; a positive minimum funds warm hosts separately from logical runner group size. Pending hosts require authenticated readiness, and operator cash ceilings and modeled billing boundaries are not actual invoices. Expand Budgeted cloud workers after the telemetry disclosure to read the private accounting projection. It shows exact held and recognized amounts for the returned UTC window, physical pool limits, all open workers and up to eight recent finite closures per pool. Select a worker to inspect readiness, distinct boot/execution/liability deadlines, and separate owned server and primary IP billing-start observations with their contract basis. Uncertain liability remains held; elapsed deadlines and empty lists do not establish zero provider charges. This read-only section does not request provider prices or activate capacity; policy configuration remains an operator CLI operation.
In Manage, Delete asks for an audit reason and the exact displayed name. Its receipt shows whether removal completed or is still pending; status refreshes do not submit another deletion. If a request fails or its outcome is unknown, follow the displayed refresh or retry action. Already deprovisioned runners, safe retired external runners and safely terminated groups can be removed without provider automation. Active work, usable credentials, or provider operations that cannot be safely stopped block removal with an explanation. When provider control is unavailable, a durably hard-stopped standalone runner can be deleted while its provider resources and cleanup history remain recorded; deletion in this case does not remove those resources. If a provider create outcome remains unknown, an explicit acknowledgment stops eligible waiting cleanup and preserves its evidence. A previously submitted create may still result in a server, which the operator must check with the provider. Explicit Hard stop fences execution and access while retaining provider resources. Archive hides an eligible fenced runner without claiming that infrastructure was removed.
Active jobs count leased and running attempts. Running-container counts instead come from the runner user's rootless Podman engine, including services and proxies; they exclude other users' engines and expire after 90 seconds. Older runners and failed samples show unavailable, never zero.
On the fleet page, expand Container telemetry & cost estimates below the readiness summary to inspect the loaded hosts. Opening this disclosure does not request provider observations; choose Load cost estimates explicitly. Applied filters and loaded pages determine its scope; unloaded hosts are excluded.
Load cost estimates inspects up to 50 loaded hosts. Infrastructure estimates use the observed cloud machine and location at the provider's current net hourly tariff, before taxes, caps, or account adjustments. Credit estimates use currently running jobs' recorded CPU allocations and immutable billing rates, not settled ledger charges. Totals cover loaded, filtered results and explicitly show partial coverage; currencies remain separate. Organization-owned hosts also link to the organization's credit ledger and show organization-wide wallet balances, not per-host spend.
Demand and idle runner hosts
An enabled managed runner saved without a cloud host can be provisioned when compatible queued work needs capacity. Provisioning creates billable provider resources. Idle hosts remain subject to their configured idle timeout and shutdown safety checks; the timeout is not a billing-period deadline.
Pipeline execution is disabled
The installation has disabled its runner control plane. You can still select repositories and references, inspect registered definitions and history, and add or remove registrations; scheduling stays disabled. A configured or idle runner does not override this application setting. Ask the installation operator to verify the effective MOZAIC_RUNNER_CONTROL_ENABLED setting and enable execution through the documented deployment procedure once the configured runner infrastructure is ready. The local development Compose stack deliberately disables execution because it does not start a runner supervisor. After activation, schedule the selected definition manually; an eligible runner must still claim the jobs.
Pipeline definition was not found or is not UTF-8
Check that the selected definition is committed at its exact supported path in the selected revision. A working-tree file that was never pushed is not part of the run.
The selected trigger is not enabled
Add the exact repository-event trigger name to triggers, commit it, and select the new revision. Pull-request checks require pull_request; manual scheduling does not require a trigger declaration.
Container image must be digest-pinned
Replace a tag with an immutable reference ending in @sha256: and 64 lowercase hexadecimal characters. A digest belongs on every job image, service image, and uses reference.
Uses reference is not allowlisted
The action must be a digest-pinned oci:// reference present in the operated allowlist. Use a normal run step with the job image or an already supported action; YAML cannot expand the allowlist.
Network destination is rejected
Set mode to http_https. Entries in allow_hosts must be public DNS hostnames without a scheme, port, IP address, wildcard, loopback name or private suffix. Explicit CONNECT uses separate allow_connect host/port pairs and can name a canonical public IP literal. Private, reserved and noncanonical addresses remain denied.
The run is queued but does not start
Check project job concurrency and service-credit availability, then confirm no dependency job has failed. See Billing & limits.
A package file exists but nothing was published
An artifact declaration only retains bytes. Add an explicit terminal publish_in_package_registry step, verify its adapter and paths match, and inspect the run's Published packages section for a per-entry failure reason.
Explicit deployment branch consent
Fixed deployment targets can use main or master. Selecting a repository default does not retarget deployment credentials. Master requires a new owner activation and separate ongoing renewal consent with fresh MFA. Check the exact branch on each approval page. Existing main grants retain their scope; release evidence must match the configured branch exactly.