From 2b619b9e76ec0161bd2bdad85e04bac0d4f0d57c Mon Sep 17 00:00:00 2001 From: OpenIPC AI team Date: Sun, 30 Aug 2026 11:00:05 +0300 Subject: [PATCH] publish with an explicit release credential, and prove it before building (#138) * publish with an explicit release credential, and prove it before building This repo was next in line for the failure OpenIPC/firmware hit on 2026-08-29. Moving the nightly off `schedule:` onto a dispatch changes the identity the run executes as, and the default GITHUB_TOKEN's release-write goes with it: a scheduled run executes as the repository's scheduling identity, a dispatched one as whoever dispatched it, and for an app/OAuth-backed identity the releases API answers 403 "Resource not accessible by integration". It does so even though the job declares `contents: write` and the run log confirms the token was granted it, which is why nothing in the workflow file looks wrong. firmware's run 33273333359 built 102 boards over two hours before finding out. This repo builds 107 devices and publishes the same way -- `gh release create` plus a PATCH of refs/tags/latest -- so it would have failed identically the first time the train reached it. It never got that far only because the train correctly refused to advance past firmware's failure. The publish step takes RELEASE_TOKEN instead of the default token, and preflight proves that credential can create a release before the matrix is allowed to spend hours on it, by creating a draft release and deleting it: invisible, no tag, two API calls. Asserting the secret is merely non-empty would not have caught the firmware incident. The token that failed existed and was simply refused, so the probe has to attempt the write. * probe: survive re-runs and clean up after itself Review caught a real defect in the credential probe. It deleted its draft with a single fail-fast call, so an interrupted or cancelled job left the draft behind -- and because github.run_id is stable across re-runs, that leftover would collide with its own re-run and report a perfectly good credential as broken. A guard that fails wrongly is worse than no guard, because it blocks good runs and teaches people to ignore it. Three changes. The probe tag now carries run_attempt as well as run_id, so a re-run can never collide with its own leftovers. Deletion moved into a trap, so a cancelled job still cleans up. And a best-effort sweep removes probe drafts stranded by earlier runs, because cleanup.yml does not know this tag pattern and they would otherwise accumulate forever. The sweep is capped at the 10 most recent releases and swallows every error. That cap is not arbitrary: listing 100 releases in these repos returns HTTP 504 because each carries hundreds of assets, measured at ~2s for 10 and ~7s for 30. Housekeeping must never be the thing that fails a build. Verified end to end against the live repo with a real token: creates the draft, reports "release credential OK", the trap removes it, no tag is created and no published release is touched. Both failure paths still fail loudly -- unset token, and a credential the API refuses. * probe: retry the write, and do not narrate garbage while sweeping Review caught that the probe treated any one-shot POST failure as a bad credential. GitHub answers 403 for secondary rate limiting as well as for refusal -- the publish step retries six times for exactly that reason -- and these repos return 504 on ordinary release listings often enough that one was hit while writing this change. A blip would have failed the nightly with "the token is wrong", sending whoever read it off to reissue a credential that was never the problem. The POST now retries three times with 5s and 10s backoff, and the error says "after 3 attempts ... if this is not a transient API error". Three rather than the publish step`s six because the value here is failing fast: a genuine misconfiguration still surfaces in well under a minute instead of after the whole matrix. Testing the retry then exposed a second defect, in the sweep added by the previous commit. gh writes its error body to STDOUT, so a failed list feeds the cleanup loop lines of JSON instead of release ids, and the log filled with `removing stranded probe draft {"status": "401"}`. The deletes were harmless -- they simply failed -- but a step whose entire purpose is to be believed must not narrate confident nonsense. Only numeric ids are acted on now. Verified against the live repo: happy path creates the draft, reports OK, the trap removes it, nothing is left behind; a refused credential retries three times and fails with the honest message; a failing list produces zero bogus removal lines. --- .github/workflows/master.yml | 106 ++++++++++++++++++++++++++++++++++- 1 file changed, 105 insertions(+), 1 deletion(-) diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml index 6690ab0..f2ebdab 100644 --- a/.github/workflows/master.yml +++ b/.github/workflows/master.yml @@ -129,6 +129,99 @@ jobs: echo "build_id=$BUILD_ID" >> "$GITHUB_OUTPUT" echo "built_at=$BUILT_AT" >> "$GITHUB_OUTPUT" + # Prove the release credential BEFORE the matrix runs. + # + # OpenIPC/firmware run 33273333359 built 102 boards over two hours and then + # died in `Publish releases` with "403 Resource not accessible by + # integration". Nothing earlier in that run had any opinion about whether + # it could publish, so the price of the misconfiguration was the whole + # matrix. This repo builds 107 devices and publishes the same way. + # + # A draft release is not visible to anyone and creates no tag, so this + # costs a couple of API calls and turns a long failure into a + # ten-second one. Checking the secret is merely non-empty would not have + # caught that incident at all: the old token existed and was simply + # refused, so the probe has to attempt the write. + # + # Three details keep the probe from becoming a problem of its own: + # + # - The tag carries run_attempt as well as run_id. run_id is stable + # across re-runs, so without the attempt a leftover draft from a + # failed run would collide with its own re-run and report a valid + # credential as broken -- a guard that fails wrongly is worse than no + # guard, because it blocks good runs. + # - Deletion happens from a trap, so an interrupted or cancelled job + # still cleans up rather than leaving the draft behind. + # - A best-effort sweep removes probe drafts stranded by earlier runs, + # because cleanup.yml does not know this tag pattern. It is capped at + # the 10 most recent releases and every failure is swallowed: listing + # 100 releases here returns HTTP 504 (each carries hundreds of + # assets), and a housekeeping step must never be what fails a build. + # + # Skipped on pull_request, which neither publishes nor has the secret. + - name: Prove the release credential can write + if: github.event_name != 'pull_request' + env: + GH_TOKEN: ${{ secrets.RELEASE_TOKEN }} + GH_REPO: ${{ github.repository }} + PROBE_TAG: ci-release-credential-probe-${{ github.run_id }}-${{ github.run_attempt }} + run: | + set -euo pipefail + + if [ -z "${GH_TOKEN:-}" ]; then + echo "::error::RELEASE_TOKEN is not set. The matrix would build"\ + "every device and then fail to publish. See the publish job." + exit 1 + fi + + probe_id="" + cleanup() { + [ -n "$probe_id" ] || return 0 + gh api -X DELETE "repos/${GH_REPO}/releases/${probe_id}" >/dev/null 2>&1 || true + } + trap cleanup EXIT + + # Strand-clearing, entirely best effort. + gh api "repos/${GH_REPO}/releases?per_page=10" \ + --jq '.[] | select(.draft and (.tag_name | startswith("ci-release-credential-probe-"))) | .id' \ + 2>/dev/null | while read -r stale; do + # gh writes its error body to STDOUT, so a failed list feeds this + # loop lines of JSON rather than ids. Deleting them is harmless -- + # the calls just fail -- but logging "removing stranded probe + # draft {"status": "401"}" is a confident lie in the log of a step + # whose whole job is to be believed. Only act on real ids. + case "$stale" in ''|*[!0-9]*) continue ;; esac + echo "removing stranded probe draft $stale" + gh api -X DELETE "repos/${GH_REPO}/releases/${stale}" >/dev/null 2>&1 || true + done || true + + # Retried, because a one-shot POST cannot tell a bad credential from + # a bad minute. GitHub answers 403 for secondary rate limiting as + # well as for refusal -- the publish step below retries six times for + # exactly that reason -- and these repos return 504 on ordinary + # release listings often enough to have been seen while writing this. + # Reporting a blip as "the token is wrong" would send whoever reads + # it to reissue a credential that was never the problem. + # + # Three attempts rather than the publish step's six: the value here is + # failing fast, and 5s + 10s of backoff still absorbs a transient + # while keeping a genuine misconfiguration well under a minute. + attempt=0 + until probe_id=$(gh api -X POST "repos/${GH_REPO}/releases" \ + -f tag_name="$PROBE_TAG" -f name="$PROBE_TAG" \ + -F draft=true --jq .id); do + attempt=$((attempt + 1)) + if [ "$attempt" -ge 3 ]; then + echo "::error::RELEASE_TOKEN could not create a release in"\ + "${GH_REPO} after 3 attempts. If this is not a transient API"\ + "error, the token needs Contents: read and write." + exit 1 + fi + echo "attempt ${attempt} failed, retrying in $((attempt * 5))s" + sleep $((attempt * 5)) + done + echo "release credential OK" + buildroot: name: Firmware needs: [preflight, select] @@ -450,7 +543,18 @@ jobs: - name: Publish releases (paced, retry-aware) if: steps.collect.outputs.count != '0' env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Not GITHUB_TOKEN. The default token's release-write depends on + # who triggered the run: a `schedule:` run executes as the repository's + # scheduling identity and could write releases, but a dispatched run + # executes as whoever dispatched it, and for an app/OAuth-backed + # identity the API answers "403 Resource not accessible by + # integration" even though the job declares `contents: write` and the + # run log confirms the token was granted it. OpenIPC/firmware hit + # exactly this on 2026-08-29 (its run 33273333359 built for two hours + # and then could not publish). This repo publishes the same way and + # was next in line. An explicit credential also makes publishing + # independent of who or what starts the run. + GH_TOKEN: ${{ secrets.RELEASE_TOKEN }} GH_REPO: ${{ github.repository }} BUILD_ID: ${{ needs.preflight.outputs.build_id }} HEAD_SHA: ${{ needs.preflight.outputs.head_sha }}