OpenIPC AI teamandGitHub
2b619b9e76
* publish with an explicit release credential, and prove it before building
This repo was next in line for the failure OpenIPC/firmware hit on
2026-08-29. Moving the nightly off `schedule:` onto a dispatch changes
the identity the run executes as, and the default GITHUB_TOKEN's
release-write goes with it: a scheduled run executes as the repository's
scheduling identity, a dispatched one as whoever dispatched it, and for
an app/OAuth-backed identity the releases API answers 403 "Resource not
accessible by integration".
It does so even though the job declares `contents: write` and the run
log confirms the token was granted it, which is why nothing in the
workflow file looks wrong. firmware's run 33273333359 built 102 boards
over two hours before finding out. This repo builds 107 devices and
publishes the same way -- `gh release create` plus a PATCH of
refs/tags/latest -- so it would have failed identically the first time
the train reached it. It never got that far only because the train
correctly refused to advance past firmware's failure.
The publish step takes RELEASE_TOKEN instead of the default token, and
preflight proves that credential can create a release before the matrix
is allowed to spend hours on it, by creating a draft release and
deleting it: invisible, no tag, two API calls.
Asserting the secret is merely non-empty would not have caught the
firmware incident. The token that failed existed and was simply refused,
so the probe has to attempt the write.
* probe: survive re-runs and clean up after itself
Review caught a real defect in the credential probe. It deleted its
draft with a single fail-fast call, so an interrupted or cancelled job
left the draft behind -- and because github.run_id is stable across
re-runs, that leftover would collide with its own re-run and report a
perfectly good credential as broken. A guard that fails wrongly is worse
than no guard, because it blocks good runs and teaches people to ignore
it.
Three changes. The probe tag now carries run_attempt as well as run_id,
so a re-run can never collide with its own leftovers. Deletion moved
into a trap, so a cancelled job still cleans up. And a best-effort sweep
removes probe drafts stranded by earlier runs, because cleanup.yml does
not know this tag pattern and they would otherwise accumulate forever.
The sweep is capped at the 10 most recent releases and swallows every
error. That cap is not arbitrary: listing 100 releases in these repos
returns HTTP 504 because each carries hundreds of assets, measured at
~2s for 10 and ~7s for 30. Housekeeping must never be the thing that
fails a build.
Verified end to end against the live repo with a real token: creates the
draft, reports "release credential OK", the trap removes it, no tag is
created and no published release is touched. Both failure paths still
fail loudly -- unset token, and a credential the API refuses.
* probe: retry the write, and do not narrate garbage while sweeping
Review caught that the probe treated any one-shot POST failure as a bad
credential. GitHub answers 403 for secondary rate limiting as well as
for refusal -- the publish step retries six times for exactly that
reason -- and these repos return 504 on ordinary release listings often
enough that one was hit while writing this change. A blip would have
failed the nightly with "the token is wrong", sending whoever read it
off to reissue a credential that was never the problem.
The POST now retries three times with 5s and 10s backoff, and the error
says "after 3 attempts ... if this is not a transient API error". Three
rather than the publish step`s six because the value here is failing
fast: a genuine misconfiguration still surfaces in well under a minute
instead of after the whole matrix.
Testing the retry then exposed a second defect, in the sweep added by
the previous commit. gh writes its error body to STDOUT, so a failed
list feeds the cleanup loop lines of JSON instead of release ids, and
the log filled with `removing stranded probe draft {"status": "401"}`.
The deletes were harmless -- they simply failed -- but a step whose
entire purpose is to be believed must not narrate confident nonsense.
Only numeric ids are acted on now.
Verified against the live repo: happy path creates the draft, reports
OK, the trap removes it, nothing is left behind; a refused credential
retries three times and fails with the honest message; a failing list
produces zero bogus removal lines.
683 lines
34 KiB
YAML
683 lines
34 KiB
YAML
name: Build
|
|
on:
|
|
# There was no pull_request trigger here: a change landed unbuilt and the
|
|
# nightly found out the next morning, or a user did. Building all 107 devices
|
|
# per PR is what made that unaffordable, so .github/scripts/ci-matrix.py picks
|
|
# only the devices a PR can reach. ready_for_review is not in the default set
|
|
# (opened, synchronize, reopened), and the selector skips drafts, so without it
|
|
# a PR opened as a draft would never build once marked ready.
|
|
pull_request:
|
|
branches:
|
|
- master
|
|
types: [opened, synchronize, reopened, ready_for_review]
|
|
# There is deliberately no `schedule:` here any more, and adding one back
|
|
# would undo a fix rather than restore a feature.
|
|
#
|
|
# This is the last stage of a three-stage release train — widgetii/majestic
|
|
# publishes the majestic tarball, OpenIPC/firmware builds the images, this
|
|
# builds from that tree. Each stage used to be its own GitHub cron guessing at
|
|
# the previous stage's finish time (17:25, 22:30, 03:00 UTC). From 2026-08-26
|
|
# GitHub's dispatcher began delivering scheduled events hours late and
|
|
# independently per repo, and this repository was the worst affected of the
|
|
# three: +10h38m and +11h49m on successive nights, because a 03:00 slot sits
|
|
# in the deepest part of the congestion band. On the 08-28 cycle that made
|
|
# this stage run FIRST, at 14:49, ahead of both the repositories it is
|
|
# supposed to follow. Every run was green, because no stage checks that its
|
|
# inputs are newer than the ones it used last time.
|
|
#
|
|
# The diagnostic, if this is ever doubted: on every late run
|
|
# `created_at == run_started_at` to the second, so the delay was entirely
|
|
# upstream of run creation, inside GitHub's dispatcher rather than in any
|
|
# runner pool, concurrency group or matrix.
|
|
#
|
|
# The ordering now lives on the machine that runs majestic's self-hosted
|
|
# runners, which dispatches each stage and waits for it to succeed before
|
|
# starting the next. See widgetii/majestic docs/ci-runners.md.
|
|
workflow_dispatch:
|
|
inputs:
|
|
platform:
|
|
description: Platform
|
|
required: false
|
|
|
|
# Cancel a PR's earlier run when a new commit lands on it. Scheduled and
|
|
# dispatched runs are deliberately left alone: they write releases, and a
|
|
# cancelled publish leaves the nightly half-uploaded.
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
jobs:
|
|
# Which devices can this change actually reach? Only ever narrows a
|
|
# pull_request -- the nightly is decided by firmware HEAD, not by this repo's
|
|
# diff, so every other event gets the whole matrix.
|
|
#
|
|
# The narrowing happens HERE, in a job that always runs, and never in a
|
|
# `paths:` filter on the trigger: a filtered-out workflow does not run at all,
|
|
# so a required check would stay pending forever and the PR could never merge.
|
|
select:
|
|
name: Select devices
|
|
# Same guard as preflight below.
|
|
if: >-
|
|
github.repository == 'OpenIPC/builder' ||
|
|
github.event_name == 'workflow_dispatch' ||
|
|
(github.event_name == 'pull_request' && !github.event.repository.private)
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
matrix: ${{ steps.pick.outputs.matrix }}
|
|
needs-build: ${{ steps.pick.outputs.needs-build }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Check the selector still describes the tree
|
|
run: python3 .github/scripts/ci-matrix.py --self-test
|
|
- id: pick
|
|
name: Pick the affected devices
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: python3 .github/scripts/ci-matrix.py
|
|
|
|
preflight:
|
|
name: Preflight
|
|
# Guard rail for clones of this repo. A clone pushed to a new repository
|
|
# (a private mirror, an internal re-host) inherits this file along with
|
|
# and the full platform matrix then runs on that owner's Actions bill on
|
|
# every internal PR they open. It used to also inherit a `cron:` and do
|
|
# that nightly and unattended, publishing nightlies nobody reads; that half
|
|
# went with the schedule trigger, the PR half did not. Mirrors
|
|
# OpenIPC/firmware#2255.
|
|
#
|
|
# Unattended runs happen only on the canonical repo. Anywhere else a
|
|
# workflow runs only if someone explicitly asked for it, or if it is both
|
|
# free and attended -- which is what the pull_request clause covers, now
|
|
# that this workflow has such a trigger.
|
|
if: >-
|
|
github.repository == 'OpenIPC/builder' ||
|
|
github.event_name == 'workflow_dispatch' ||
|
|
(github.event_name == 'pull_request' && !github.event.repository.private)
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
should_build: ${{ steps.gate.outputs.should_build }}
|
|
head_sha: ${{ steps.gate.outputs.head_sha }}
|
|
short_sha: ${{ steps.gate.outputs.short_sha }}
|
|
build_id: ${{ steps.gate.outputs.build_id }}
|
|
built_at: ${{ steps.gate.outputs.built_at }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- id: gate
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
HEAD=$(git rev-parse HEAD)
|
|
SHORT=$(git rev-parse --short HEAD)
|
|
BUILD_ID="nightly-$(date -u +%Y%m%d)-${SHORT}"
|
|
BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
|
# The nightly rebuilds unconditionally. builder.sh re-clones
|
|
# OpenIPC/firmware at HEAD on every run, so the relevant inputs are
|
|
# mostly outside this repo (firmware HEAD, toolchain release, kernel
|
|
# tarball, vendor osdrv, individual package upstreams). The previous
|
|
# gate skipped when only this repo's HEAD matched the last published
|
|
# nightly, which made upstream fixes (e.g. an openhisilicon mipi_rx
|
|
# regression fix that lands in firmware) invisible to users until
|
|
# something here happened to change. Always build.
|
|
#
|
|
# Note this is why removing the `schedule:` trigger needed no gate
|
|
# change here, unlike majestic and firmware: there is no event-name
|
|
# condition to go stale, because there is no skip.
|
|
echo "Build: $BUILD_ID (event=${{ github.event_name }})"
|
|
echo "should_build=true" >> "$GITHUB_OUTPUT"
|
|
echo "head_sha=$HEAD" >> "$GITHUB_OUTPUT"
|
|
echo "short_sha=$SHORT" >> "$GITHUB_OUTPUT"
|
|
echo "build_id=$BUILD_ID" >> "$GITHUB_OUTPUT"
|
|
echo "built_at=$BUILT_AT" >> "$GITHUB_OUTPUT"
|
|
|
|
# Prove the release credential BEFORE the matrix runs.
|
|
#
|
|
# OpenIPC/firmware run 33273333359 built 102 boards over two hours and then
|
|
# died in `Publish releases` with "403 Resource not accessible by
|
|
# integration". Nothing earlier in that run had any opinion about whether
|
|
# it could publish, so the price of the misconfiguration was the whole
|
|
# matrix. This repo builds 107 devices and publishes the same way.
|
|
#
|
|
# A draft release is not visible to anyone and creates no tag, so this
|
|
# costs a couple of API calls and turns a long failure into a
|
|
# ten-second one. Checking the secret is merely non-empty would not have
|
|
# caught that incident at all: the old token existed and was simply
|
|
# refused, so the probe has to attempt the write.
|
|
#
|
|
# Three details keep the probe from becoming a problem of its own:
|
|
#
|
|
# - The tag carries run_attempt as well as run_id. run_id is stable
|
|
# across re-runs, so without the attempt a leftover draft from a
|
|
# failed run would collide with its own re-run and report a valid
|
|
# credential as broken -- a guard that fails wrongly is worse than no
|
|
# guard, because it blocks good runs.
|
|
# - Deletion happens from a trap, so an interrupted or cancelled job
|
|
# still cleans up rather than leaving the draft behind.
|
|
# - A best-effort sweep removes probe drafts stranded by earlier runs,
|
|
# because cleanup.yml does not know this tag pattern. It is capped at
|
|
# the 10 most recent releases and every failure is swallowed: listing
|
|
# 100 releases here returns HTTP 504 (each carries hundreds of
|
|
# assets), and a housekeeping step must never be what fails a build.
|
|
#
|
|
# Skipped on pull_request, which neither publishes nor has the secret.
|
|
- name: Prove the release credential can write
|
|
if: github.event_name != 'pull_request'
|
|
env:
|
|
GH_TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
GH_REPO: ${{ github.repository }}
|
|
PROBE_TAG: ci-release-credential-probe-${{ github.run_id }}-${{ github.run_attempt }}
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
if [ -z "${GH_TOKEN:-}" ]; then
|
|
echo "::error::RELEASE_TOKEN is not set. The matrix would build"\
|
|
"every device and then fail to publish. See the publish job."
|
|
exit 1
|
|
fi
|
|
|
|
probe_id=""
|
|
cleanup() {
|
|
[ -n "$probe_id" ] || return 0
|
|
gh api -X DELETE "repos/${GH_REPO}/releases/${probe_id}" >/dev/null 2>&1 || true
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
# Strand-clearing, entirely best effort.
|
|
gh api "repos/${GH_REPO}/releases?per_page=10" \
|
|
--jq '.[] | select(.draft and (.tag_name | startswith("ci-release-credential-probe-"))) | .id' \
|
|
2>/dev/null | while read -r stale; do
|
|
# gh writes its error body to STDOUT, so a failed list feeds this
|
|
# loop lines of JSON rather than ids. Deleting them is harmless --
|
|
# the calls just fail -- but logging "removing stranded probe
|
|
# draft {"status": "401"}" is a confident lie in the log of a step
|
|
# whose whole job is to be believed. Only act on real ids.
|
|
case "$stale" in ''|*[!0-9]*) continue ;; esac
|
|
echo "removing stranded probe draft $stale"
|
|
gh api -X DELETE "repos/${GH_REPO}/releases/${stale}" >/dev/null 2>&1 || true
|
|
done || true
|
|
|
|
# Retried, because a one-shot POST cannot tell a bad credential from
|
|
# a bad minute. GitHub answers 403 for secondary rate limiting as
|
|
# well as for refusal -- the publish step below retries six times for
|
|
# exactly that reason -- and these repos return 504 on ordinary
|
|
# release listings often enough to have been seen while writing this.
|
|
# Reporting a blip as "the token is wrong" would send whoever reads
|
|
# it to reissue a credential that was never the problem.
|
|
#
|
|
# Three attempts rather than the publish step's six: the value here is
|
|
# failing fast, and 5s + 10s of backoff still absorbs a transient
|
|
# while keeping a genuine misconfiguration well under a minute.
|
|
attempt=0
|
|
until probe_id=$(gh api -X POST "repos/${GH_REPO}/releases" \
|
|
-f tag_name="$PROBE_TAG" -f name="$PROBE_TAG" \
|
|
-F draft=true --jq .id); do
|
|
attempt=$((attempt + 1))
|
|
if [ "$attempt" -ge 3 ]; then
|
|
echo "::error::RELEASE_TOKEN could not create a release in"\
|
|
"${GH_REPO} after 3 attempts. If this is not a transient API"\
|
|
"error, the token needs Contents: read and write."
|
|
exit 1
|
|
fi
|
|
echo "attempt ${attempt} failed, retrying in $((attempt * 5))s"
|
|
sleep $((attempt * 5))
|
|
done
|
|
echo "release credential OK"
|
|
|
|
buildroot:
|
|
name: Firmware
|
|
needs: [preflight, select]
|
|
if: >-
|
|
needs.preflight.outputs.should_build == 'true' &&
|
|
needs.select.outputs.needs-build == 'true'
|
|
runs-on: ubuntu-latest
|
|
env:
|
|
BUILD_ID: ${{ needs.preflight.outputs.build_id }}
|
|
BUILD_SHA: ${{ needs.preflight.outputs.head_sha }}
|
|
BUILD_PLATFORM: ${{ matrix.platform }}
|
|
|
|
strategy:
|
|
fail-fast: false
|
|
# The device list lives in .github/scripts/ci-matrix.py, which is
|
|
# also what narrows it. Everything except pull_request gets all of it.
|
|
matrix: ${{ fromJson(needs.select.outputs.matrix) }}
|
|
steps:
|
|
- name: Checkout source
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Prepare firmware
|
|
run: |
|
|
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
|
|
echo CACHE_DATE=$(date +%m) >> ${GITHUB_ENV}
|
|
INPUT=${{inputs.platform}}
|
|
MATRIX=$(echo ${{matrix.platform}} | cut -d_ -f1)
|
|
if [ -z ${INPUT} ] || [ ${INPUT} = ${MATRIX} ]; then
|
|
echo RUN=${MATRIX} >> ${GITHUB_ENV}
|
|
fi
|
|
|
|
- name: Setup ccache
|
|
if: env.RUN
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: /tmp/ccache
|
|
key: ${{matrix.platform}}-${{env.CACHE_DATE}}
|
|
|
|
# builder.sh does `rm -rf openipc` at the top of every run to force a
|
|
# clean re-clone of OpenIPC/firmware, which also nukes the default
|
|
# `openipc/output/dl/` directory Buildroot uses for downloaded source
|
|
# tarballs. Without a cache, every nightly run re-downloads ~1-3 GB of
|
|
# tarballs from upstream mirrors — and any one of them randomly
|
|
# failing (e.g. cgit `snapshot/` URLs that regenerate tarballs per
|
|
# request with shifting sha256, like wireguard-tools on
|
|
# git.zx2c4.com) takes down the matrix entry for that board. See
|
|
# kaeru wireguard-zx2c4-tarball-regeneration-hash-drift for the
|
|
# specific incident this caching addresses.
|
|
#
|
|
# Cache to /tmp/builder-dl (outside the openipc/ tree so builder.sh's
|
|
# rm -rf doesn't touch it) and point Buildroot at it via BR2_DL_DIR
|
|
# env var (honored per Buildroot manual section "Location of source
|
|
# tarballs"). Mirrors OpenIPC/firmware/.github/workflows/build.yml's
|
|
# output/dl cache, adapted for builder's clean-clone behavior.
|
|
- name: Setup dl cache
|
|
if: env.RUN
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: /tmp/builder-dl
|
|
key: builder-dl-${{env.CACHE_DATE}}
|
|
restore-keys: builder-dl-
|
|
|
|
- name: Refresh moving-ref package downloads
|
|
if: env.RUN
|
|
run: |
|
|
# builder caches BR2_DL_DIR (/tmp/builder-dl) under a monthly key
|
|
# (date +%m) and actions/cache only saves on a key miss, so the first build
|
|
# of the month freezes it for weeks. Packages pinned to a moving ref keep a
|
|
# constant download filename, so buildroot reuses the stale tarball — even
|
|
# though builder re-clones firmware HEAD each run for freshness. Two forms:
|
|
# - dash: <pkg>-<ref>.tar.gz e.g. majestic-webui-dist.tar.gz
|
|
# - dot: <pkg>.<ref>.tar.bz2 e.g. majestic.t31.lite.master.tar.bz2
|
|
# The dot form is majestic's S3 tarball: it LOOKS S3-pinned but is a moving
|
|
# `master` build re-uploaded every majestic CI run, with no .hash to force a
|
|
# refetch — so it must be evicted too. (Empirically: a 06-14 nightly shipped a
|
|
# ~2-week-old majestic-webui; later a t31 lite image shipped a pre-x-groups
|
|
# majestic against the current webui — "No settings groups in schema".) Drop
|
|
# both forms so buildroot re-fetches the current ref. Only genuinely
|
|
# content-addressed tarballs (semver / SHA in the name) keep their cache.
|
|
find /tmp/builder-dl -type f -regextype posix-extended \
|
|
-regex '.*[-.](HEAD|master|main|dist)\.tar\.(gz|bz2|xz)' \
|
|
-print -delete 2>/dev/null || true
|
|
|
|
- name: Build firmware
|
|
if: env.RUN
|
|
run: |
|
|
export GIT_HASH=$(git rev-parse --short ${GITHUB_SHA})
|
|
export GIT_BRANCH=${GITHUB_REF_NAME}
|
|
echo GIT_HASH=${GIT_HASH} >> ${GITHUB_ENV}
|
|
echo GIT_BRANCH=${GIT_BRANCH} >> ${GITHUB_ENV}
|
|
|
|
mkdir -p /tmp/ccache
|
|
ln -s /tmp/ccache ${HOME}/.ccache
|
|
|
|
mkdir -p /tmp/builder-dl
|
|
export BR2_DL_DIR=/tmp/builder-dl
|
|
|
|
NAME=${{matrix.platform}}
|
|
# Retry budget for transient upstream toolchain/CDN flakes
|
|
# (matches OpenIPC/firmware/.github/workflows/build.yml).
|
|
backoffs="30 60 120 300 600 1200"
|
|
attempt=1
|
|
for sleep_for in $backoffs ""; do
|
|
bash builder.sh ${NAME} && break
|
|
if [ -z "$sleep_for" ]; then
|
|
echo "::error::build failed after ${attempt} attempts"
|
|
exit 1
|
|
fi
|
|
echo "::warning::attempt ${attempt} failed, retrying after ${sleep_for}s"
|
|
sleep "$sleep_for"
|
|
attempt=$((attempt + 1))
|
|
done
|
|
cd openipc
|
|
|
|
TIME=$(date -d @${SECONDS} +%M:%S)
|
|
echo TIME=${TIME} >> ${GITHUB_ENV}
|
|
COMMON=$(echo ${NAME} | awk -F_ '{print NF-1}')
|
|
|
|
NORFW=$(find output/images -name openipc*nor*)
|
|
if [ ! -z ${NORFW} ]; then
|
|
if [ ${COMMON} -eq 1 ]; then
|
|
echo NORFW=openipc/${NORFW} >> ${GITHUB_ENV}
|
|
else
|
|
mv ${NORFW} ../${NAME}-nor.tgz
|
|
echo NORFW=${NAME}-nor.tgz >> ${GITHUB_ENV}
|
|
fi
|
|
fi
|
|
|
|
NANDFW=$(find output/images -name openipc*nand*)
|
|
if [ ! -z ${NANDFW} ]; then
|
|
if [ ${COMMON} -eq 1 ]; then
|
|
echo NANDFW=openipc/${NANDFW} >> ${GITHUB_ENV}
|
|
else
|
|
mv ${NANDFW} ../${NAME}-nand.tgz
|
|
echo NANDFW=${NAME}-nand.tgz >> ${GITHUB_ENV}
|
|
fi
|
|
fi
|
|
|
|
if [ -z ${NORFW} ] && [ -z ${NANDFW} ]; then
|
|
exit 1
|
|
fi
|
|
|
|
- name: Build size report
|
|
if: env.NORFW || env.NANDFW
|
|
run: |
|
|
cd openipc
|
|
NAME=${{matrix.platform}}
|
|
make BOARD=${NAME} size-report || \
|
|
{ echo "::warning::size-report failed for ${NAME}"; exit 0; }
|
|
SIZES=$(find output/images -name 'sizes.*.json' | head -1)
|
|
if [ -z "${SIZES}" ]; then
|
|
exit 0
|
|
fi
|
|
# Mirror the .tgz rename trick: compound matrix entries
|
|
# (`<soc>_<variant>_<vendor>-<model>`) share `<soc>_<variant>` and
|
|
# would collide on the firmware-side `sizes.<soc>-<variant>.json`
|
|
# filename when uploaded to the same release tag. Rename to
|
|
# `sizes.<NAME>.json` so every compound device gets its own asset.
|
|
COMMON=$(echo ${NAME} | awk -F_ '{print NF-1}')
|
|
if [ ${COMMON} -eq 1 ]; then
|
|
echo "SIZES=openipc/${SIZES}" >> ${GITHUB_ENV}
|
|
else
|
|
mv ${SIZES} ../sizes.${NAME}.json
|
|
echo "SIZES=sizes.${NAME}.json" >> ${GITHUB_ENV}
|
|
fi
|
|
|
|
# Stage everything this device produced into one flat directory, so the
|
|
# artifact has a predictable shape no matter which of the two naming
|
|
# schemes the build used (compound devices rename to <device>-nor.tgz in
|
|
# the workspace root, simple ones leave openipc.<soc>-nor-<variant>.tgz
|
|
# under openipc/output/images/).
|
|
# Runs on pull_request too. Staging and the upload below are run-scoped
|
|
# and touch no secret and no shared state, so unlike `Send binary` and the
|
|
# `publish` job -- which stay gated, and must -- there is nothing here for
|
|
# a PR to leak or clobber. Without this a contributor's PR compiles a full
|
|
# image and then discards it, so they cannot test their own device profile
|
|
# without asking a maintainer to dispatch build-one for them.
|
|
- name: Stage artifacts
|
|
if: env.NORFW || env.NANDFW
|
|
run: |
|
|
mkdir -p dist
|
|
for f in "${NORFW:-}" "${NANDFW:-}" "${SIZES:-}"; do
|
|
# An `if` rather than a `&&` chain: the step runs under `bash -e`,
|
|
# where a trailing `&&` that tests false is only harmless because
|
|
# something after it resets the status.
|
|
if [ -n "$f" ] && [ -e "$f" ]; then
|
|
cp -a "$f" dist/
|
|
fi
|
|
done
|
|
ls -la dist
|
|
|
|
# Hand the images to the single `publish` job instead of writing releases
|
|
# from here. 107 matrix jobs each deleting and re-uploading assets on the
|
|
# SAME three shared releases is ~400 concurrent writes; OpenIPC/firmware
|
|
# ran exactly that design and the releases API answered HTTP 500 under the
|
|
# contention (run 27108181857: every board built, 14 jobs red in their
|
|
# upload step). Consolidating every release write into one job removes the
|
|
# concurrent-writer contention that causes it.
|
|
#
|
|
# `publish` carries its own `github.event_name != 'pull_request'`, so
|
|
# producing fw-* on a PR cannot reach a release; it only makes the image
|
|
# downloadable from the run that built it.
|
|
#
|
|
# Retention is per-event deliberately. A nightly's artifact is consumed by
|
|
# `publish` inside the same run and no one downloads it afterwards, and a
|
|
# nightly builds the entire device matrix -- holding those a week would
|
|
# keep seven full matrices in storage instead of one, for no reader. A PR
|
|
# builds a handful and has a person waiting on them, plausibly in another
|
|
# timezone, so one day is too short a window there.
|
|
- name: Upload build artifacts
|
|
if: env.NORFW || env.NANDFW
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: fw-${{ matrix.platform }}
|
|
if-no-files-found: ignore
|
|
retention-days: ${{ github.event_name == 'pull_request' && 7 || 1 }}
|
|
path: dist/*
|
|
|
|
- name: Send binary
|
|
if: github.event_name != 'pull_request' && env.NORFW
|
|
run: |
|
|
TG_MSG="Build: ${BUILD_ID}\nCommit: ${GIT_HASH}\nBranch: ${GIT_BRANCH}\nTime: ${TIME}\n\n"
|
|
TG_ICON="\xE2\x9C\x85 GitHub Actions"
|
|
TG_HEADER=$(echo -e ${TG_MSG}${TG_ICON})
|
|
TG_TOKEN=${{secrets.TELEGRAM_TOKEN_BOT_OPENIPC}}
|
|
TG_CHANNEL=${{secrets.TELEGRAM_CHANNEL_OPENIPC_DEV}}
|
|
HTTP=$(curl -s -o /dev/null -w %{http_code} https://api.telegram.org/bot${TG_TOKEN}/sendDocument -F chat_id=${TG_CHANNEL} -F caption="${TG_HEADER}" -F document=@${NORFW})
|
|
echo Telegram response: ${HTTP}
|
|
|
|
# All release writes happen here, once, so only a SINGLE job ever touches the
|
|
# shared `nightly`/`latest` releases and the per-run dated one. Best-effort by
|
|
# design: publish whatever the matrix produced even if some devices failed,
|
|
# which matches enrich_manifest.py -- it indexes whatever assets exist. Never
|
|
# runs on a pull request; there are no artifacts to publish there.
|
|
publish:
|
|
name: Publish releases
|
|
needs: [preflight, select, buildroot]
|
|
# The leading status function is load-bearing. A job whose `needs` failed is
|
|
# skipped by default, and an `if:` lifts that only when it CONTAINS a status
|
|
# function -- reading `needs.buildroot.result` inside the `contains()` below
|
|
# does not count. So since #121 gave this job the matrix as a dependency,
|
|
# any single failing device skipped it outright and the whole release went
|
|
# unwritten: 2026-08-24 lost every device's nightly to one failure out of 98
|
|
# (ssc333_lite_meari-speed-6s), and 2026-08-21 lost it to two out of 98.
|
|
# There is no nightly-20260821-* or nightly-20260824-* release as a result.
|
|
# Before #121 each device uploaded from its own matrix job, so a bad device
|
|
# took down only itself; this restores that without giving up the
|
|
# single-writer fix that stopped the concurrent-upload rate limits.
|
|
#
|
|
# `!cancelled()` rather than `always()`: the `contains()` already drops a
|
|
# cancelled or skipped matrix, but a run cancelled AFTER a green matrix
|
|
# would still start uploading under `always()`, and a half-written nightly
|
|
# is the thing this job exists to avoid.
|
|
if: >-
|
|
!cancelled() &&
|
|
github.event_name != 'pull_request' &&
|
|
needs.preflight.outputs.should_build == 'true' &&
|
|
contains(fromJSON('["success", "failure"]'), needs.buildroot.result)
|
|
runs-on: ubuntu-latest
|
|
permissions:
|
|
contents: write
|
|
steps:
|
|
# continue-on-error only so that a matrix where every device failed --
|
|
# and therefore uploaded no fw-* artifact at all -- does not hard-fail
|
|
# here before the gate can report why. Collect below turns that back into
|
|
# a failure for every case except that one, so a download that breaks can
|
|
# no longer publish nothing and call it success.
|
|
- name: Download build artifacts
|
|
id: download
|
|
uses: actions/download-artifact@v4
|
|
continue-on-error: true
|
|
with:
|
|
pattern: fw-*
|
|
path: dist
|
|
merge-multiple: true
|
|
|
|
- name: Collect assets
|
|
id: collect
|
|
run: |
|
|
mkdir -p dist
|
|
# tr because wc pads its output on some platforms (BSD does, the GNU
|
|
# coreutils on the runner does not) and a padded " 0" would read
|
|
# as non-empty to the `!= '0'` guard below.
|
|
count=$(find dist -type f | wc -l | tr -d '[:space:]')
|
|
echo "Collected ${count} asset(s):"
|
|
ls -la dist || true
|
|
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
|
|
|
# A matrix that went green must have produced artifacts. Zero assets
|
|
# after that means the download failed, not that there was nothing to
|
|
# publish -- and publishing nothing quietly is exactly how a nightly
|
|
# goes missing while the run stays green.
|
|
if [ "${{ needs.buildroot.result }}" = "success" ]; then
|
|
if [ "${{ steps.download.outcome }}" != "success" ]; then
|
|
echo "::error::artifact download failed after a fully successful matrix"
|
|
exit 1
|
|
fi
|
|
if [ "${count}" -eq 0 ]; then
|
|
echo "::error::no artifacts collected although every device built"
|
|
exit 1
|
|
fi
|
|
elif [ "${count}" -eq 0 ]; then
|
|
echo "::notice::matrix failed and produced nothing; no release to write"
|
|
else
|
|
# The partial case this job was always meant to serve, unreachable
|
|
# until the `!cancelled()` above. Say so out loud: the devices that
|
|
# did not build keep their PREVIOUS image on nightly/latest, which
|
|
# is the right outcome for a flasher but is invisible next to the
|
|
# freshly-written ones.
|
|
echo "::warning::partial publish: matrix failed, writing the ${count} asset(s) that did build; nightly and latest keep their previous image for every device that did not build — see the failed jobs in this run for which"
|
|
fi
|
|
|
|
# Drive the release writes with gh rather than softprops/action-gh-release,
|
|
# which uploads assets CONCURRENTLY with no throttle or retry knob. Firing
|
|
# a few hundred asset writes at three releases trips GitHub's per-actor
|
|
# secondary rate limit; here every asset goes up one at a time, paced under
|
|
# the per-minute mutation ceiling, with exponential backoff so a transient
|
|
# 403/429 is retried rather than failing the nightly.
|
|
- name: Publish releases (paced, retry-aware)
|
|
if: steps.collect.outputs.count != '0'
|
|
env:
|
|
# Not GITHUB_TOKEN. The default token's release-write depends on
|
|
# who triggered the run: a `schedule:` run executes as the repository's
|
|
# scheduling identity and could write releases, but a dispatched run
|
|
# executes as whoever dispatched it, and for an app/OAuth-backed
|
|
# identity the API answers "403 Resource not accessible by
|
|
# integration" even though the job declares `contents: write` and the
|
|
# run log confirms the token was granted it. OpenIPC/firmware hit
|
|
# exactly this on 2026-08-29 (its run 33273333359 built for two hours
|
|
# and then could not publish). This repo publishes the same way and
|
|
# was next in line. An explicit credential also makes publishing
|
|
# independent of who or what starts the run.
|
|
GH_TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
GH_REPO: ${{ github.repository }}
|
|
BUILD_ID: ${{ needs.preflight.outputs.build_id }}
|
|
HEAD_SHA: ${{ needs.preflight.outputs.head_sha }}
|
|
SHORT_SHA: ${{ needs.preflight.outputs.short_sha }}
|
|
BUILT_AT: ${{ needs.preflight.outputs.built_at }}
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
gh_retry() {
|
|
local n=0 max=6 delay=15
|
|
until "$@"; do
|
|
n=$((n + 1))
|
|
if [ "$n" -ge "$max" ]; then
|
|
echo "::error::gh failed after ${max} attempts: $*"
|
|
return 1
|
|
fi
|
|
echo "::warning::gh attempt ${n} failed, sleeping ${delay}s before retry"
|
|
sleep "$delay"
|
|
delay=$((delay * 2))
|
|
done
|
|
}
|
|
|
|
ensure_release() { # <tag> [extra gh release create args...]
|
|
local tag="$1"; shift
|
|
if ! gh release view "$tag" >/dev/null 2>&1; then
|
|
gh_retry gh release create "$tag" --target "$HEAD_SHA" --notes "$NOTES" "$@"
|
|
fi
|
|
}
|
|
|
|
# One asset at a time, paced ~1/s to stay under the per-minute
|
|
# content-mutation ceiling that triggers the secondary rate limit.
|
|
upload_paced() { # <tag> <file>...
|
|
local tag="$1"; shift
|
|
local total=$# i=0
|
|
for f in "$@"; do
|
|
i=$((i + 1))
|
|
echo "[${tag}] (${i}/${total}) ${f##*/}"
|
|
gh_retry gh release upload "$tag" "$f" --clobber
|
|
sleep 1
|
|
done
|
|
}
|
|
|
|
NOTES=$(printf 'sha=%s\nshort=%s\nbuilt_at=%s\n' "$HEAD_SHA" "$SHORT_SHA" "$BUILT_AT")
|
|
|
|
# Full asset set (images + size sidecars) -> the dated release, which
|
|
# is the only one enrich_manifest.py reads.
|
|
mapfile -t DATED < <(find dist -maxdepth 1 -type f | sort)
|
|
# nightly/latest are firmware-delivery aliases for flashers, so ship
|
|
# only the images there. Re-uploading the size sidecars to two more
|
|
# releases is pure rate-limit cost: the manifest never reads them from
|
|
# these tags, and nothing else does either.
|
|
mapfile -t IMAGES < <(find dist -maxdepth 1 -type f -name '*.tgz' | sort)
|
|
|
|
# --- dated: immutable per-build history ---
|
|
ensure_release "$BUILD_ID" --prerelease --title "$BUILD_ID"
|
|
upload_paced "$BUILD_ID" "${DATED[@]}"
|
|
|
|
# nightly and latest only ever move when there is firmware to move
|
|
# them to. dist/ can hold sidecars and no images -- one device that
|
|
# produced a size report and no .tgz is enough -- and pointing the two
|
|
# tags users flash from at a build with no images behind them is worse
|
|
# than leaving them on yesterday's.
|
|
if [ "${#IMAGES[@]}" -eq 0 ]; then
|
|
echo "::warning::no images in this build; leaving nightly and latest where they are"
|
|
else
|
|
# --- rolling nightly: move tag + body to this build, refresh images ---
|
|
ensure_release nightly
|
|
gh_retry gh release edit nightly --notes "$NOTES"
|
|
gh_retry gh api -X PATCH "repos/${GH_REPO}/git/refs/tags/nightly" -f sha="$HEAD_SHA" -F force=true
|
|
upload_paced nightly "${IMAGES[@]}"
|
|
|
|
# --- latest: legacy alias ---
|
|
ensure_release latest
|
|
gh_retry gh api -X PATCH "repos/${GH_REPO}/git/refs/tags/latest" -f sha="$HEAD_SHA" -F force=true
|
|
upload_paced latest "${IMAGES[@]}"
|
|
fi
|
|
|
|
# Single umbrella status check covering the whole run, so branch protection
|
|
# can require one context instead of a hardcoded "Build (<device>)" per
|
|
# device -- the matrix is now dynamic, and its job names change per PR.
|
|
# There was no gate here at all before: nothing reported a verdict on a
|
|
# change, because nothing built one.
|
|
ci-gate:
|
|
name: CI Gate
|
|
needs: [select, preflight, buildroot, publish]
|
|
# always() so the gate still reports when a needed job fails -- but not on
|
|
# a run the guard above deliberately skipped, where it would read
|
|
# select=skipped and turn a saved build into a red run.
|
|
if: >-
|
|
always() && (github.repository == 'OpenIPC/builder' ||
|
|
github.event_name == 'workflow_dispatch' ||
|
|
(github.event_name == 'pull_request' && !github.event.repository.private))
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Require selection + device matrix to succeed
|
|
run: |
|
|
echo "select=${{ needs.select.result }} preflight=${{ needs.preflight.result }} buildroot=${{ needs.buildroot.result }} publish=${{ needs.publish.result }}"
|
|
if [ "${{ needs.select.result }}" != "success" ]; then
|
|
echo "::error::device selection did not succeed"; exit 1
|
|
fi
|
|
if [ "${{ needs.preflight.result }}" != "success" ]; then
|
|
echo "::error::preflight did not succeed"; exit 1
|
|
fi
|
|
# Check the matrix against what select asked for. Blanket-accepting
|
|
# 'skipped' would mean a selector bug that narrowed a real change
|
|
# down to nothing reads as a green PR.
|
|
if [ "${{ needs.select.outputs.needs-build }}" = "true" ] && \
|
|
[ "${{ needs.preflight.outputs.should_build }}" = "true" ]; then
|
|
if [ "${{ needs.buildroot.result }}" != "success" ]; then
|
|
echo "::error::device matrix result=${{ needs.buildroot.result }}"; exit 1
|
|
fi
|
|
echo "device matrix OK (success)"
|
|
else
|
|
# Nothing in the change reaches a build, or the PR is still a draft.
|
|
if [ "${{ needs.buildroot.result }}" != "skipped" ]; then
|
|
echo "::error::device matrix ran when it should not have (${{ needs.buildroot.result }})"; exit 1
|
|
fi
|
|
echo "device matrix OK (no devices to build)"
|
|
fi
|
|
# publish is skipped on PRs and on a run that built nothing; only a
|
|
# real failure of the single release-writer should fail the gate.
|
|
case "${{ needs.publish.result }}" in
|
|
success|skipped) echo "publish OK (${{ needs.publish.result }})" ;;
|
|
*) echo "::error::publish result=${{ needs.publish.result }}"; exit 1 ;;
|
|
esac
|