5e2d58f6442eb14850a6e87c3f12bcf9e209c9a2
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 OpenIPC AI teamandGitHub
|
6fef62c0f3 |
ci: watch for drift from firmware instead of finding it by accident (#129)
* ci: watch for drift from firmware instead of finding it by accident builder.sh copies devices/<item>/* over a fresh firmware clone, so a device directory that ships its own copy of a firmware file replaces it outright. A fleet-wide change in firmware reaches every device that only references a path and none that ships its own copy, silently, because nothing in either repository is looking. One comment on OpenIPC/firmware#2308 turned up three instances of that in an afternoon: 13 board configs that missed the CONFIG_VT sweep (#126), 97 defconfigs that kept BR2_PACKAGE_JSONFILTER after firmware retired it and put three devices over their rootfs cap (#128), and excludes lists naming files that no longer exist (firmware#2313 reports those at build time). The trigger is the design. Every one of those was caused by a commit in firmware while this repository sat untouched, so a pull_request check would never have fired for any of them -- nobody opens a builder PR when firmware changes. firmware-drift.yml runs on a schedule and files an issue rather than failing: drift someone else introduced is not a reason to redden a nightly that is otherwise fine. Only the self-test gates a merge here, out of lint.yml. Two checks, because the two kinds of drift are not the same shape. Shadowed files: comparing content is useless, a builder copy is *supposed* to differ. What matters is whether firmware's version moved since a human last reconciled them, so each of the 13 entries pins the blob it was reconciled against. Re-pinning is the act of looking again. The mapping is hand-authored -- nothing can infer that gk7205v200.generic-fpv derives from gk7205v200.generic. Defconfig symbols: nothing here copies a firmware defconfig, so pinning does not apply. Every BR2_PACKAGE_*=y is resolved against buildroot, firmware and this tree and sorted into "resolves nowhere" (a dead line kconfig ignores) and "resolves, but no firmware defconfig selects it" (needs an allowlist entry). An allowlist entry can also fence a symbol to the devices that need it; without that, writing JSONFILTER down as builder-only for devices/apfpv would equally bless it creeping back onto 95 unrelated defconfigs, which is #128 verbatim. Replayed against the three regressions, each reproduced by hand on a clean tree: firmware moves a shadowed config -> br-ext-chip-goke/board/gk7205v200/gk7205v200.generic.config moved in firmware since this copy was reconciled (2026-08-25). builder copy: devices/common/.../gk7205v200.generic-fpv.config pinned 7450c6803f6d, firmware now f97dce84c0f8 firmware commits since: 85233f0e kernel: drop CONFIG_VT and the busybox applets ... (#2308) jsonfilter creeps back onto a non-apfpv device -> BR2_PACKAGE_JSONFILTER is allowlisted only for apfpv/*/configs/*_defconfig, but 1 other defconfig(s) select it. firmware retires a symbol builder still selects -> BR2_PACKAGE_LIBUBOX is selected here but by no firmware defconfig. Resolving against buildroot is not optional and getting it wrong is how a checker lies. Buildroot is vendored by neither repository, so a first pass over OpenIPC packages alone called HOSTAPD, IW, PHP, UHTTPD, LIBZIP and BWM_NG dead; all six are upstream. Matching only "Config.in" then called PHP_EXT_ZIP dead, because buildroot declares php's extensions in package/php/Config.ext. Without --buildroot that half is skipped and says so rather than inventing findings. Two symbols really do resolve nowhere today -- BR2_PACKAGE_APFPV_GREG on the four greg-generic devices and BR2_PACKAGE_WIFIBROADCAST_EXT in devices/common. Both are recorded as known-dead and reported as a notice on every run rather than removed: whether those devices still want the packages is a maintainer's call, not a side effect of adding a check. The three new paths are classified in ci-matrix.py so this PR and every future re-pin cost no device builds. Without that a blob update is a 107-device run. * ci: discover shadowed files instead of listing them by hand Qodo caught the shadow list omitting the device-local t21 and t31 kernel configs. It was right, and the hole was much bigger than the five it named: a hand-written list of 13 board configs against 93 same-path shadows covering 35 firmware paths. The missing ones are not incidental -- load_goke, load_hisilicon, load_sigmastar and four vendor .mk files are all shadowed by some device, and those are exactly the shared files a fleet-wide fix lands in. firmware regression-tests load_hisilicon's os_mem_size derivation; two devices here ship their own copy of that script and would not have seen the fix. Listing them by hand was the wrong shape. builder.sh copies devices/<item>/* over the firmware clone, so any file at a path firmware also has replaces it -- which makes same-path shadows discoverable rather than something to remember. discover_shadows() walks devices/ and reports anything sitting on a firmware path with no entry in firmware-drift.json, so the list cannot silently rot and a device added tomorrow cannot quietly introduce an unreconciled shadow. The manifest goes from 13 entries to 101: the 88 discovered ones, plus the 13 hand-authored entries that discovery cannot find because the name differs from the file they replace -- gk7205v200.generic-fpv shadows gk7205v200.generic, and nothing can infer that. Two self-test cases for the new path: a same-path shadow with no entry is reported, and an entry silences it. Verified against the tree by dropping the load_hisilicon entries, which surfaces both devices immediately. |
||
|
OpenIPC AI teamandGitHub
|
7bfcab91ff |
ci: stop a NOT_BUILT device widening the matrix to all of them (#127)
NOT_BUILT is documented as the opt-out that narrows to nothing, and a change to
one of those devices' defconfigs does exactly that. Anything else in the same
directory did the opposite: targets_in() only ever returns devices CI builds, so
a directory holding nothing but opted-out devices came back empty and fell
through to the "is in no directory CI builds" widen. Touching a device CI
deliberately skips cost a full 107-device run.
$ echo devices/gk7205v200_rubyfpv_generic/br-ext-chip-goke/board/\
gk7205v200/gk7205v200.generic-fpv.config | python3 .github/scripts/ci-matrix.py --stdin
ci-matrix: 107/107 devices (needs_build=True) --- ... is in no directory CI builds
$ echo devices/gk7205v200_rubyfpv_generic/br-ext-chip-goke/configs/\
gk7205v200_rubyfpv_generic_defconfig | python3 .github/scripts/ci-matrix.py --stdin
ci-matrix: 0/107 devices (needs_build=False) --- nothing that reaches a build
Same device, same opt-out, opposite answers. #126 touches three NOT_BUILT
devices and is building all 107 because of it.
The empty result was carrying two meanings: "every device here is opted out" and
"this directory is not one I know". Only the second should widen. devices_in()
answers the second question -- it reads directory_of, which is built from every
defconfig in the tree rather than the built subset -- so an unrecognised
directory still widens, which is the half of the rule worth keeping.
Three self-test cases, one per meaning: a kernel config and an overlay file in a
NOT_BUILT device reach nothing, and a devices/ directory with no defconfig at
all still widens. With this, #126's own file list narrows from 107 to 23.
|
||
|
|
fc476c895d |
ci: only lint real steps, and find the real end of an expression (#124)
Two bugs in the linter #122 added, found by Qodo reviewing the port of it to OpenIPC/firmware#2290 and confirmed against this copy rather than taken on trust. walk() collected any mapping with a scalar `run` key, anywhere in the document. That is one key name away from linting things that are not shell: an action input, an env var or a matrix field called `run` holds arbitrary text, and feeding it to bash -n fails a workflow that is fine. On a fixture with one real step plus an `env: run:` and a `with: run:`, this copy collects three blocks. Steps now have to come out of a `steps:` sequence, which is the actual schema invariant and still not a hardcoded jobs.*.steps[*] path, so composite actions (`runs: steps:`) keep working. The expression substitution used `\$\{\{.*?\}\}`, which stops at the first `}}` even when it is inside a string literal. On x=${{ fromJSON('{"a": {"b": 1}}') }} this copy produces `x=__GHA_EXPR__') }}` and bash -n then reports an unbalanced quote -- a false failure on a valid block, in the direction that trains people to ignore the job. Replaced with a scanner that tracks GitHub's single-quoted strings, including the doubled '' escape, and returns the text untouched if an expression is never closed. Both are latent here: nothing in this repo has a `run` key outside a step or a `}}` inside an expression string, and the block count is unchanged at 23. They are fixed because the failure mode is a red job with nothing wrong with the tree, which is how a check stops being believed. Self-tests for both. The docstring described the behaviour this replaces, so it is corrected too, and the now-unused `re` import drops. Keeps this copy identical to firmware's apart from the incident paragraph and the block floor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
0455db0027 |
ci: close the if that cost last night's release, and parse-check the rest (#122)
The Collect guard added to the publish job in #121 shipped without its closing `fi`. Run 32096242353 -- the first nightly to reach the new single publish job -- built all 107 devices, staged 219 assets, and then died in `bash -e` with line 25: syntax error: unexpected end of file from `if' command on line 14 before writing anything. No nightly-20260818-* release exists, `nightly` and `latest` still carry the 17th's images, and ci-gate went red. The gate did its job; there was simply nothing left to publish by then. Nothing caught it earlier because the publish job carries `if: github.event_name != 'pull_request'` -- there are no artifacts to publish on a PR, so the entire path is unreachable from PR CI by construction. Both of #121's runs were green without ever executing the step. #121 said as much ("none of this path can run on a pull request") and checked the four states offline, but offline-checking the logic does not check that the script parses, and that is what broke. So the one-line fix, plus the check that would have caught it in seconds without a runner: lint-workflow-shell.py parses every `run:` block in .github/workflows/ with `bash -n`, and lint.yml runs it on PRs and on master pushes that touch either. Notes on the checker: - ${{ }} is not shell, so each expression is replaced with a plain word first. That is a real limitation -- an expression interpolating shell syntax is checked as the word, not as what it expands to -- and the substitution preserves line counts so reported lines still point at the right line. --self-test asserts an unterminated `if` containing an expression is still rejected while the closed form passes, because the way this breaks is by making everything pass. - Steps are found by walking for any mapping with a `run:` key rather than by the jobs.*.steps[*] path, so nothing depends on guessing where they live, and a discovery floor fails the run if the walk stops finding them. A checker that silently checks nothing looks exactly like a clean tree. - Syntax only. It says nothing about quoting or `-e` semantics; a shellcheck pass wants per-block disables for the substitution above and is a separate change. actionlint would cover more and is worth considering later; keeping this narrow is what makes it noise-free now. ci-matrix.py classifies both new files as no-build. They were names it had never heard of, and unknown widens by design, so the first push queued all 107 devices to prove a one-line YAML fix -- #121, which touched only master.yml, ran 19. This PR still took the full matrix because it edits ci-matrix.py itself, which always widens; the selector is not trusted to pick a smaller matrix for its own changes. PyYAML is installed explicitly rather than inherited from the runner image, and the workflow declares `contents: read` rather than taking the repo default. Both from review; neither would have failed the job today. Verified: the checker flags the merged master.yml with the same message the runner gave, passes the fixed tree, and all five guard states behave as #121 intended -- normal night publishes, a broken download and a green-but-empty dist both fail, and a dead matrix with nothing built still exits clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
edc0a0f88f |
ci: build pull requests, and only the devices they reach (#120)
* ci: build pull requests, and only the devices they reach This repo had no pull_request trigger at all. A change landed unbuilt, and the nightly found out the next morning -- or a user did. Turning PR CI on meant 107 device builds per push, which is why it stayed off. Almost every change here is one device: 639 of the file touches in the last 200 commits sit under a single devices/<dir>/, and the median commit reaches 2 devices. So narrowing first is what makes PR CI affordable at all. .github/scripts/ci-matrix.py maps the paths a PR touches to the devices that build them, and master.yml takes its matrix from it. The mapping is not written down, it is read off the tree the same way builder.sh reads it. builder.sh line 121 locates a device by its defconfig and copies that WHOLE devices/<dir>/ tree over the firmware clone, so the devices a file affects are exactly the devices whose defconfig shares its directory. That distinction is load-bearing: devices/common/ backs 18 targets and devices/apfpv/ backs 2, and treating a directory as one device would skip 17 real builds. A defconfig still names one target even inside a shared directory. THE NIGHTLY IS NOT NARROWED. builder.sh re-clones OpenIPC/firmware at HEAD on every run, so what a nightly builds is decided mostly outside this repo; master.yml already carries the scar of an earlier gate that skipped on this repo's HEAD and left upstream fixes invisible to users. Every event except pull_request gets the whole matrix, and --self-test asserts that for schedule, workflow_dispatch and push. Nothing on a pull_request may write a release. The three upload steps were gated only on an image existing, so with a pull_request trigger added they would have clobbered the published nightly and latest assets that users flash. They and the Telegram post are now guarded on the event. master.yml and check-adjacent plumbing get a 15-device smoke set rather than all 107: they decide how a build runs, not what it produces. --self-test enforces the cover -- every vendor, architecture, toolchain tuple and variant, plus both shared device directories -- rather than the list, so the set can be re-picked while the property holds. It already earned itself: the first run rejected my hand-picked set for missing arm-openipc-linux-gnueabi, which exactly one target in the tree has, and musleabihf. Also adds the CI Gate umbrella check, since a dynamic matrix cannot be named in branch protection and nothing reported a verdict before; and freezes the 7 defconfigs that exist but are in no matrix, so a device falling out of CI is a test failure rather than a silent skip. Replaying the last 200 commits: 20330 device-builds if every PR built everything, 5930 with the selector, 70.8% less, no coverage leaks. 43 of the 49 that still widen do so on paths that no longer exist (devices/ssc338q_apfpv/ before it became devices/apfpv/, the old top-level common/, a deleted builder-local.sh); restricted to commits whose paths survive, it is 78.2%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: derive the device list instead of writing it down Adding a camera IS the work in this repo: 8 of the 10 PRs open right now add a device, and 14% of recent commits do. With the list written down in ci-matrix.py, every one of those PRs also had to edit that file -- and a change to that file cannot narrow, because it is what does the narrowing. So registering one camera cost a full 107-device build to prove one device. The most common change got the worst outcome. The matrix is now every defconfig under devices/, minus a NOT_BUILT opt-out that keeps the 7 already in the tree but out of CI. Adding a device is creating its directory, nothing else, and it builds exactly that device. This is the opposite call to OpenIPC/firmware's ci-matrix.py, on purpose. There the matrix is curated -- 29 defconfigs are deliberately never built and boards are not the product -- so an explicit list is the honest description and an opt-out list would be noise. Here devices are the product and the list is just "all of them". The derived matrix is identical to the 107 master.yml carried: nothing lost, nothing gained. --self-test now checks the opt-outs still name real defconfigs, so a rename cannot leave a device silently building under its new name while its old name sits in NOT_BUILT. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci-matrix: own $GITHUB_OUTPUT instead of being redirected into it The docstring said the default mode writes $GITHUB_OUTPUT; it printed key=value lines to stdout and left the workflow to redirect them there. Raised in review on #120. Fixing the prose would have made the file honest, but the redirect itself is the weaker half. Under `>> ${GITHUB_OUTPUT}` every print() in this file is one keystroke away from corrupting the step outputs, and a crash between the first line and the last leaves a half-written file that Actions still reads. That the diagnostics go to stderr is what keeps it safe today -- a rule that has to hold forever, rather than a property of the code. So the script appends the three lines itself when $GITHUB_OUTPUT is set, and prints them when it is not, so a local run still shows its decision. --stdin never touches the file: it is the mode for reading with your eyes. master.yml drops the redirect, though leaving one in place would have been harmless -- with the file written directly, stdout is empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
21de4f7b92 |
ci: drop sizes.json URLs from manifest.json (CORS-blocked in browsers) (#103)
Mirror of OpenIPC/firmware#2169. PR #102 extended enrich_manifest.py with SIZES_COMPOUND_RE + SIZES_SIMPLE_RE + parse_sizes() so the public manifest carried `platforms[<plat>].sizes.url` and `@sizes` lines pointing at each release's sizes.json asset. The intent was a browser-side consumer; the GitHub release-asset URLs return no Access-Control-Allow-Origin header, so every browser fetch was blocked by CORS. This commit reverts only the manifest-indexing half of #102. The release-asset upload itself stays in master.yml — server-side consumers can still read the JSON via `gh release download` or any CI workflow. Only the in-manifest URL is removed, because that URL only made sense for an in-browser reader that turns out can't reach it. The resulting enrich_manifest.py is bit-for-bit identical to the pre-#102 state. Co-authored-by: John <zekccsripfwrocp-15689@yn.furvionx.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
da13271f94 |
ci: mirror firmware's size-report sidecar (OpenIPC/firmware#2166) (#102)
Firmware PR #2166 added a `size-report` Makefile target + the `general/scripts/size_report.py` analyser that emits a per-package / per-kernel-module `sizes.<soc>-<variant>.json` next to the .tgz. Builder clones firmware fresh on every run and shells to `make BOARD=...`, so the script + target are already available downstream — this PR adds the CI wiring and manifest extension so the JSON ships as a release asset and surfaces through https://openipc.github.io/builder/manifest.json. Three changes: * `builder.sh`: best-effort `make size-report` after the main build, and copy `sizes.*.json` into the timestamped archive next to `.tgz` / `rootfs.squashfs.*`. Failure is swallowed so a legacy `OPENIPC_FW_REV` pin to a pre-#2166 firmware ref can't sink the build. * `.github/workflows/master.yml`: new `Build size report` step after the existing `Build firmware`. Mirrors the bimodal rename trick the .tgz path already does — compound matrix entries (`<soc>_<variant>_<vendor>-<model>`) share `<soc>_<variant>` and would otherwise collide on the firmware-side `sizes.<soc>-<variant>.json` filename when uploaded to the same release tag. Compound entries get the JSON renamed to `sizes.<NAME>.json`; simple entries keep `sizes.<soc>-<variant>.json`. `${{env.SIZES}}` is added to all three upload `files:` blocks (dated / nightly / latest). * `.github/scripts/enrich_manifest.py`: `SIZES_COMPOUND_RE` + `SIZES_SIMPLE_RE`, a `parse_sizes()` helper that tries compound first (because `[^._-]+` in the simple soc class would otherwise accept the compound form too), the sidecar is recorded at the platform level under a `"sizes"` key in `manifest.json`, and a new `@sizes <build_id> <platform> <size> <url>` line type is emitted in `manifest.flat` alongside the existing `@channel` directive. Schema is unchanged for existing consumers — on-device sysupgrade reads positional 4-column `manifest.flat` lines and stays inert on unknown `@`-prefixed directives. Co-authored-by: John <zekccsripfwrocp-15689@yn.furvionx.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
63994266e2 |
ci/nightly: manifest aggregator + 90-build retention sweep (#92)
PR-Bld-B in the mirror of OpenIPC/firmware's nightly redesign. Adds
the two workflows that turn dated nightly-* releases (from PR-Bld-A)
into a queryable index served via GitHub Pages.
`manifest.yml` — triggers on successful `Build` workflow_run completion
(or manual dispatch). Runs `.github/scripts/enrich_manifest.py` to
enumerate dated releases, keep the 90 newest, parse the body
(sha/short/built_at from PR-Bld-A) and asset list, then emit:
- `manifest.json` — rich JSON for hosts/agents/CI.
- `manifest.flat` — whitespace-delimited columns
`build_id platform flash size url` plus `@channel` records,
parseable by pure busybox `awk` (no jq, no jsonfilter) for
on-device sysupgrade in Phase 2.
Both committed to the `gh-pages` branch (already bootstrapped).
`cleanup.yml` — Mondays 06:00 UTC (offset by +1h from firmware's
05:00 UTC cleanup). Deletes dated nightly releases beyond the 90
newest via `gh release delete --cleanup-tag`, then re-triggers
manifest.yml.
`enrich_manifest.py` differs from firmware's only in the asset parser:
builder produces TWO filename forms (per `master.yml`'s COMMON awk
trick that counts underscores in the matrix entry name):
- **Compound** `<soc>_<variant>_<vendor>-<model>-nor.tgz` →
platform key = the full string. This is the common case
(per-device builds).
- **Simple** `openipc.<soc>-<flash>-<variant>.tgz` (firmware-style,
when matrix name has one underscore) → platform key =
`<soc>_<variant>`.
The platform key gets carried into manifest.flat as a single column.
sysupgrade's existing `awk '$2==p'` lookup works unchanged; Phase 3
(model-aware BUILD_PLATFORM field in os-release) will let cameras
filter against the full per-model key.
URLs after merge:
https://openipc.github.io/builder/manifest.json
https://openipc.github.io/builder/manifest.flat
(`gh-pages` branch + Pages config already set up; mirrors firmware-
side setup.)
Concurrency-grouped `gh-pages-manifest` so manifest.yml and
cleanup.yml can't race the gh-pages push.
Includes the retry hardening from firmware OpenIPC/firmware#2129
(4-attempt backoff for transient GitHub API 401/5xx, fast-fail on
permanent 404).
First-run behaviour validated locally: empty index emits explicit
placeholder. Parser unit-tested across 7 synthetic filenames covering
compound, simple, and edge cases.
See ~/.claude/plans/mirror-nightly-redesign-to-builder.md.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|