* ci: build pull requests, and only the devices they reach
This repo had no pull_request trigger at all. A change landed unbuilt,
and the nightly found out the next morning -- or a user did. Turning PR
CI on meant 107 device builds per push, which is why it stayed off.
Almost every change here is one device: 639 of the file touches in the
last 200 commits sit under a single devices/<dir>/, and the median
commit reaches 2 devices. So narrowing first is what makes PR CI
affordable at all. .github/scripts/ci-matrix.py maps the paths a PR
touches to the devices that build them, and master.yml takes its matrix
from it.
The mapping is not written down, it is read off the tree the same way
builder.sh reads it. builder.sh line 121 locates a device by its
defconfig and copies that WHOLE devices/<dir>/ tree over the firmware
clone, so the devices a file affects are exactly the devices whose
defconfig shares its directory. That distinction is load-bearing:
devices/common/ backs 18 targets and devices/apfpv/ backs 2, and
treating a directory as one device would skip 17 real builds. A
defconfig still names one target even inside a shared directory.
THE NIGHTLY IS NOT NARROWED. builder.sh re-clones OpenIPC/firmware at
HEAD on every run, so what a nightly builds is decided mostly outside
this repo; master.yml already carries the scar of an earlier gate that
skipped on this repo's HEAD and left upstream fixes invisible to users.
Every event except pull_request gets the whole matrix, and --self-test
asserts that for schedule, workflow_dispatch and push.
Nothing on a pull_request may write a release. The three upload steps
were gated only on an image existing, so with a pull_request trigger
added they would have clobbered the published nightly and latest assets
that users flash. They and the Telegram post are now guarded on the
event.
master.yml and check-adjacent plumbing get a 15-device smoke set rather
than all 107: they decide how a build runs, not what it produces.
--self-test enforces the cover -- every vendor, architecture, toolchain
tuple and variant, plus both shared device directories -- rather than
the list, so the set can be re-picked while the property holds. It
already earned itself: the first run rejected my hand-picked set for
missing arm-openipc-linux-gnueabi, which exactly one target in the tree
has, and musleabihf.
Also adds the CI Gate umbrella check, since a dynamic matrix cannot be
named in branch protection and nothing reported a verdict before; and
freezes the 7 defconfigs that exist but are in no matrix, so a device
falling out of CI is a test failure rather than a silent skip.
Replaying the last 200 commits: 20330 device-builds if every PR built
everything, 5930 with the selector, 70.8% less, no coverage leaks. 43
of the 49 that still widen do so on paths that no longer exist
(devices/ssc338q_apfpv/ before it became devices/apfpv/, the old
top-level common/, a deleted builder-local.sh); restricted to commits
whose paths survive, it is 78.2%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: derive the device list instead of writing it down
Adding a camera IS the work in this repo: 8 of the 10 PRs open right now
add a device, and 14% of recent commits do. With the list written down
in ci-matrix.py, every one of those PRs also had to edit that file --
and a change to that file cannot narrow, because it is what does the
narrowing. So registering one camera cost a full 107-device build to
prove one device. The most common change got the worst outcome.
The matrix is now every defconfig under devices/, minus a NOT_BUILT
opt-out that keeps the 7 already in the tree but out of CI. Adding a
device is creating its directory, nothing else, and it builds exactly
that device.
This is the opposite call to OpenIPC/firmware's ci-matrix.py, on
purpose. There the matrix is curated -- 29 defconfigs are deliberately
never built and boards are not the product -- so an explicit list is the
honest description and an opt-out list would be noise. Here devices are
the product and the list is just "all of them".
The derived matrix is identical to the 107 master.yml carried: nothing
lost, nothing gained. --self-test now checks the opt-outs still name
real defconfigs, so a rename cannot leave a device silently building
under its new name while its old name sits in NOT_BUILT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci-matrix: own $GITHUB_OUTPUT instead of being redirected into it
The docstring said the default mode writes $GITHUB_OUTPUT; it printed
key=value lines to stdout and left the workflow to redirect them there.
Raised in review on #120.
Fixing the prose would have made the file honest, but the redirect
itself is the weaker half. Under `>> ${GITHUB_OUTPUT}` every print() in
this file is one keystroke away from corrupting the step outputs, and a
crash between the first line and the last leaves a half-written file
that Actions still reads. That the diagnostics go to stderr is what
keeps it safe today -- a rule that has to hold forever, rather than a
property of the code.
So the script appends the three lines itself when $GITHUB_OUTPUT is set,
and prints them when it is not, so a local run still shows its decision.
--stdin never touches the file: it is the mode for reading with your
eyes. master.yml drops the redirect, though leaving one in place would
have been harmless -- with the file written directly, stdout is empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A clone of this repo pushed to a new repository -- a private mirror, an
internal re-host -- inherits .github/workflows together with its cron.
The full platform matrix then runs on that owner's Actions bill, nightly
and unattended. Forks are only partially covered: GitHub disables
scheduled workflows there, but nothing covers a fresh repository, and
nothing warns the owner.
This mirrors OpenIPC/firmware#2255, where the same inherited cron was
measured at 97 jobs and 2,146 billed minutes from a single unattended
night, and never converged: publishing failed on the mirror, so the sha
skip-gate could never engage and the matrix rebuilt in full every night.
The rule: unattended runs happen only on the canonical repo. Anywhere
else a workflow runs only if someone explicitly asked for it
(workflow_dispatch), or if it is both free and attended (pull_request on
a public repo). This is deliberately stricter than gating purely on cost
-- a public re-host's minutes are free and its cron is still skipped,
because a re-host quietly publishing nightly releases under its own name
is a worse outcome than wasted minutes.
master carries the guard on preflight; buildroot needs it and inherits
the skip. manifest gates its cron-originated workflow_run path the same
way, because a nightly whose jobs all skipped still completes and fires
workflow_run. cleanup carries it on its weekly cron. build-one is
dispatch-only and needs nothing.
The pull_request clause is unreachable here today -- no workflow in this
repo has a pull_request trigger -- and is kept on purpose: it is the
clause that lets contributor PRs run, and omitting it would mean adding
a pull_request trigger later silently skips CI on every fork. That
over-block was the first thing reviewers caught upstream.
Upstream impact is none: on OpenIPC/builder the first disjunct is true,
so every added condition is a tautology and each job's if reduces to
what it was before.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds gk7205v200_otg_generic, a firmware profile that puts the single DWC3
controller into dual-role so the board can either consume a USB webcam
(existing usbcam path, host role) or present itself to a PC as one (UVC
gadget, device role). The roles are mutually exclusive -- xHCI HCSPARAMS1
reports MaxPorts = 1 -- but usb-mode switches between them at runtime, so
one image covers both and no reboot is needed.
Kernel side, three patches:
0015 sets dr_mode = "otg" in the DTS so dwc3 registers both an xHCI host
and a UDC.
0016 makes device mode actually work on a part that has only ever run as
a host. Three fixes: re-run dwc3_event_buffers_setup() on gadget start
(an xHCI reset wipes GEVNTADR), soft-reset the core before taking the
lock so the PHY is re-initialised (without it high-speed chirp fails),
and swallow the run_stop timeout on pullup(0), which otherwise
permanently poisons cdev->deactivations.
0017 disables the vendor's UVC_SG_REQ path. It hands a whole frame to the
UDC as one scatter-gather request; dwc3 chains those sg entries into a
single isochronous service interval, which on a full-speed link is one
1023-byte packet, so every frame larger than that reached the host
truncated and H.264 decoders reported "bytestream -5" on roughly half the
frames. Goke never shipped this path enabled -- CONFIG_USB_CONFIGFS_F_UVC
is unset in every vendor defconfig, including gk7605v100's, and the SDK
carries no userspace UVC sample -- so it was never exercised. The
mainline path (32 plain one-packet requests) is used instead.
Userspace side: usb-mode switches roles with the unbind/flip/bind
sequencing dwc3_set_mode() does not do on its own, uvc-gadget-setup
composes the configfs gadget, and S72usbmode forces a concrete role at
boot -- not optional, since dr_mode = "otg" leaves GCTL.PRTCAPDIR in a
state this 4.9 dwc3 never moves off by itself.
The descriptors advertise exactly the one format, frame size and interval
majestic is configured to serve, read from /etc/majestic.yaml. majestic
answers every PROBE/COMMIT with its own bFormatIndex/bFrameIndex because
it has one stream and cannot follow the host, so anything extra is an
invitation to select something that will not arrive: asking for MJPEG got
raw H.264 (an SPS where a JPEG SOI belongs, garbage to every consumer),
and asking for 1080p got 720p.
Profile drops the air-unit packages (wifibroadcast, rtl8812au, mavfwd,
mavlink-router, vtund, linux-firmware, wireless-tools) and the whole
wireless kernel stack, which pays for usb_f_uvc.ko and for keeping the
full sensor list -- the fpv trim to IMX307/IMX335 is keyed on
OPENIPC_VARIANT="fpv", and a general-purpose USB webcam should not be
sensor-locked.
Measured on a gk7205v200 lab board over a full-speed link: H.264
1280x720 at 30 fps with zero decoder errors and no dropped buffers, and
MJPEG delivering valid JPEGs at the jpeg channel's own rate.
Requires a majestic built with ENABLE_UVC_GADGET to feed the gadget;
that is a separate change.
PR #105 added a "Refresh moving-ref package downloads" step that drops
*-{HEAD,master,main,dist}.tar.gz before each build so buildroot re-fetches
the current ref. That fixed the majestic-webui-dist.tar.gz staleness (the
*-dist.tar.gz case), but it never matched majestic itself.
majestic downloads from S3 as majestic.<family>.<variant>.master.tar.bz2
(majestic.mk) — a DOT separator, .tar.bz2, and an S3 host the old pattern
deliberately spared as "content-addressed". But that S3 object is a moving
`master` build, re-uploaded on every majestic CI run, with no .hash to force
a refetch. So it stayed frozen in the monthly builder-dl cache while the
webui refreshed — the exact desync that shipped a pre-x-groups majestic in a
t31 lite image against the current webui, blanking the settings page with
"No settings groups in schema."
Broaden the eviction to a regex covering both -ref and .ref separators across
gz/bz2/xz, so majestic.*.master.tar.bz2 and majestic-webui-dist.tar.gz are
both dropped while semver/SHA-pinned tarballs keep their cache.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dahua-OEM 2 MP camera. The SC223A is on hardware I2C0 routed to the
VI-RAW pads (0x112c0060/0x64 func2) rather than the default mipi-sensor
pads (0x30/0x34 func1) the SDK probes, and needs a 27 MHz MCLK
(0x120100f0=0x19; SDK default 0x11/div4 leaves it unresponsive).
S71devmem re-muxes the pads, selects the SENSOR_RSTN reset (GPIO8_7
func2) and fixes MCLK after S70vendor/load_goke; the stock
open_sensor_i2c then reaches the sensor over /dev/i2c-0. No kernel
module needed.
Verified on hardware: chip-id 0xcb3e over HW I2C0, MIPI lock 1920x1080
zero errors, majestic streams correct colour 1080p.
Co-authored-by: build <build@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
builder caches BR2_DL_DIR (/tmp/builder-dl) under a monthly key (date +%m), and
actions/cache only saves on a key miss, so the first build of the month freezes it for
weeks. Packages pinned to a moving ref (VERSION = HEAD/branch, or the majestic-webui
`dist` release asset) download as a constant <pkg>-<ref>.tar.gz, so buildroot keeps
reusing the stale tarball — defeating the clean firmware-HEAD re-clone builder does for
freshness.
Empirically, a 2026-06-14 builder nightly shipped a ~2-week-old majestic-webui (full
bootstrap.min.css, no mj-settings.js) while the firmware nightly of the same day shipped
the current slim UI. Drop *-{HEAD,master,main,dist}.tar.gz before each build (master.yml
+ build-one.yml) so buildroot re-fetches the current ref. Mirrors the same fix in
OpenIPC/firmware.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Firmware PR #2166 added a `size-report` Makefile target + the
`general/scripts/size_report.py` analyser that emits a per-package /
per-kernel-module `sizes.<soc>-<variant>.json` next to the .tgz. Builder
clones firmware fresh on every run and shells to `make BOARD=...`, so the
script + target are already available downstream — this PR adds the CI wiring
and manifest extension so the JSON ships as a release asset and surfaces
through https://openipc.github.io/builder/manifest.json.
Three changes:
* `builder.sh`: best-effort `make size-report` after the main build, and
copy `sizes.*.json` into the timestamped archive next to `.tgz` /
`rootfs.squashfs.*`. Failure is swallowed so a legacy `OPENIPC_FW_REV` pin
to a pre-#2166 firmware ref can't sink the build.
* `.github/workflows/master.yml`: new `Build size report` step after the
existing `Build firmware`. Mirrors the bimodal rename trick the .tgz path
already does — compound matrix entries (`<soc>_<variant>_<vendor>-<model>`)
share `<soc>_<variant>` and would otherwise collide on the firmware-side
`sizes.<soc>-<variant>.json` filename when uploaded to the same release
tag. Compound entries get the JSON renamed to `sizes.<NAME>.json`; simple
entries keep `sizes.<soc>-<variant>.json`. `${{env.SIZES}}` is added to
all three upload `files:` blocks (dated / nightly / latest).
* `.github/scripts/enrich_manifest.py`: `SIZES_COMPOUND_RE` +
`SIZES_SIMPLE_RE`, a `parse_sizes()` helper that tries compound first
(because `[^._-]+` in the simple soc class would otherwise accept the
compound form too), the sidecar is recorded at the platform level under
a `"sizes"` key in `manifest.json`, and a new `@sizes <build_id>
<platform> <size> <url>` line type is emitted in `manifest.flat`
alongside the existing `@channel` directive.
Schema is unchanged for existing consumers — on-device sysupgrade reads
positional 4-column `manifest.flat` lines and stays inert on unknown
`@`-prefixed directives.
Co-authored-by: John <zekccsripfwrocp-15689@yn.furvionx.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The preflight gate skipped scheduled runs whenever this repo's HEAD
matched the SHA recorded in the previous nightly release body. That key
ignores every upstream input the build actually depends on:
OpenIPC/firmware HEAD, the toolchain release, the kernel tarball, the
vendor osdrv packages, and individual buildroot package upstreams.
builder.sh re-clones OpenIPC/firmware fresh on every run, so those
inputs do move between nightlies even when this repo is quiet.
The concrete failure mode this caused: on 2026-05-26 OpenIPC/firmware
bumped its hisilicon-opensdk pin to b958db4, which contains the V4
mipi_rx vsync IRQ fix (openhisilicon PR #195, closes#194 — torn
frames / 'Timeout from venc channel 0' on hi3516ev200, gk7205v200,
gk7205v210, gk7205v300, hi3516ev300 hardware). The next builder
nightly (2026-05-27 07:00 UTC) ran with this repo unchanged since
2026-05-25, so preflight skipped the entire Firmware matrix. Field
cameras (verified: gk7205v210_lite_vixand-ivg-g4h at IP redacted,
running BUILD_ID nightly-20260526-ee40da8, srcversion
8EF923DF907C1F7F3C6898E — the pre-fix open_mipi_rx fingerprint from
PR #195's verification table) kept pulling the stale 'latest' release
asset and kept showing the regression after sysupgrade.
A cross-input dedupe key would be possible (firmware HEAD + toolchain
URL hash + ...) but enumerates every upstream this build pulls. Simpler
and correct: scheduled nightlies always build. workflow_dispatch is
unaffected — that path was already unconditional.
Co-authored-by: Dmitry Ilyin <zekccsripfwrocp-15689@yn.furvionx.com>
OpenIPC/firmware/.github/workflows/build.yml has a `output/dl` cache
step that mirrors the Buildroot download directory across runs. This
saves both runner-minutes and — importantly — masks transient upstream
mirror failures: once any cron has cached the canonical tarball for a
package, every subsequent run reuses it regardless of whether the
primary URL is healthy that day.
builder's master.yml + build-one.yml had no equivalent. Each cron run
fully re-downloads the entire dl/ tree, so every package's upstream
must be healthy in every run. Hit hard 2026-05-24 by
wireguard-tools-1.0.20210914.tar.xz: it's served from a cgit
`snapshot/` URL on git.zx2c4.com which regenerates the tarball
per-request with shifting sha256, and `sources.buildroot.net` also
served drifted bytes — see kaeru
'wireguard-zx2c4-tarball-regeneration-hash-drift'. Result: random
matrix entries fail with hash mismatches each cron, with selection
roulette-style (93 boards enable wireguard; on any given run a few
roll snake-eyes on the download).
Mirror firmware's pattern with one wrinkle: builder.sh does
`rm -rf openipc` at the top of every run to force a clean re-clone of
firmware, which would obliterate `openipc/output/dl/` if we cached the
default path. Cache `/tmp/builder-dl` instead (outside the openipc
tree) and point Buildroot at it via BR2_DL_DIR env var, honored per
Buildroot manual section "Location of source tarballs". Same restore +
save lifecycle as firmware's, same monthly key shape.
Effect:
- First cron after merge: full download as usual; warms /tmp/builder-dl.
- Subsequent crons: instant cache hit for unchanged packages; only
re-download for version bumps. Removes the wireguard-roulette class
of failure entirely once cache is populated.
build-one.yml gets the same treatment so single-board dispatches share
the cache pool.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the OpenIPC/firmware nightly redesign (PRs #2111-#2129) to
builder. This commit is PR-Bld-A: the build.yml-side changes only.
Subsequent PRs add the manifest aggregator + retention sweep
(PR-Bld-B) and the build-one.yml commit input (PR-Bld-C).
Changes:
1. **Preflight SHA gate**. New `preflight` job reads `sha=` from the
rolling `nightly` release body. On schedule events it short-
circuits the matrix when HEAD matches the previously-published
SHA; PRs and workflow_dispatch always build. Mirrors firmware's
#2111.
2. **Three release tags per build**:
- `nightly-YYYYMMDD-<short>` (prerelease, content-addressable,
the unit of history — 90-day retention via PR-Bld-B's
cleanup.yml).
- `nightly` (rolling channel pointer; body carries sha/short/
built_at).
- `latest` (legacy alias, kept one cycle so existing sysupgrade
clients hitting the old hardcoded URL don't break).
3. **`BUILD_ID` / `BUILD_SHA` / `BUILD_PLATFORM` env** exported at
the job level from preflight outputs. firmware's rootfs_script.sh
(which builder.sh reuses via its clone of firmware) picks BUILD_ID
+ BUILD_SHA into /etc/os-release automatically. BUILD_PLATFORM is
set to the matrix entry (e.g. `gk7205v200_fpv_caddx-fly`) and is
forward-compatible with Phase 3 of the mirror plan (the firmware-
side rootfs_script.sh update that emits BUILD_PLATFORM as an
os-release field).
4. **Retry budget** around `bash builder.sh`. Matches firmware's
`30 60 120 300 600 1200` ladder, covering toolchain/CDN flakes
(#2115 + #2129 in firmware).
5. **Telegram caption** leads with `Build: <build_id>`; obsolete
`Tag: ${TAG_NAME}` line gone; workflow-level `env: TAG_NAME=
latest` removed.
No on-device firmware changes; builder cameras keep their existing
sysupgrade legacy URL until Phase 2 of the mirror (sysupgrade variant
routing) lands in OpenIPC/firmware.
See ~/.claude/plans/mirror-nightly-redesign-to-builder.md.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rationale:
End users might not know exactly what hardware they have, they just want to find RubyFPV firmware options, so in the release page, in the list of released firmware packages, it makes sense to have all RubyFPV related configs in a single place in the list (so all start with the same RubyFPV prefix).
