summaryrefslogtreecommitdiffstats
path: root/meta/classes-global
Commit message (Collapse)AuthorAgeFilesLines
* sstate: apply proper umask when fetching from SSTATE_MIRRORRasmus Villemoes2025-07-041-10/+12
| | | | | | | | | | | | | | | | | | Currently, files and directories created under ${SSTATE_DIR} when fetching from an sstate mirror are not created with group write, unlike when the sstate artifacts are generated locally. That's inconsistent, and problematic when the local sstate dir is shared among multiple users. Wrap the fetching in a bb.utils.umask() context manager, and for simplicity move the mkdir of SSTATE_DIR inside that. (From OE-Core rev: e56aa6c62e41667d0eeec5a862b7004d21b4da48) Signed-off-by: Rasmus Villemoes <ravi@prevas.dk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a6038553aaef3b88b834a09018c524c4fa41e625) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* package_rpm.bbclass: Remove empty build directoryRobert Yang2025-07-041-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | Fixed: $ bitbake acl $ ls tmp/work/core2-64-poky-linux/acl/2.3.2/build/acl-2.3.2-build/ This empty directory is created by rpmbuild, just remove it like others above. Note, master branch doesn't have this problem after upgrade to rpm 4.20.1, but it's not easy to find which patch(es) fixed the problem, and the issues haven't been fixed completely, there is still an empty SRPMS directory, so just remove the build directory as others above. If we don't fix this, there might be warnings when a recipe uses: install -m 0644 ${B}/* ${DEPLOYDIR}/boot.bin-extracted/ install: omitting directory '${B}/xilinx-bootbin-1.0-build' (From OE-Core rev: c4f95abacdb7376f8f299679f3f0b2dd1dc28c27) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* insane.bbclass: Report all invalid PACKAGECONFIGs for a recipe at oncePeter Kjellerstedt2025-05-141-4/+4
| | | | | | | | | | Rather than reporting each invalid PACKAGECONFIG with a separate error message, report them all with one error message. (From OE-Core rev: c311d5ddc102f26ac1813c540520175cfed3a5a6) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* insane.bbclass: Move test for invalid PACKAGECONFIGs to do_recipe_qaPeter Kjellerstedt2025-05-141-10/+10
| | | | | | | | | | This makes sure invalid PACKAGECONFIGs are reported also for recipes that have no do_configure task, e.g., packagegroups. (From OE-Core rev: ed0e8f5142bc9c3a5c2fca7068135b2dc11ec3db) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* sanity: Check if the C++ toolchain supports --std=gnu++20Yoann Congal2025-03-201-0/+5
| | | | | | | | | | | | This is needed to build nodejs from meta-oe. Check this early to avoid an error later in the build. Fixes [YOCTO #15804] (From OE-Core rev: 19004950ad5691ef51f7fcf946558bd0de561403) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity: Refactor "C++ toolchain support this flag" checkYoann Congal2025-03-201-5/+5
| | | | | | | | | This can be use to check for other flags. (From OE-Core rev: 42d012e8414736d89618422fb6f9bce9e4a2a391) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity: Bump minimum python version requirement to 3.9Richard Purdie2025-03-201-3/+3
| | | | | | | | | | | | | | | | Since Ubuntu 20.04 is heading to EoL and that we're having other issues on that platform needing buildtools anyway, we're about to lose the last python 3.8 platform we were supporting. Bump the minimum version to 3.9 since there are many developers wanting access to newer python features. This likely won't be seen since the bitbake minimum has also been increased but helps for documentation purposes. (From OE-Core rev: c3db241c6e97420ca0705d1535a7cdd0c273b761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* mirrors: rationalise Debian mirrorsRoss Burton2025-03-061-17/+1
| | | | | | | | | | | | | | | Change DEBIAN_MIRROR to point at the canonical server, deb.debian.org. This is a CDN-backed server using network magic to load balance across the planet so there's no need to set a slew of regional mirrors. Also add a more recent snapshot.debian.org from the beginning of 2025. (From OE-Core rev: 3d95d45836accd29916dd8cb9bfe624d63d6c202) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats: Avoid rare UnboundLocalErrorRichard Purdie2025-03-031-6/+9
| | | | | | | | | | | | In rare cases BUILDNAME can seemingly be None outside of heartbeat events which leads to UnboundLocalErrors as bsdir and taskdir aren't defined. Skip the code in these cases rather than generate tracebacks which cause bitbake server to exit. (From OE-Core rev: 0f74d804ba0daf7e8bd6481597740b9d89821414) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/insane: do not leak host uid/gid into package_qa sstate signaturesAlexander Kanavin2025-02-281-0/+1
| | | | | | | | | | | | | This prevented package_qa sstate from being reusable unless host uid/gid values would match exactly (and they unfortunately do on the yocto autobuilder worker machines which all share a 'pokybuild' user). I noticed this when testing CDN sstate reuse, which otherwise works well. (From OE-Core rev: 6ea8b4b10b0549c858427a8411bf2a4cd5c0eb7b) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity: Check for non ascii chars in TOPDIRSofiane HAMAM2025-02-201-0/+9
| | | | | | | | | | | | | | Some modules (like Perl's MakeMaker) do not support non ASCII characters in build folder's path, this would cause build failures of software that does not support non ASCII. A sanity check is added to warn the user. Fixes [YOCTO #15764] (From OE-Core rev: 2b3be97a0d0d60d026786a4465b24b6f6752ba32) Signed-off-by: Sofiane HAMAM <sofiane.hamam@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base: Fix PACKAGECONFIG handling for cross recipesRichard Purdie2025-02-181-1/+1
| | | | | | | | | | We don't have many cross recipes that use PACKAGECONFIG but gdb-cross does, so correctly remap dependencies for that case allowing the gdb recipe to be simplified. (From OE-Core rev: c5bb7976f4d6e8559b4b87e4c3f39135dbb40ef8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_ipk: Use preferred form of --force-postinstallRichard Purdie2025-02-111-1/+1
| | | | | | | | | | opkg accepts both - and _ in its options but use the help text preferred format. From https://github.com/yoctoproject/poky/pull/14 from Philipp-Alexander Blum <blum@gessler.de>. (From OE-Core rev: 4bea64347407dfb1eb46dacffd05af3bd200b419) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: switch p7zip to 7zipPeter Marko2025-02-101-2/+2
| | | | | | | | | | | | meta-oe has switched from p7zip to 7zip. p7zip recipe does not exist anymore and p7zip is provided and rprovided by 7zip recipe. Use real provider instead of replaced one. (From OE-Core rev: 5aa516bfa295d5be919459dfe45f452cdec45e81) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/meta-selftest: Fix variable assignment whitespaceRichard Purdie2025-02-011-1/+1
| | | | | | | | | | Recipes are much more readable with whitespace around the assignment operators. Fix various assignments in OE-Core to show this is definitely the preferred formatting. (From OE-Core rev: 30ea609d3357fb3de911f2f6a5e6856c151b976a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity: Add test for functional c++ toolchainChristos Gavros2025-01-271-0/+25
| | | | | | | | | | | | Users reported issues caused by missing the right libstdc++-version-dev. A new function 'check_cpp_toolchain' added in sanity.bbclass to test linking libstdc++ [YOCTO #15712] (From OE-Core rev: 611c1a26212dfbfe8d0640d9fefe5df49f7b69b8) Signed-off-by: Christos Gavros <gavrosc@yahoo.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/recipes: Switch virtual/XXX-gcc to virtual/cross-cc (and c++/binutils)Richard Purdie2025-01-212-13/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea of the base class dependency is to say "yes, I need a C cross compiler" and this was never meant to be gcc specific. Looking at the codebase, whilst we code triplets into this, it does overcomplicate things as there are only ever limited, "target", "sdk" and the class extended versions like mutlilib. After much thought, we can simplify this to virtual/cross-cc and virtual/nativesdk-cross-cc. This lets us remove the "gcc" specific element as well as removing the over complicated triplet usage. At the same time, change the much less widely used "g++" variant to "c++" for similar reasons and remove the triplet from virtual/XXX-binutils too. Backwards compatibility mappings could be left but are just going to confuse things in future so we'll just require users to update. This simplification, whilst disruptive for any toolchain focused layers, will make improved toolchain selection in the future much easier. Since we no longer have overlapping variables, some code for that can just be removed. The class extension code does need to start remapping some variables but not the crosssdk target recipe names. This patch is in two pieces, this one handles the renaming with the functional changes separate in a second for easier review even if this breaks bisection. (From OE-Core rev: 4ccc3bc8266c327bcc18c9a3faf7536210dfb9f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* kernel-yocto: make kernel commits reproducibleEnrico Jörns2025-01-211-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The git commit hashes for the kernel checkout are not reproducible under certain conditions: - If the git repository is initialized on an archive (rather than a git), the initial git commit not only has the current user name set, it also uses the current system time as committer and author date. This will affect the initial git hash and thus all subsequent ones. - The patches applied by the kern-tools have a valid author and date. However, their committer again depends on the user building the BSP. This is an issue, for example, if one compiles a kernel with CONFIG_LOCALVERSION_AUTO enabled where the commit hash lands into the kernel and thus the package version. This not only makes the package version non-reproducible, but also leads to version mismatches between kernel modules built against a fresh kernel checkout and the kernel retrieved from the sstate cache. The class uses 'check_git_config' from utils.bbclass, but this only sets the git user and only if none existed before. Thus it doesn't really help here. Since in Git the committer information can be set only from the environment variables GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, and GIT_COMMITTER_DATE, we introduce a helper function to set those and apply the author settings in the same way. As values simply use PATCH_GIT_USER_NAME, PATCH_GIT_USER_EMAIL (from patch.bbclass) and SOURCE_DATE_EPOCH. For convenience, put the new helper 'reproducible_git_committer_author' into utils.bbclass next to 'check_git_config' so others can use it, too. Using this helper in kernel-yocto.bbclass makes the committer and author date/name/email for the initial commit reproducible, as well as the committer name/email for the patches applied with kern-tools. For debugging purpose, allow disabling the reproducibility features by setting KERNEL_DEBUG_TIMESTAMPS to "1". Suggested-by: Felix Klöckner <F.Kloeckner@weinmann-emt.de> (From OE-Core rev: aab4517b4649917abd519ea85a20fd9d51bf3d99) Signed-off-by: Enrico Jörns <ejo@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/logging: Added the bbverbnote command to the logging setBastien JAUNY2025-01-201-0/+10
| | | | | | | | | | | Fixes [YOCTO #15688] Reviewed-by: Yoann Congal <yoann.congal@smile.fr> (From OE-Core rev: 48e4b8c06e503868404b99bf45a46a52794baa5a) Signed-off-by: Bastien JAUNY <bastien.jauny@smile.fr> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Ensure errors cause task failureRichard Purdie2025-01-081-0/+4
| | | | | | (From OE-Core rev: c1491ef7afc8722927c568b0ac0ddbbc84837ca3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_rpm: set a timestamp handling policy with the new macro ↵Bin Lan2025-01-041-1/+1
| | | | | | | | | | | | | | | | %build_mtime_policy There is the following warning when the do_package_write_rpm task is executed when bitbake linux-yocto: warning: %clamp_mtime_to_source_date_epoch is deprecated, please use %build_mtime_policy After RPM is upgraded to 4.20.0, the %clamp_mtime_to_source_date_epoch macro is now deprecated in favor of %build_mtime_policy set to clamp_to_source_date_epoch. (From OE-Core rev: fa9b9e599bbf3ac73c7a9b1d15ccb0e0bbbe4c11) Signed-off-by: Bin Lan <bin.lan.cn@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity.bbclass: Lower the length of TMPDIR limit to 400Changqing Li2025-01-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When TMPDIR has length 410, bitbake uninative-tarball will fail at do_populate_sdk with error: Failed to open backup database: "TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite" sqlite3 has a hardcode max pathname limit 512, refer [1], and there is a check, refer [2], pathname + 8 > max path name limit(512) is not allowed. So the pathname length need to <= 504. While, length of "TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite" is 505, so error happened. We had met similar issue [3], and refer the comments, upstream reject the change of the hardcode length, see [4][5]. This patch try to improve and workaround this issue by lower max TMPDIR length to 400, at least for distro name length <=13, the build can success. [1] https://github.com/sqlite/sqlite/blob/master/src/os_unix.c#L180 [2] https://github.com/sqlite/sqlite/blob/master/src/pager.c#L4777 [3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=12374 [4] https://sqlite.org/cgi/src/tktview/c060923a5422590b3734 [5] https://sqlite.org/forum/forumpost/0b1b8b5116 (From OE-Core rev: da1c0998bdaf5def7ae94f44a33b9a505edc99c1) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base/features_check: Make PARSE_ALL_RECIPES and SOURCE_MIRROR_FETCH booleanRobert Yang2024-12-231-3/+4
| | | | | | | | | | So that value "0" can turn them off. (From OE-Core rev: 1a2afcd0773c5ec5946d1dfc27bde585c52a5724) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/utility-tasks: port do_listtasks to use bb.build.listtasksRoss Burton2024-12-121-12/+11
| | | | | | | | | | | Instead of iterating every datastore value by hand to find tasks, use the new bb.build.listtasks() function (bitbake 185c4b) (From OE-Core rev: 466c3ed0d01bc70caa29d5eb8bb99f7d0e6e710c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rpm: update 4.19.1.1 -> 4.20.0Alexander Kanavin2024-12-121-1/+1
| | | | | | | | | | | | | | | | | | | | | With this release, as expected, deprecated openpgp support has been fully removed and replaced with sequoia: https://github.com/rpm-software-management/rpm-sequoia/ Alas, it's written in rust, and has no recipe, so for now signing rpm packages has to be disabled. Remove package name parameter to %prep in source archiving, as it's been dropped in the new rpm, and wasn't needed to begin with. Drop 0001-perl-disable-auto-reqs.patch (files removed upstream). (From OE-Core rev: 8c15b4577d5e554cc2dd5adfb88b816894b05a9a) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base.bbclass: fix RRECOMMENDS extend issue for multilibChangqing Li2024-12-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During handle PACKAGECONFIG, DEPENDS/RDEPENDS/RRECOMMENDS is appended by calling function appendVar according to the PACKAGECONFIG setting: PACKAGECONFIG[f1] = "\ --with-f1, \ --without-f1, \ build-deps-for-f1, \ runtime-deps-for-f1, \ runtime-recommends-for-f1, \ packageconfig-conflicts-for-f1" [snip of base.bbclass] pkgconfigflags = d.getVarFlags("PACKAGECONFIG") or {} if pkgconfigflags: ... appendVar('DEPENDS', extradeps) appendVar('RDEPENDS:${PN}', extrardeps) appendVar('RRECOMMENDS:${PN}', extrarrecs) ... [snip of base.bbclass] After following commit in meta-oe, [fe3ba7ac5c cryptsetup: demote udev runtime dependency to recommendation] 'udev lvm2' changed from RDEPENDS to RRECOMMENDS, but RRECOMMENDS is not handled in this process, and causes following error: $bitbake lib32-cryptsetup ERROR: QA Issue: lib32-cryptsetup package lib32-cryptsetup - suspicious values 'udev lvm2' in RRECOMMENDS [multilib] In layer openembedded-core, we do have very few recipes that have runtime-recommends-for-f1 in PACKAGECONFIG setting, but f1 is not enabled in PACKAGECONFIGS, so not report error when multilib is enabled. (From OE-Core rev: 2f539a1318a80d3ad5d0d43a9ba4070465e402af) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes-global/insane: Handle case where RPROVIDER is also a providerJoshua Watt2024-12-121-8/+13
| | | | | | | | | | | | | | | | | | | The check to see if a provider of a given package is listed first checks for an exact match of the provider name. However, if this match existed, but didn't match in the task dependencies, it would not continue to look for other providers of package. This would manifest if one (non-virtual) recipe package RPROVIDES the name of a package produced by another recipe. Fix this, and also clean up the code to make it more readable by using a function to check if a runtime dependency is in the task dependencies. In addition, if no provider is found, list all the possible providers instead of the last one that was looked at. (From OE-Core rev: f13de6ab616eb1e38960a2296111febe2a9f4a28) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib: move buildstats.py and rootfspostcommands.py into oeChen Qi2024-12-051-2/+2
| | | | | | | | | | | These two files are the only ones that are left in meta/lib. They logically belong to meta/lib/oe, so move them there. (From OE-Core rev: c65dd0e3e463d6072b9364ac74e1fef0d998068f) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* yocto-check-layer: expand to cover all required QA checksDenys Dmytriyenko2024-12-031-0/+28
| | | | | | | | | | | | | | insane.bbclass now defines CHECKLAYER_REQUIRED_TESTS list with required QA checks that are becoming mandatory for Yocto Project Compatible layers. Update yocto-check-layer.bbclass in order to catch when packages from such layers try to skip any of the required QA checks. (From OE-Core rev: 9c3ba88628853b20fb4c98c99cf3fe8349024016) Signed-off-by: Denys Dmytriyenko <denys@konsulko.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: Split ERROR_QA into CHECKLAYER_REQUIRED_TESTSRichard Purdie2024-11-281-19/+18
| | | | | | | | | | | | | | | We'd like to start requiring some set of checks are enabled for Yocto Project Compatible Status and to pass yocto-check-layer. Start by splitting ERROR_QA into two sets, the ones we think can be required and the ones we know have challenges to implement (e.g. with prebuilt binaries). To change the required list, the YP TSC would need to approve. (From OE-Core rev: ce2e42ace2d15fb6745437cf0a7f07d28398ca12) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_rpm: restrict rpm to 4 threadsAlexander Kanavin2024-11-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | TL;DR version: with this, and the previous compression level changes I am seeing drastic speedups in package_write_rpm completion times: webkitgtk goes from 78 seconds to 37 seconds glibc-locale goes from 399 seconds to 58 seconds (!) The long version: rpm uses multithreading for two purposes: - spawning compressors (which are nowadays themselves multi-threaded, so the feature is not as useful as it once was) - parallel file classification While the former behaves well on massively parallel CPUs (it was written and verified here :), the latter was then added by upstream and only benchmarked on their very old, slow laptop, apparently: https://github.com/rpm-software-management/rpm/commit/41f0e214f2266f02d6185ba11f797716de8125d4 On anything more capable it starts showing pathologic behavior, presumably from spawning massive amount of very short-lived threads, and then having to synchronize them. For example classifying glibc-locale takes 5m20s with 256 threads (default on my machine!) 1m49s with 64 threads 59s with 16 threads 48s with 8 threads Even a more typical recipe like webkitgtk is affected: 47s with 256 threads 32s with 64 threads 27s with 16 or 8 threads I have found that the optimal amount is actually four: this also means that only four compressors are running at a time, but as they're themselves using threads, and typical recipes are dominated by just two or three large packages, this does not affect overall completion time. (From OE-Core rev: 286d456e71ee2730c197ce394d6be2c7eeced18d) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_rpm: use zstd's default compression levelAlexander Kanavin2024-11-211-1/+1
| | | | | | | | | | | | | | | | | | | | zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that), while 19 is the highest and slowest. It's not clear why 19 was picked to begin with, possibly I copy-pasted it from rpm's examples without thinking: https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66 This brings significant speedups in rpm's compression step: for example compressing webkitgtk takes 11s instead of 36s. The rpm size increases from 175648k to 234860k. I think it's a worthy default tradeoff. (From OE-Core rev: f1c7d76361425c85d68ad1f61be1e7ff05df4bab) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: rewrite sstate_archive_package in pythonRoss Burton2024-11-181-27/+26
| | | | | | | | | | | | | | As sstate_archive_package just calls tar, writing the function in shell is actually more complex and opaque than the equivalent python. Don't check for zstd vs pzstd, we have pzstd in HOSTTOOLS so it will always be available. (From OE-Core rev: c9ac5d9d94f254292cf3cafdf273dd6b61d3baa7) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* do_package/sstate/sstatesig: Change timestamp clamping to hash output onlyRichard Purdie2024-10-281-16/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code was changing the timestamps of the files in the do_package output, particularly the files added for debug sources. This was to do two things: a) make do_package sstate more reproducible b) ensure better hash equivalence matching Unfortuately the debug source files are hardlinks into the source tree for efficiency so touching these, touches a lot of files in ${B} and ${S}. This causes unpredictable effects if compile is run again for example, or could cause compiling in the install task. The hash equivalence matching is of key importance but we can mimic that using clamping of the file timestamps in the depsig output used to generate the hashes. This patch drops the global timestamp clamping, instead allowing the files to retain their creation timestamps into sstate. This makes do_package sstate slightly less reproducibile. We could clamp the sstate timestamps but that would lead to two different sets of timestamps depending on whether the data came from sstate or not. I'd prefer to have consistent code behaviour, rather than differing behavhour depending on whether data came from sstate or not. If we wanted to have reproducibiliy and fix the "corruption" of S/B and have consistent codepaths, the only other option would be two copies of the sources, which could end up huge and seems the least desireable option. This patch therefore drops the timestamp clamping in the sstate files and tweaks the depsig data generation to clamp the timestamps for do_package instead since this seems the best compromise. I validated that rpm/deb/ipk files still generate correctly as before. (From OE-Core rev: 475759fdab7200488b2a568b2ba1aa31a456d113) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/package: Check incompatible licenses at packaging timeJoshua Watt2024-10-252-5/+0
| | | | | | | | | | | | | | | | | | | Instead of checking for incompatible licenses in the anonymous python and setting '_exclude_incompatible-', (re)check all the packages in populate_packages(). This ensures that all packages are processed, even dynamically generated ones. The use of the '_exclude-incompatible-' variable set in base.bbclass has been the mechanism used for per-packages licenses since it was added as a feature (although with different names for the variable throughout history). However, since this misses dynamic packages, calling oe.license.skip_incompatible_package_licenses() a second time on the actual final package set is a better solution. (From OE-Core rev: 1816c5a5e724a7f907c9afe4a47b6699ea081ace) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/license: Move package license skip to libraryJoshua Watt2024-10-251-31/+4
| | | | | | | | | | Moves the code that skips packages with incompatible licenses to the library code so that it can be called in other locations (From OE-Core rev: 86eb409e3c1b30110869ec5a0027ae2d48bbfe7f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes-global/license: Move functions to library codeJoshua Watt2024-10-252-170/+5
| | | | | | | | | | | | | | Moves several of the functions in license.bbclass to be library code New function dependencies were manually verified using bitbake-dumpsigs to ensure that bitbake identified the same dependencies even though they are now in library code (although the new function names mean that the task hashes still change) (From OE-Core rev: 0333e04e353991260c5f67a72f80f3ab9dcf526a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: Ensure package_qa tasks run in builds when expectedRichard Purdie2024-10-241-0/+2
| | | | | | | | | | | | | Currently, if you "bitbake XXX" and XXX depends on something else, the do_package_qa teask for that something may not run. Users would generally expect it to have though. Add in the missing dependency to ensure that do_build does trigger the right package_qa tasks. (From OE-Core rev: e0beb64c6d3cf1d649f79a8704fb25cdf83b4a8b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity.bbclass: skip check_userns for non-local uidChangqing Li2024-10-221-0/+2
| | | | | | | | | | | | | Bitbake preserve network for non-local uid, refer [1], so check_userns is not needed for non-local uid [1] https://git.openembedded.org/bitbake/commit/?id=4eafae7904bae6e5c6bc50356e8a9077f2e207fa (From OE-Core rev: 808d0cece22bcbee15236717e158da247cbedaf1) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: avoid race condition when DEBIAN/CONTROL entries are removedRoss Burton2024-10-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a race condition when iterating directories which are being altered whilst iterating, which is something that can and does happen when do_package_qa runs at the same time as eg do_package_write_ipkg (the opkg metadata is written inside the build tree). The race is that naive code will list a directory contents and then stat() each name to determine if its a directory or file. The classic failure that we see is that CONTROL/ is found on a listdir but deleted by the time the stat happens, so is incorrectly listed as a file (because it is not a directory). Since Python 3.5, os.walk() uses scandir() instead of listdir() which mitigates this race by returning the file type alongside the name, so a stat is no longer needed to identify the type. However, cachedpath.walk() was copied from Python before this, so it uses listdir() and has this race condition. Since I changed insane to use cachedpath.walk()[1] I inadvertently reintroduced this race. I believe there's actually no need to use cachedpath.walk() and a logical fix is to simply use os.walk(): With os.walk() each directory is listed and categorised in a single os.scandir() as the underlying syscall, getdents64, returns the type. However, cachedpath.walk() uses os.listdir() which ignores the type field returned and has to do a stat() on every file to determine the type. Thus, we should switch users of cachedpath.walk() to os.walk(): there's no real gain in what is effectively just a prefetch for the stat cache, but depending on what the calling code does may result in more stat() calls than needed. In the future we may want to redesign cachedpath to reimplement walk so that it can also cache the DirEntry instances as returned by scandir() as that will avoid needing to call stat() at all in many cases. However I believe we should instead use a caching pathlib.Path instance instead. [1] cad3c8 insane: use oe.cachedpath.CachedPath instead of os.path (From OE-Core rev: 22e4486d65e4874bf48d89160d69118f318278e8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: Don't used cachedpath for os.lstat()Richard Purdie2024-10-141-3/+2
| | | | | | | | | | | The functions behave slightly differently to the functions they're caching and the use in insane.bbclass isn't compatible. For now, to avoid build failures, switch back to the stat calls. We may be able to improve cachedpath or change the call sites. (From OE-Core rev: fa771ae887ab5152f043748cf3419735831bcf7b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_rpm: Check if file exists before open()Pavel Zhukov2024-10-111-4/+12
| | | | | | | | | | | | | | | | Exception handler tries to read() /etc/passwd file in sysroot and if file doesn't exist for any reason then it raises FileNotFoundError exception which mask the original source of the problem and makes debugging of the issue more difficult. Fixes: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/codebuild/output/src1899304708/src/build/tmp-container/work/core2-64-oe-linux/emqx-bin/4.3.12/recipe-sysroot/etc/passwd' (From OE-Core rev: 4ad9a0e0b11eb7bc5a3dd45fc8945e094ea949e9) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: rewrite package_qa_check_archRoss Burton2024-10-111-16/+25
| | | | | | | | | | Reorder and comment the architecture checks to make it clearer what they are actually checking. (From OE-Core rev: 78db9e79e1a307ffb8436e26656bfb98efb513bc) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: use oe.cachedpath.CachedPath instead of os.pathRoss Burton2024-10-111-14/+28
| | | | | | | | | | | | | | | | | | | The insane QAPATHTESTs make many os.stat() calls, the majority of which are redundant with caching as the initial sweep does a stat() on every entry to determine if it is a file or a directory, and from then on each test that does further stat()s is redundant as the tree doesn't change. Switch os.stat() and friends (os.path.isfile(), etc) to use a common oe.cachedpath.CachedPath() instance that is shared between all of the functions, meaning only one stat is done. In my test case of ltp:do_package_qa, this reduces the time taken from 44s to 37s. (From OE-Core rev: cad3c889439fd6a007debd6f2f6578f4a1e16c9c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: optimise test in package_qa_check_shebang_sizeRoss Burton2024-10-111-2/+1
| | | | | | | | | | Check whether the elf is not None first, before doing os.stat() calls on disk. Also don't check anything that isn't a file, not just FIFOs. (From OE-Core rev: 38454a2675f38c7db55efcb67bbb8b9fef7e0bf1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: micro-optimise the sweep of pkgfilesRoss Burton2024-10-111-16/+11
| | | | | | | | | | | | | | Don't actively do more work: - Exit early if there are no packages being generated - Don't iterate repeatedly when removing CONTROL and DEBIAN - Extend a list with another list instead of appending item by item - Remove unused variables (From OE-Core rev: 79ffb8896d570dd935d3aea9d28ee20b52e1674a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: tidy up objdump preloading in package_qa_walk()Ross Burton2024-10-111-10/+11
| | | | | | | | | | | | | | Move the prepopulate function out of global scope, and access the dictionary once instead of repeatedly. This still results in each ELF being opened twice, but this avoids opening all of the files at once and the ELFFile.open() call is fairly fast. (From OE-Core rev: cda3647b32703f43c4fe2af3bab977e5698633f6) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: check for RUNPATH as well as RPATHRoss Burton2024-10-111-4/+4
| | | | | | | | | | | | | | Since oe-core 66f8a7 merged in 2023[1], ld sets DT_RUNPATH instead of DT_RPATH when -rpath is specified, which we don't check for. Update the insane tests to look at both RPATH and RUNPATH. [1] oe-core 66f8a745668a067d8d763fa2af3e65f26c9c1ebe (From OE-Core rev: d6c5076d179a3d5ebb74b719ec4d523c197c1918) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane: only parse ELFs if they're files, not symlinksRoss Burton2024-10-111-19/+2
| | | | | | | | | | | This reduces the number of files that need to be swept by not scanning eg the library symlinks, and means we can remove the explicit islink() checks in many of the tests. (From OE-Core rev: aa9ec4b5c719bf610ad953095d1111e4c257747e) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity: check for working user namespacesRoss Burton2024-09-131-0/+24
| | | | | | | | | | | | | | | | If user namespaces are not available (typically because AppArmor is blocking them), alert the user. We consider network isolation sufficiently important that this is a fatal error, and the user will need to configure AppArmor to allow bitbake to create a user namespace. [ YOCTO #15592 ] (From OE-Core rev: b6af956fe6e876957a49d4abf425e8c789bf0459) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>