summaryrefslogtreecommitdiffstats
path: root/meta/lib/oe
Commit message (Collapse)AuthorAgeFilesLines
...
* classes/create-spdx: Add SPDX_PRETTY optionJoshua Watt2022-06-151-2/+2
| | | | | | | | | | | Adds an option to make the SPDX more human-readable (at the expense of a larger files) (From OE-Core rev: 4799594b26f77ed259dc661bf077519b338390c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base/buildhistory/image-buildinfo: Use common buildcfg functionRichard Purdie2022-06-111-0/+7
| | | | | | | | Reduce code duplication to a common function in oe.buildcfg. (From OE-Core rev: 0b191a1df4c3722defb09dde0c16b1d9e7fe7ef6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildcfg: Drop unused svn revision functionRichard Purdie2022-06-111-11/+0
| | | | | | | | | This isn't used anywhere and everyone is using git now, the functions are now hardcoded as such too. Drop this function. (From OE-Core rev: 09ba96a2d7fa02c7a82758bbf4109b04ffca2c55) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/buildcfg: Share common clean/dirty layer functionRichard Purdie2022-06-111-1/+14
| | | | | | | | | The comments even say this was copy/paste code. Move to a shared library function. (From OE-Core rev: ac3de2f850a418673b87e1c454970cb099e191b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/buildcfg: Move git/layer revision code into new OE module buildcfgRichard Purdie2022-06-111-0/+40
| | | | | | | | | There is a load of duplicated git/layer/revision code which makes most sesne as a python library, not bbclass code. Start to refactor as such. (From OE-Core rev: 439cdf8a1e52fd2c4dc81dc40ce7e6af282ce7ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rootfs.py: close kernel_abi_ver_fileMartin Jansa2022-06-081-1/+3
| | | | | | | | | | | | * fixes: oe-core/meta/lib/oe/rootfs.py:331: ResourceWarning: unclosed file <_io.TextIOWrapper name='pkgdata/mach/kernel-depmod/kernel-abiversion' mode='r' encoding='UTF-8'> kernel_ver = open(kernel_abi_ver_file).read().strip(' \n') (From OE-Core rev: f9dd8ee063c1132265248457fcd628e1e93727be) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rootfs.py: find .ko.zst kernel modulesSean Anderson2022-06-061-1/+1
| | | | | | | | | | | With CONFIG_MODULE_COMPRESS_ZSTD enabled, kernel modules will have a .ko.zst extension. This fixes depmod not being run. Fixes: 1b696a45ddb ("rootfs.py: Add check for kernel modules before running depmod") (From OE-Core rev: 425efac7110f0f42d70643e0a448e834d0f01a7a) Signed-off-by: Sean Anderson <sean.anderson@seco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: move update_symlinks to a libraryMarta Rybczynska2022-06-061-0/+10
| | | | | | | | | Move the function to a library, it could be useful in other places. (From OE-Core rev: debd37abcdde8788761ebdb4a05bc61f7394cbb8) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* patch.py: make sure that patches/series file exists before quilt popMartin Jansa2022-05-311-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Since quilt upgrade to 0.67 some recipes sometimes fail in do_patch with errors like: ERROR: Applying patch 'GPLv2.patch' on target directory '/OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31' CmdError('quilt --quiltrc /OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout: stderr: File series fully applied, ends at patch GPLv2.patch ') * It affects only recipes with S = "${WORKDIR}", which wipe only ${S}/patches, because in other cases whole ${S} is wiped when do_unpack is re-executed. * It was originally added in: https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2 since then it was extended to wipe whole ${S} when ${S} != ${WORKDIR} in: https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2 https://git.openembedded.org/openembedded-core/commit/?id=eccae514b71394ffaed8fc45dea7942152a334a1 this is now causing issues to quilt-0.67 because it checks that ${S}/patches/series exists during 'quilt pop -a -f' which we call from QuiltTree.Clean to undo patches possibly already applied in ${S} in previous do_patch execution. * There are couple recipes affected by this e.g. keymaps (.patch already removed in oe-core), makedevs (.patch removal sent to ML yesterday https://lists.openembedded.org/g/openembedded-core/message/166172), devmem2 (https://lists.openembedded.org/g/openembedded-devel/message/97270), but there are other recipes with S = "${WORKDIR}" where you can trigger this e.g. by having a .patch file in DISTRO layer .bbappend (e.g. tzdata with webOS https://github.com/webosose/meta-webosose/blob/06e5298d9f5c47679b679081d9930f8d1c776142/meta-webos/recipes-extended/tzdata/tzdata.bbappend#L10) This do_patch issue is caused by: https://git.savannah.nongnu.org/cgit/quilt.git/commit/?id=8b39a960afcf45cd4f5804ae62b6b0656bdb191d introduced in kirkstone with: https://git.openembedded.org/openembedded-core/commit/?h=kirkstone&id=fa71afcee9ab42198c619333b77a15bd2ae02b20 The shortest sequence to reproduce this is just bitbake keymaps -c patch bitbake keymaps -c unpack -f bitbake keymaps -c patch with https://git.openembedded.org/openembedded-core/commit/?id=17d981005a0c0c97702ad88602b7181b69bcc9eb reverted. And the change in quilt behavior is causing QuiltTree.Clean (quilt pop -a -f) in: https://git.openembedded.org/openembedded-core/tree/meta/lib/oe/patch.py?id=17d981005a0c0c97702ad88602b7181b69bcc9eb#n601 to silently fail with "No series file found" before undoing the patches in ${S} and then quilt push failing, because all the patches are _still_ applied in ${S}. Removing ".pc" doesn't help, because we really need quilt's help to undo the patches (in this case to delete COPYING file from WORKDIR before applying the .patch which tries to add it again), because do_unpack cannot just wipe S and start over (because S == WORKDIR) - nor selectively removing the files listed in SRC_URI, because COPYING file isn't listed there. Using skip_series_check in 'quilt pop' (partially reverting the change from upstream) does fix this as well and it's simple one line patch (just adding skip_series_check=1 in pop.in), but might be difficult to upstream, because it's this strange OE specific behavior that we remove 'patches' directory and then still need quilt pop to work. (From OE-Core rev: c9d36882044b1c633d8611a77df54cd68c9bee25) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/sstatesig: Fix find_siginfo to match sstate filename generationRichard Purdie2022-04-291-2/+2
| | | | | | | | | | | | | | | | sstate filename generation was changed a while ago and taskname has to be passed into core functions for the correct filename to be generated. Update find_siginfo to match those changes and pass in taskname via SSTATE_CURRTASK. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting. [YOCTO #14774] (From OE-Core rev: 51d7310b6cf8ef9033d461868c07f686656070ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve_check: skip remote patches that haven't been fetched when searching for ↵Ross Burton2022-04-281-2/+3
| | | | | | | | | | | | | | | | | CVE tags If a remote patch is compressed we need to have run the unpack task for the file to exist locally. Currently cve_check only depends on fetch so instead of erroring out, emit a warning that this file won't be scanned for CVE references. Typically, remote compressed patches won't contain our custom tags, so this is unlikely to be an issue. (From OE-Core rev: cefc8741438c91f74264da6b59dece2e31f9e5a5) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* terminal.py: Restore error output from TerminalPeter Kjellerstedt2022-04-261-1/+2
| | | | | | | | | | | | | In bitbake commit 1ecc1d94 (process: Do not mix stderr with stdout), bb.process.Popen() was changed to no longer combine stdout and stderr by default. However, the Terminal class was not updated to reflect this and subsequently only output stdout in case of failures. (From OE-Core rev: 116d0bb07ba044cf8847bf3d5c3996ad7e58b7ae) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_manager: sign DEB package feedsFerry Toth2022-04-051-3/+16
| | | | | | | | | | | | | | | | | | Implement debian package repository signature. For each Release file created in repository subdirectory, a signature Release.gpg is created. Signature is performed using gpg backend when the following variables are set in local.conf: PACKAGE_CLASSES += "sign_package_feed" PACKAGE_FEED_GPG_NAME = "<Id of GPG key>" PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>" (From OE-Core rev: fcc3cee276999efe6402959eb295e7a0e1e96f96) Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net> Signed-off-by: Ferry Toth <ftoth@exalondelft.nl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gpg-sign: Add parameters to gpg signature functionXavier Berger2022-04-051-1/+5
| | | | | | | | | | | output_suffix: If defined, add output_suffix as file name extension. use_sha256: If True, use sha256 for gpg as digest algorithm (From OE-Core rev: cfcaa54dc73925df448099fb60f75b18350b2a3b) Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net> Signed-off-by: Ferry Toth <ftoth@exalondelft.nl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: add json formatMarta Rybczynska2022-03-301-0/+16
| | | | | | | | | | | | | | | | | | | | | Add an option to output the CVE check in a JSON-based format. This format is easier to parse in software than the original text-based one and allows post-processing by other tools. Output formats are now handed by CVE_CHECK_FORMAT_TEXT and CVE_CHECK_FORMAT_JSON. Both of them are enabled by default. The JSON output format gets generated in a similar way to the text format with the exception of the manifest: appending to JSON arrays requires parsing the file. Because of that we first write JSON fragments and then assemble them in one pass at the end. (From OE-Core rev: df567de36ae5964bee433ebb97e8bf702034994a) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* image.bbclass: make sure do_rootfs run from a clean workspaceMing Liu2022-03-131-4/+0
| | | | | | | | | | | | Add ${IMAGE_ROOTFS} and ${IMGDEPLOYDIR} to do_rootfs[dirs] and do_rootfs[cleandirs], this ensures do_rootfs run from a clean workspace, with this change, we can now remove two bb.utils.mkdirhier lines from meta/lib/oe/rootfs.py. (From OE-Core rev: cb8b6f7eee4e059bb311330c57068e11bc477366) Signed-off-by: Ming Liu <liu.ming50@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/scripts: Improve internal variable namingSaul Wold2022-03-101-2/+2
| | | | | | | | | Update internal variable names to improve the terms used. (From OE-Core rev: f408068e5d7998ae165f3002e51bc54b380b8099) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ipk: Decode byte data to string in manifest handlingAndrew Jeffery2022-03-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | ``` File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 69, function: create_full 0065: output = pm.dummy_install(pkgs_to_install) 0066: 0067: with open(self.full_manifest, 'w+') as manifest: 0068: pkg_re = re.compile('^Installing ([^ ]+) [^ ].*') *** 0069: for line in set(output.split('\n')): 0070: m = pkg_re.match(line) 0071: if m: 0072: manifest.write(m.group(1) + '\n') 0073: Exception: TypeError: a bytes-like object is required, not 'str' ``` Change-Id: Ifefb13bfa22c766d20ab9f73f7abe5163b3df86f (From OE-Core rev: cf9df9e8d89fee9cea4785c94a1e3004a5f3469d) Signed-off-by: Andrew Jeffery <andrew@aj.id.au> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* ipk: Import re in manifest moduleAndrew Jeffery2022-03-041-0/+1
| | | | | | | | | | | | | | | | | | | | | | ``` File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 67, function: create_full 0063: 0064: output = pm.dummy_install(pkgs_to_install) 0065: 0066: with open(self.full_manifest, 'w+') as manifest: *** 0067: pkg_re = re.compile('^Installing ([^ ]+) [^ ].*') 0068: for line in set(output.split('\n')): 0069: m = pkg_re.match(line) 0070: if m: 0071: manifest.write(m.group(1) + '\n') Exception: NameError: name 're' is not defined ``` Change-Id: I769a2ab5e57c7b60598ea0390b576d707356db9d (From OE-Core rev: e240a8ede690e56bcf53a97b3be5592e6e4a5a15) Signed-off-by: Andrew Jeffery <andrew@aj.id.au> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* copy_buildsystem: allow more layer pathsDaniel Wagenknecht2022-03-041-8/+4
| | | | | | | | | | | | | | | | | | Layers could be located anywhere. The eSDK should work with them even if they are not located in TOPDIR or in the same parent directory as COREBASE. For layers located in the same parent directory as COREBASE this preserves the intent from the previous copy_buildsystem: include layer tree during build structure creation commit. Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b (From OE-Core rev: 16d330d42e03085769eddb1b60ba1df7228baf36) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rust: Introduce arch_to_rust_arch()Andrew Jeffery2022-03-021-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On modern Power systems `uname -m` yields 'ppc64le' while the toolchain knows the architecture as 'powerpc64le'. Provide a mapping from one to the other to integrate with the existing architecture configuration flags. arch_to_rust_arch() only exists to map the OE *_ARCH variables before any further processing, unlike arch_to_rust_target_arch() which is specific to the internal triple handling of rust. On Linux ppc64le systems the changes give the following config: ``` $ cat ./tmp/work/ppc64le-linux/rust-native/1.58.0-r0/targets/ppc64le-linux.json { "llvm-target": "powerpc64le-unknown-linux-gnu", "data-layout": "e-m:e-i64:64-n32:64-v256:256:256-v512:512:512", "max-atomic-width": 64, "target-pointer-width": "64", "target-c-int-width": "64", "target-endian": "little", "arch": "powerpc64", "os": "linux", "env": "gnu", "vendor": "unknown", "target-family": "unix", "linker": "gcc", "cpu": "generic", "dynamic-linking": true, "executables": true, "linker-is-gnu": true, "linker-flavor": "gcc", "has-rpath": true, "has-elf-tls": true, "position-independent-executables": true, "panic-strategy": "unwind" } ``` Change-Id: Ief0c01189185d7d4da31d307270bec4e1de674ca (From OE-Core rev: 9ab61e3cfef0157393cb870d606c2f362e190889) Signed-off-by: Andrew Jeffery <andrew@aj.id.au> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license/insane: Show warning for obsolete license usageRichard Purdie2022-03-021-0/+10
| | | | | | | | | | | | | | | We want to use SPDX identifiers in LICENSE variables. There is now a conversion script to make most of the translations. Add a list of strings which have been replaced so we can show warnings to users if they're still used anywhere. Add checks to the package as insane check. This is currently a warning by default but can be turned off or made an error as per the other standard checks. (From OE-Core rev: 9379f80f484f94686a4d494e9e237fadfb72a938) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base/license: Rework INCOMPATIBLE_LICENSE variable handlingSaul Wold2022-03-021-0/+5
| | | | | | | | | | | | | | | | | | | | This re-writes the INCOMPATIBLE_LICENSE checking code to replace the WHITELIST_<lic> with INCOMPATIBLE_LICENSE_EXCEPTIONS = '<pkg>:<lic> <pkg>:<lic> ...' This initial change leaves most of the code structure in place, but the code in base.bbclass needs to be re-written to make the check more consistent around packages (PKGS) and not recipe names (PN). This also is taking into account the changes for SPDX licenses. The aim is to provide a mode consistent variable where the variable name is known and can easily be queried. (From OE-Core rev: 0d19c45ba6cf43518f380ca5afe9753a2eda0691) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: do not add TARGET_ARCH to pkgarch for cross recipes.Alexander Kanavin2022-03-021-1/+1
| | | | | | | | | | | | This is redundant (target arch is already in PN), and breaks compiling a cross-canadian toolchain, as that needs populating the sysroot with two different native-hosted toolchains built from cross recipes. Inserting TARGET_ARCH allows only one or the other. (From OE-Core rev: 33fc1792cd782feb8dbb4285e3006bb588f7978f) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license.py: Correct a commentPeter Kjellerstedt2022-02-271-10/+7
| | | | | | | (From OE-Core rev: 1f7a34c6d246c6f42ab823ffd0bd0306705ad88d) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* licenses: Fix logic error introduced in renameRichard Purdie2022-02-251-2/+2
| | | | | | | | The previous commit introduced a small logic error. Fix the renaming issue. (From OE-Core rev: b01e10b27d23ea1b4ac58376c2423505a70832d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license.py: rename variablesSaul Wold2022-02-251-14/+17
| | | | | | | | | Update the comment to reflect new variable names (From OE-Core rev: 7fbab1aefc127f0e1834f51a8a793b0d7e7b4f07) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* patch.py: Prevent git repo reinitializationPavel Zhukov2022-02-251-3/+8
| | | | | | | | | | There were few bugs in the _isInitialized() function which might trigger git repo to be reinitialized and patches failing to apply. (From OE-Core rev: 80500ecda4c1bc8812e6e078b6b0db5ec46624de) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* create-spdx: add support for SDKsAndres Beltran2022-02-251-0/+4
| | | | | | | | | | Currently, SPDX SBOMs are only created for images. Add support for SDKs. (From OE-Core rev: c3acbb936a339636153903daf127eec9f36de79b) Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* recipeutils: Add missing get_srcrev() callRichard Purdie2022-02-181-0/+1
| | | | | | | | | | | Updates within bitbake now require this call to be triggered somewhere with floating git source revs. Add the missing call. Issue can be reproduced with: devtool check-upgrade-status dbus-wait (From OE-Core rev: d7088b2b5012f9e386914a6837685f5869379cc5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib/useradd: The option -P is deprecatedDaiane Angolini2022-02-121-2/+0
| | | | | | | | | | | | This is also covered in documentation since: https://git.yoctoproject.org/yocto-docs/commit/?id=3f3e5574ac9801ad92940168b61b532e0bd53a80 [YOCTO 14605] (From OE-Core rev: 3f9153986e4e6f667b4bbe97613ec0b279665a97) Signed-off-by: Daiane Angolini <daiane.angolini@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* insane.bbclass: use multiprocessing for collecting 'objdump -p' outputAlexander Kanavin2022-02-081-0/+6
| | | | | | | | | | | | | This was prompted by ltp's unreasonably long package_qa times; it has a massive amount of executables and insane runs objdump for all of them, serially. This reduces the time from 4 minutes to 1m20s on my machine. (From OE-Core rev: fac984b99fdb46949879516cb87153860f402c75) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* spdx: Add set helper for list propertiesJoshua Watt2022-02-071-1/+4
| | | | | | | | | | | This adds the set helper for list properties to allow list members to be replaced with a new list. This is necessary as it is (currently) the only way to replace the default value for a list member. (From OE-Core rev: 8280b857c510646cd10167dd2ea9a0c2d5f1e800) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* prelink: Drop support for itRichard Purdie2022-02-052-48/+0
| | | | | | | | | | | | | | Prelink is being dropped by glibc in 2.36. It already causes issues with binary corruption, has a number of open bugs and is of questionable benefit without disabling load address randomization and PIE executables. We disabled it by default a while back but left people able to use it. We would be unable to maintain it alone without glibc support so remove the remaining pieces. (From OE-Core rev: 23c0be78106f1d1e2bb9c724174a1bb8c56c2469) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/native: Propagate dependencies to outhashJoshua Watt2022-01-171-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Native task outputs are directly run on the build system during the build after being built. Even if the output of a native recipe doesn't change, a change in one of its dependencies may cause a change in the output it generates (e.g. rpm output depends on the output of its dependent zstd library). This can cause poor interactions with hash equivalence, since this recipe's output-changing dependency is "hidden" and downstream tasks only see that this recipe has the same outhash and therefore is equivalent. This can result in different output in different cases and issues with reproducible builds in parcular (e.g. rpm compression changes for the same content). To resolve this, unhide the output-changing dependency by adding it's unihash to this tasks outhash calculation. Unfortunately, we don't know specifically know which dependencies are output-changing, so we have to add all of them. [YOCTO #14685] (From OE-Core rev: d6c7b9f4f0e61fa6546d3644e27abe3e96f597e2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Add support for kernel strippingSaul Wold2022-01-131-1/+8
| | | | | | | | | | | | | | Extend runstrip() to accept additional argument to enable sharing it with the kernel do_strip() so that KERNEL_IMAGE_STRIP_EXTRA_SECTIONS can be passed. Since is_elf() understands kernel modules there is no need to keep a seperate list for kernmodules or hardcode the values to runstrip. (From OE-Core rev: e09a8fa931fe617afc05bd5e00dca5dd3fe386e8) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/reproducible: correctly set .git location when recursively looking ↵Alexander Kanavin2022-01-051-1/+1
| | | | | | | | | for git repos (From OE-Core rev: ffdaa1a0527691d66dd28e86bd015bfad7a020f6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_manager: ipk: Fix host manifest generationOleksiy Obitotskyy2022-01-041-0/+4
| | | | | | | | | | | Since honister host manifest stopped to generate, i.e. manifest file is empty but all ipks/files into sdk is ok. (From OE-Core rev: 79a2392f5d2a4cb6509a83afb40bca01bac59914) Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* patch.py: Initialize git repo before patchingPavel Zhukov2021-12-031-1/+15
| | | | | | | | | | | | | | | If PATCHTOOL="git" has been specified but workdir is not git repo bitbake fails to apply the patches with error message: Command Error: 'git rev-parse --show-toplevel' exited with 0 Output: fatal: not a git repository (or any of the parent directories): .git Fix this by initializing the repo before patching. This allows binary git patches to be applied. (From OE-Core rev: 6184b56a7a0fc6f5d19fdfb81e7453667f7da940) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity/lib: Replace usage of LooseVersion() with bb.utils.vercmp_string_op()Richard Purdie2021-11-292-5/+4
| | | | | | | | | distutils is going away and we have functionality in bitbake which can handle these comparisions so switch to the bb.utils function. (From OE-Core rev: fe624b520e6c75e16a8f394785ab0216341402f9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe/utils: by default cap cpu_count() to 64 coresRoss Burton2021-11-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | Larger systems may have large numbers of cores, but beyond a certain point they can't all be used for compiling: whilst purely compute-intensive jobs can be parallelised to hundreds of cores, operations such as compressing (needs lots of RAM) or compiling (lots of I/O) don't scale linearly. For example, the Marvel ThunderX2 has 32 cores, each capable of executing four threads, and can be configured with two sockets, making 256 CPUs according to Linux. Zstd using 256 threads has been seen to fail to allocate memory during even small recipes such as iso-codes. Add a default cap of 64 CPUs to the cpu_count() method so that extreme parallisation is limited. 64 is high enough that meaningful gains beyond it are unlikely, but high enough that most systems won't be effected. (From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe/utils: allow naming threads in ThreadedPoolRoss Burton2021-11-251-6/+5
| | | | | | | | | | When looking at logs involving thread pools it is useful if the threads can be named. (From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* overlayfs: all overlays unitVyacheslav Yurkov2021-10-301-0/+5
| | | | | | | | | | | | Application can depend on several overlayfs mount points. Provide a systemd unit application can depend on to make sure all overlays are mounted before it is started to avoid any race conditions (From OE-Core rev: b38e194db0c6825f28c56123cf88af94d3f52beb) Signed-off-by: Bruno Knittel <Bruno.Knittel@bruker.com> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* spdx.py: Add annotation to relationshipSaul Wold2021-10-281-1/+5
| | | | | | | | | | | Having annotations on relationship can provide additional information about the relationship such as how it was derived. (From OE-Core rev: d98585aa89e1d3819f8139a07fb7376ef89b37f8) Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe/utils: log exceptions in ThreadedWorker functionsRoss Burton2021-10-181-1/+2
| | | | | | | | | | | | | | If the function a ThreadedWorker is executing raises an exception, don't use print() as that mostly disappears. Instead, output it to the logger. This is done using bb.mainlogger.debug directly instead of bb.debug() as this allows us to pass the exception instance directly, which is then incorporated into the log stream. (From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/qa,insane: Move extra error handling functions to libraryMike Crowe2021-10-171-0/+34
| | | | | | | | | | | | | | | | | | | | | | | Extract package_qa_write_error, package_qa_handle_error and package_qa_add_message functions from insane.bbclass to lib/oe/qa.py and drop the package_qa_ prefixes. Update various bbclasses to use the new functions. No import is required since base.bbclass puts oe.qa in OE_IMPORTS. Stop requiring callers to manually track whether a fatal error has been encountered via a "sane" flag. Instead replace the QA_SANE variable with QA_ERRORS_FOUND and call oe.qa.exit_if_errors or oe.qa.exit_with_message_if_errors at the end of each task. Inspired by discussion resulting from https://lists.openembedded.org/g/openembedded-core/message/156793 and https://lists.openembedded.org/g/openembedded-core/message/156900 (From OE-Core rev: f0ad152ef4cc15c042bc9eeefb6af096d054b220) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Drop BUILD_REPRODUCIBLE_BINARIES variableRichard Purdie2021-10-161-1/+1
| | | | | | | | | | We want things to be reproduicble and the variable doesn't really change much any more. Drop the remaining uses and make those code paths always active. (From OE-Core rev: d15fb02c7ee7da50e322d74bc6a545234e20c7f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Merge code into base.bbclassRichard Purdie2021-10-161-0/+51
| | | | | | | | | | Reproducibility is here to stay and needs to be part of our default workflow. Move the remaining code to base.bbclass so it is always a first class citizen and it is clear people need to be mindful of it. (From OE-Core rev: abb0671d2cebfd7e8df94796404bbe9c7f961058) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Move class function code into libraryRichard Purdie2021-10-161-0/+33
| | | | | | | | | To try and avoid parse/memory overhead of functions within bitbake, move the bulk of the reproducibility functions to the function library. (From OE-Core rev: f2fd1c9d75e774c8a5271cdc1ec6f65c4492f941) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate: Allow validation of sstate singatures against list of keysDaniel McGregor2021-10-141-5/+22
| | | | | | | | | | Allow a user to validate sstate objects against a list of keys, instead of just any known key in the user's keychain. (From OE-Core rev: 52ba0c5e6e2e3d5d01dc3f01404f0ab1bb29b3b5) Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>