summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/selftest
Commit message (Collapse)AuthorAgeFilesLines
...
* oeqa/selftest/wic: add test for excluding symlinksYi Zhao2025-04-081-5/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add test to check if --exclude-path option can exclude symlinks. This test validates commit[1]. Test result: $ oe-selftest -r wic.Wic.test_exclude_path 2025-04-03 15:11:25,211 - oe-selftest - INFO - meta-selftest layer not found in BBLAYERS, adding it 2025-04-03 15:11:30,016 - oe-selftest - INFO - Adding layer libraries: 2025-04-03 15:11:30,017 - oe-selftest - INFO - /buildarea/poky/meta/lib 2025-04-03 15:11:30,017 - oe-selftest - INFO - /buildarea/poky/meta-yocto-bsp/lib 2025-04-03 15:11:30,017 - oe-selftest - INFO - /buildarea/poky/meta-selftest/lib 2025-04-03 15:11:30,019 - oe-selftest - INFO - Checking base configuration is valid/parsable NOTE: Starting bitbake server... 2025-04-03 15:11:31,652 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea/poky/build-st/conf/local.conf 2025-04-03 15:11:31,653 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2025-04-03 15:11:31,653 - oe-selftest - INFO - test_exclude_path (wic.Wic) 2025-04-03 15:43:11,341 - oe-selftest - INFO - ... ok 2025-04-03 15:43:11,341 - oe-selftest - INFO - ---------------------------------------------------------------------- 2025-04-03 15:43:11,342 - oe-selftest - INFO - Ran 1 test in 1899.900s 2025-04-03 15:43:11,342 - oe-selftest - INFO - OK 2025-04-03 15:43:14,834 - oe-selftest - INFO - RESULTS: 2025-04-03 15:43:14,835 - oe-selftest - INFO - RESULTS - wic.Wic.test_exclude_path: PASSED (1899.69s) 2025-04-03 15:43:14,836 - oe-selftest - INFO - SUMMARY: 2025-04-03 15:43:14,836 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 1899.900s 2025-04-03 15:43:14,836 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) [1] https://git.openembedded.org/openembedded-core/commit/?id=42e829ac1e9d74646b6dfb327b18b15f6b0df60b (From OE-Core rev: 0dd455bed9b52c0cf237ea2f8bd1a8f7890078e9) Signed-off-by: Yi Zhao <yi.zhao@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: spdx: Add tar test for SPDX 2.2Joshua Watt2025-03-241-3/+6
| | | | | | | | | | | The base-files test for SPDX 2.2 did not give good coverage, since base-files doesn't have any dependencies. Add building tar as another test which more fully exercises the code (From OE-Core rev: d678e25419c89e09c1c438363bf3a940ce903d43) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Build cpio with C17 stdKhem Raj2025-03-241-2/+2
| | | | | | | | | | | | cpio is not yet buildable with C23 standard which is default with GCC 15, therefore ensure to apply needed bandage to keep it compiling in C17 mode even with GCC 15 (From OE-Core rev: 0c637099887f1be421c8e1203f99631a1e040150) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: reset: Escape command line input used in regular expressionPeter Kjellerstedt2025-03-201-0/+19
| | | | | | | | | | | | | | | | | | | Running, e.g., `devtool reset sdbus-c++` would result in the following error: re.error: multiple repeat at position 35 This was due to the ++ in the recipe name, which would be treated as an incorrect regular expression in _reset(). Use re.escape() to make sure all characters in the recipe name are treated literally. (From OE-Core rev: 6e73bd9b3e6d529752db93879f2c0ed53873dd1a) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage add more u-boot testsAdrian Freihofer2025-03-181-6/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new test function which checks that the device-tree of U-Boot contains the public keys which are required for checking the signature of the kernel FIT image at run-time. Use this new _check_kernel_dtb function in the existing test_sign_cascaded_uboot_fit_image test case which already creates a build configuration with UBOOT_SIGN_ENABLE = "1" and keys for the kernel. But so far there was no check that the keys for the kernel verification got added to U-Boot's DTB. This test case checks the configuration where only the configuration nodes of the kernel FIT image are signed. A new test case test_sign_uboot_kernel_individual checks the configuration with two keys and signed image and signed configuration nodes. This test case covers the use case which recently broke with commit: OE-Core rev: 259bfa86f384206f0d0a96a5b84887186c5f689e u-boot: kernel-fitimage: Fix dependency loop if UBOOT_SIGN_ENABLE and UBOOT_ENV enabled and got fixed with commit OE-Core rev: 0106e5efab99c8016836a2ab71e2327ce58a9a9d u-boot: kernel-fitimage: Restore FIT_SIGN_INDIVIDUAL="1" behavior This patch also fixes a few more details: - Simplify the code by moving all the U-Boot related variables to the _fit_get_bb_vars function. - Do not set FIT_GENERATE_KEYS = "1" without inheriting the kernel-fitimage.bbclass which handles this variable. (From OE-Core rev: dda1fcbc85c26d1851dda6ed235238b15939998e) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage improve bb_vars accessAdrian Freihofer2025-03-181-9/+12
| | | | | | | | | | | | | | | Make the code slightly more robust by using e.g. bb_vars.get('UBOOT_SIGN_ENABLE') instead of bb_vars['UBOOT_SIGN_ENABLE'] for variables which are potentially undefined. This is a general cleanup but also a preparation for additional test cases. Log bb_vars in verbose mode. Drop one no longer used log message. (From OE-Core rev: 22a554e1d5a2449c4c1b34f4d5cfe1572a72a142) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: gdbserver: Adjust regex for change to outputAlex Kiernan2025-03-171-1/+1
| | | | | | | | | | | The updated kmod build (using meson) causes the output which results from `info line kmod_help` to change. Relax the regex so that both old and new reponses will pass. (From OE-Core rev: 8783e7b8109b02847cf3b07421582c4fdb18949c) Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage add more kernel testsAdrian Freihofer2025-03-111-2/+149
| | | | | | | | | | | | | | | * Test with only one externally provided ssh key not only with two keys generated by the kernel-fitimage.bbclass itself. * Add a test which signs only the configuration but not the image nodes. There was no test case which covered the probably much more important use case of setting FIT_SIGN_INDIVIDUAL = "0". * Cover also the unbundled initramfs use case. Also this use case is probably much more relevant than the bundled initramnfs use case. (From OE-Core rev: 0a5b65b83dcd9f8d1d22d074fdfad1f1e472827c) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage cleanupAdrian Freihofer2025-03-111-668/+800
| | | | | | | | | | | | | | | | | | | | | | | This is a comprehensive cleanup of the fitImage related test cases. The existing test cases were essentially the same code copied and pasted 9 times. All 9 test cases contained the code to parse an its file and to parse the output of the dumpimage utility in slightly different variants. Changing the kernel-fitimage.bbclass or the uboot-sign.bbclass would mean changing 9 test cases individually. This is no longer maintainable. This cleanup converts the code into reusable functions. The new test code is more like a reverse implementation of the bbclasses to be tested than a collection of straightforward test sequences. This also means that the test code evaluates the same bitbake variables as the implementation. This makes it much easier to add new test cases, as a test case is basically just another local.conf file. The code is not yet complete. But it can now be improved step by step in this direction. (From OE-Core rev: 1dfa03a182d9d9e9e38c410847c7dac4ed7e8ce6) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage sort testsAdrian Freihofer2025-03-111-133/+129
| | | | | | | | | | | | This is a trivial refactoring. The goal is to have all tests for kernel-fitimage.bbclass and all tests for uboot-sign.bbclass together. This refactoring greatly simplifies the diff of the next commit. (From OE-Core rev: 638abab58b653a1fb120368bd5e30bc80478757c) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: adapt u-boot tests to latest changesAdrian Freihofer2025-03-111-101/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For u-boot test cases (bitbake virtual/bootloader) inheriting the kernel-fitimage.bbclass is no longer needed. Also setting any variable which is evaluated by the kernel-fitimage.bbclass but not by uboot-sign.bbclass is pointless since: * Commit OE-Core rev: 5e12dc911d0c541f43aa6d0c046fb87e8b7c1f7e changed the test case from bitbake virtual/kernel to bitbake virtual/bootloader * Commit OE-Core rev: 259bfa86f384206f0d0a96a5b84887186c5f689e has finally removed the dependency of uboot-sign.bbclass on the kernel-fitimage.bbclass completely. Remove the related lines of code which are now without any effect. The two test cases test_uboot_fit_image and test_uboot_sign_fit_image do the exact same test. Both generate a binary equal its file: /dts-v1/; / { description = "A model description"; #address-cells = <1>; images { uboot { description = "U-Boot image"; data = /incbin/("u-boot-nodtb.bin"); type = "standalone"; os = "u-boot"; arch = "arm"; compression = "none"; load = <0x80080000>; entry = <0x80080000>; }; fdt { description = "U-Boot FDT"; data = /incbin/("u-boot.dtb"); type = "flat_dt"; arch = "arm"; compression = "none"; }; }; configurations { default = "conf"; conf { description = "Boot with signed U-Boot FIT"; loadables = "uboot"; fdt = "fdt"; }; }; }; The code diff between the two equal test cases looks like: @@ -1,8 +1,9 @@ - def test_uboot_fit_image(self): + def test_uboot_sign_fit_image(self): """ Summary: Check if Uboot FIT image and Image Tree Source (its) are built and the Image Tree Source has the - correct fields. + correct fields, in the scenario where the Kernel + is also creating/signing it's fitImage. Expected: 1. u-boot-fitImage and u-boot-its can be built 2. The type, load address, entrypoint address and default values of U-boot image are correct in the @@ -26,16 +27,15 @@ UBOOT_LOADADDRESS = "0x80080000" UBOOT_ENTRYPOINT = "0x80080000" UBOOT_FIT_DESC = "A model description" - -# Enable creation of Kernel fitImage KERNEL_IMAGETYPES += " fitImage " -KERNEL_CLASSES = " kernel-fitimage" +KERNEL_CLASSES = " kernel-fitimage " UBOOT_SIGN_ENABLE = "1" FIT_GENERATE_KEYS = "1" UBOOT_SIGN_KEYDIR = "${TOPDIR}/signing-keys" UBOOT_SIGN_IMG_KEYNAME = "img-oe-selftest" UBOOT_SIGN_KEYNAME = "cfg-oe-selftest" FIT_SIGN_INDIVIDUAL = "1" +UBOOT_MKIMAGE_SIGN_ARGS = "-c 'a smart U-Boot comment'" """ self.write_config(config) Conclusion: The test case test_uboot_sign_fit_image looks redundant. Contrary to its name, it does not insert any signature nodes into the its-file and therefore does not test any type of signature. Code history: - Commit OE-Core rev: e71e4c617568496ae3bd6bb678f97b4f73cb43d8 introduces both test cases. - Commit OE-Core rev: 5e12dc911d0c541f43aa6d0c046fb87e8b7c1f7e changes both test cases like this: - bitbake("virtual/kernel") + bitbake("virtual/bootloader") It looks like the original implementation of test_uboot_sign_fit_image was supposed to test the interaction between the kernel-fitimage.bbclass and uboot-sign.bbclass which does not longer work like that. When compiling u-boot, the variable that is relevant for creating an its file with signature nodes is: SPL_SIGN_ENABLE. This is what the test case test_sign_standalone_uboot_fit_image verifies. Lets just delete the now obsolete test_uboot_sign_fit_image test case. (From OE-Core rev: de8bfdff0f997f59a2bd27842a2ffcd365f725f3) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage: add testcases to test ATF and TEEJamin Lin2025-02-271-0/+281
| | | | | | | | | | | | | | | | | | | Add "test_uboot_atf_tee_fit_image" test caste to check u-boot FIT image and Image Tree Source(ITS) are built and the ITS has the correct fields. Add "test_sign_standalone_uboot_atf_tee_fit_image" test case to check if u-boot FIT image and Image Tree Source (ITS) are created and signed correctly for the scenario where only the u-boot proper fitImage is being created and signed. Currently, ATF and TEE(optee-os) recipes are placed in meta-arm layer. OpenEmbedded-Core is a basic and core meta layer. To avoid OpenEmbedded-core depends meta-arm, both test cases are used dummy images for testing. (From OE-Core rev: 92e51452f8831f74e0907b960135eef8cecd012a) Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: add a newline in local.conf (newbuilddir)Christos Gavros2025-02-271-0/+7
| | | | | | | | | | | | | | | If the build-st/conf/local.conf does not end with a newline when is generated then add one [YOCTO #15734] CC: Yoann Congal <yoann.congal@smile.fr> CC: Randy MacLeod <randy.macleod@windriver.com> CC: Alexander Kanavin <alex.kanavin@gmail.com> (From OE-Core rev: d2fcd9e880126bc33be2ef14e678cc1aa72683c3) Signed-off-by: Christos Gavros <gavrosc@yahoo.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/devtool: use 'config.toml' instead of plain 'config'Enrico Scholz2025-02-251-1/+1
| | | | | | | | | | | After changing naming of cargo config in cargo_common.bbclass, adapt devtool to use the new name. (From OE-Core rev: 715d27f0b4301c97f05ed3cbbaace0ba01c28f39) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rust: Oe-selftest changes for rust v1.82.0Deepesh Varatharajan2025-02-201-0/+2
| | | | | | | | | | | | A few tests are getting failed with x86 arch.The unsupported/failing tests are added to the exclude list and ignore the failing unit tests. Upstream-Status: Pending (From OE-Core rev: b1340173be2a3a91fbb135eb0e24e50c3b996425) Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Drop dependency on systemd-nativeVyacheslav Yurkov2025-02-182-2/+0
| | | | | | | | | | The tests don't need it. On top of that, this extra requirement creates a dependency loop between systemd-systemctl-native and util-linux. (From OE-Core rev: b5770d8c56036bdfef8d596d27433d8f408ee36f) Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/cases/signing.py: Re-enable self-testZoltán Böszörményi2025-02-181-1/+2
| | | | | | | | | With all the pieces in place, the self test can be re-enabled. (From OE-Core rev: 46e8b94582ea9734117d20cd62c39fb4450c00c4) Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage add u-boot env scriptAdrian Freihofer2025-02-111-2/+42
| | | | | | | | | | | | | | | Extend all kernel-fitimage tests to add a boot.cmd script to the fitImage and verify the script ends up in the fitImage. The test covers the use case which is documented here: https://docs.u-boot.org/en/latest/usage/cmd/source.html#fit-image. But instead of adding the sctip inline a script file is generated. (From OE-Core rev: 816f25359935f54881b557d55c29a390dd991609) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fitimage split run_dumpimage functionAdrian Freihofer2025-02-111-21/+23
| | | | | | | | | | | Provide a re-usable function which calls the dumpimage utiliy. This is a refactoring. (From OE-Core rev: f69f54d72d89fb381ea1945ea52f4c9963552e9f) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "selftest/sstatetests: run CDN mirror check only once"Richard Purdie2025-02-061-0/+2
| | | | | | | | | | | This reverts commit 0d3901b768272abc2e27ba2ab807dad24917e0cf. We've still having CDN issues so go back to the double attempt to see if this improves things. (From OE-Core rev: 136f0edb75d9601a0ae56e025419d3cddfb90219) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Add a method to test a single recipeYoann Congal2025-02-051-2/+8
| | | | | | | | | | | | | | | Introduce a new variable "OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS". It can be used like OEQA_REPRODUCIBLE_TEST_TARGET but will try to use sstate for the dependencies. This can be used to "quickly" test the reproducibility of a single recipe. [YOCTO #15701] (From OE-Core rev: e2440618bd12d7288a7f3f7cfb92d377ae61a3b4) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Move a comment to follow the line it concernsYoann Congal2025-02-051-1/+1
| | | | | | | | | | The code line was moved in commit 5a9a5ad6a7be ("reproducibility: continue testing in case of build failure") (From OE-Core rev: b4b60941153ebeb5019b26eeed593d0a1cdb4012) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: also copy local changes from the 'scripts/' dirChris Laplante2025-02-011-1/+5
| | | | | | | | | | | | | | The 'devtool' cases make a copy of 'poky', but before this patch, that only included modifications to the 'meta/' subdirectory. It's very frustrating to make changes to scripts/ and have them be silently ignored by oe-selftest. (From OE-Core rev: 35de7080c53808ade526b3b97cb54f528357deca) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/recipetool: Update create_go test.Alexander Yurkov2025-01-271-196/+16
| | | | | | | | | | | | Adjust test_recipetool_go_create test to a smaller repository. Combine test_recipetool_go_create and test_recipetool_go_replace_modules tests into test_recipetool_go_create, the new test performs both tests. (From OE-Core rev: d1256f8ff4a94d4f32b2b9ca855359dd2d691beb) Signed-off-by: Alexander Yurkov <alexander.v.yurkov@gmail.com> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/virgl: add a link to RHEL merge request to address the missing vgem ↵Alexander Kanavin2025-01-231-0/+2
| | | | | | | | | | module (From OE-Core rev: ee674a0494a667a9fd2a3691618f736f6a58fb76) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Ross Burton <ross.burton@arm.com>
* oeqa/poisoning: fix gcc include poisoning testRoss Burton2025-01-221-3/+2
| | | | | | | | | | | | | | | | | | | | | The test code in poison was flawed: as long as one CPP/CC/CXX has fatal poisoning enabled then the test passes. However, at the moment due to a bad rebase only CPP has fatal poisoning and CC/CXX do not. Rewrite the do_compile() task to more carefully check the output so the test harness itself just has to bitbake the recipe. Note that this results in the test failing: ERROR: poison-1.0-r0 do_compile: C Compiler is not poisoned. Exit status 0, output: cc1: warning: include location "/usr/include" is unsafe for cross-compilation [-Wpoison-system-directories] ERROR: poison-1.0-r0 do_compile: C++ Compiler is not poisoned. Exit status 0, output: cc1plus: warning: include location "/usr/include" is unsafe for cross-compilation [-Wpoison-system-directories] (From OE-Core rev: 5b413d1fdb4bdbaec86d630bb52c3ccf68aae789) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Print error context on failureYoann Congal2025-01-141-7/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Printed logs during the reproducible test look like this in case of failure: 2025-01-12 00:39:00,807 - oe-selftest - INFO - Building reproducibleA (sstate allowed)... 2025-01-12 00:40:08,323 - oe-selftest - ERROR - Bitbake failed! but keep going... Log: 2025-01-12 00:40:08,323 - oe-selftest - INFO - Found ERROR (line 126): 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: recipe opkg-utils-native-0.7.0-r0: task do_populate_sysroot_setscene: Succeeded 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: Running setscene task 74 of 333 (virtual:native:.../poky/meta/recipes-devtools/opkg-utils/opkg-utils_0.7.0.bb:do_recipe_qa_setscene) 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: recipe bash-5.2.37-r0: task do_populate_sysroot_setscene: Succeeded 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: recipe cve-update-nvd2-native-1.0-r0: task do_unpack: Succeeded 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: Running task 4 of 974 (.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_cve_check) 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: recipe bash-5.2.37-r0: task do_packagedata_setscene: Succeeded 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_packagedata_setscene: Started 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: Running setscene task 77 of 333 (.../poky/meta/recipes-extended/bash/bash_5.2.37.bb:do_recipe_qa_setscene) 2025-01-12 00:40:08,323 - oe-selftest - INFO - NOTE: Running setscene task 78 of 333 (.../poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_populate_sysroot_setscene) 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_populate_sysroot_setscene: Started 2025-01-12 00:40:08,324 - oe-selftest - INFO - ERROR: hello-fail-1.0-r0 do_populate_sysroot_setscene: this is an error 2025-01-12 00:40:08,324 - oe-selftest - INFO - ERROR: hello-fail-1.0-r0 do_populate_sysroot_setscene: Error executing a python function in exec_func_python() autogenerated: 2025-01-12 00:40:08,324 - oe-selftest - INFO - 2025-01-12 00:40:08,324 - oe-selftest - INFO - The stack trace of python calls that resulted in this exception/failure was: 2025-01-12 00:40:08,324 - oe-selftest - INFO - File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 2025-01-12 00:40:08,324 - oe-selftest - INFO - 0001: 2025-01-12 00:40:08,324 - oe-selftest - INFO - *** 0002:do_populate_sysroot_setscene(d) 2025-01-12 00:40:08,324 - oe-selftest - INFO - 0003: 2025-01-12 00:40:08,324 - oe-selftest - INFO - File: '.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb', lineno: 17, function: do_populate_sysroot_setscene 2025-01-12 00:40:08,324 - oe-selftest - INFO - 0013:} 2025-01-12 00:40:08,324 - oe-selftest - INFO - Found ERROR (line 127): 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: Running setscene task 74 of 333 (virtual:native:.../poky/meta/recipes-devtools/opkg-utils/opkg-utils_0.7.0.bb:do_recipe_qa_setscene) 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe bash-5.2.37-r0: task do_populate_sysroot_setscene: Succeeded 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe cve-update-nvd2-native-1.0-r0: task do_unpack: Succeeded 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: Running task 4 of 974 (.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_cve_check) 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe bash-5.2.37-r0: task do_packagedata_setscene: Succeeded 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_packagedata_setscene: Started 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: Running setscene task 77 of 333 (.../poky/meta/recipes-extended/bash/bash_5.2.37.bb:do_recipe_qa_setscene) 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: Running setscene task 78 of 333 (.../poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_populate_sysroot_setscene) 2025-01-12 00:40:08,324 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_populate_sysroot_setscene: Started 2025-01-12 00:40:08,324 - oe-selftest - INFO - ERROR: hello-fail-1.0-r0 do_populate_sysroot_setscene: this is an error 2025-01-12 00:40:08,324 - oe-selftest - INFO - ERROR: hello-fail-1.0-r0 do_populate_sysroot_setscene: Error executing a python function in exec_func_python() autogenerated: 2025-01-12 00:40:08,325 - oe-selftest - INFO - 2025-01-12 00:40:08,325 - oe-selftest - INFO - The stack trace of python calls that resulted in this exception/failure was: 2025-01-12 00:40:08,325 - oe-selftest - INFO - File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 2025-01-12 00:40:08,325 - oe-selftest - INFO - 0001: 2025-01-12 00:40:08,325 - oe-selftest - INFO - *** 0002:do_populate_sysroot_setscene(d) 2025-01-12 00:40:08,325 - oe-selftest - INFO - 0003: 2025-01-12 00:40:08,325 - oe-selftest - INFO - File: '.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb', lineno: 17, function: do_populate_sysroot_setscene 2025-01-12 00:40:08,325 - oe-selftest - INFO - 0013:} 2025-01-12 00:40:08,325 - oe-selftest - INFO - 0014: 2025-01-12 00:40:08,325 - oe-selftest - INFO - End of log: 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: Running task 955 of 974 (.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_populate_sysroot) 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_populate_sysroot: Started 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: recipe hello-fail-1.0-r0: task do_populate_sysroot: Succeeded 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: Running noexec task 973 of 974 (.../poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_build) 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: Running noexec task 974 of 974 (.../poky/meta/recipes-core/hello-single/hello-norepro_1.0.bb:do_build) 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 974 tasks of which 958 didn't need to be rerun and all succeeded. 2025-01-12 00:40:08,325 - oe-selftest - INFO - NOTE: Generating JSON CVE summary 2025-01-12 00:40:08,325 - oe-selftest - INFO - Complete CVE JSON report summary created at: .../poky/build-master-st/reproducibleA/tmp/log/cve/cve-summary.json 2025-01-12 00:40:08,325 - oe-selftest - INFO - 2025-01-12 00:40:08,325 - oe-selftest - INFO - Summary: There were 2 ERROR messages, returning a non-zero exit code. 2025-01-12 00:40:08,326 - oe-selftest - ERROR - reproducibleA build failed. Trying to compute built packages differences but the test will fail. Fixes [YOCTO #15655] (second half). (From OE-Core rev: b3bc820e58b65a976e0abce85fcdffd067697ecf) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Print full log path on failureYoann Congal2025-01-141-1/+3
| | | | | | | | | | | | | Printed message looks like this: 2025-01-12 00:40:08,326 - oe-selftest - ERROR - reproducibleA build failed. Trying to compute built packages differences but the test will fail. 2025-01-12 00:40:08,326 - oe-selftest - INFO - Failure log for reproducibleA will be copied to <snip>/oe-reproducible-20250112-mza9w940/bitbake-reproducibleA.log Fixes [YOCTO #15655] (first half). (From OE-Core rev: 1c2287849a2742445d80c81d6a1c6c3e3668765e) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/sstatetests: Cover aarch64 SDKMACHINE in testsRichard Purdie2025-01-141-9/+17
| | | | | | | | | We want to ensure ARM nativesdk signatures match those for the 32 and 64 bit x86 versions. Add a test to ensure this is the case. (From OE-Core rev: dced735e2cc9663527ff8b386bf20a5e1fb9ab07) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: Rework patch parsingColin McAllister2025-01-081-0/+205
| | | | | | | | | | | | | | | | | | | | | The cve_check functionality to parse CVE IDs from the patch filename and patch contents have been reworked to improve parsing and also utilize tests. This ensures that the parsing works as intended. Additionally, the new patched_cves dict has a few issues I tried to fix as well. If multiple patch files exist for a single CVE ID, only the last one will show up with the "resource" key. The value for the "resource" key has been updated to hold a list and return all patch files associated with a given CVE ID. Also, at the end of get_patch_cves, CVE_STATUS can overwrite an existing entry in the dict. This could cause an issue, for example, if a CVE has been addressed via a patch, but a CVE_STATUS line also exists that ignores the given CVE ID. A warning has been added if this ever happens. (From OE-Core rev: 87c6da681609b4f8e048eca2a27ae8e068c724e1) Signed-off-by: Colin McAllister <colinmca242@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: devtool ide-sdk use modify debug-buildAdrian Freihofer2024-12-171-1/+1
| | | | | | | (From OE-Core rev: 9ab1e35a0969af144caedcc25cd095a17d758bf9) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-config-build: add a plugin for config fragmentsAlexander Kanavin2024-12-131-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | This allows fine-tuning local configurations with pre-frabricated configuration snippets in a structured, controlled way. It's also an important building block for bitbake-setup. The tool requires that each fragment contains a one-line summary, and one or more lines of description, as BB_CONF_FRAGMENT_SUMMARY style metadata. There are three (and a half) operations (list/enable/disable/disable all), and here's the 'list' output: alex@Zen2:/srv/storage/alex/yocto/build-64$ bitbake-config-build list-fragments NOTE: Starting bitbake server... Available fragments in selftest layer located in /srv/work/alex/poky/meta-selftest: Enabled fragments: selftest/test-fragment This is a configuration fragment intended for testing in oe-selftest context Unused fragments: selftest/more-fragments-here/test-another-fragment This is a second configuration fragment intended for testing in oe-selftest context (From OE-Core rev: fdb611e13bd7aa00360d3a68e4818ef5f05c8944) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rpm: update 4.19.1.1 -> 4.20.0Alexander Kanavin2024-12-121-0/+1
| | | | | | | | | | | | | | | | | | | | | With this release, as expected, deprecated openpgp support has been fully removed and replaced with sequoia: https://github.com/rpm-software-management/rpm-sequoia/ Alas, it's written in rust, and has no recipe, so for now signing rpm packages has to be disabled. Remove package name parameter to %prep in source archiving, as it's been dropped in the new rpm, and wasn't needed to begin with. Drop 0001-perl-disable-auto-reqs.patch (files removed upstream). (From OE-Core rev: 8c15b4577d5e554cc2dd5adfb88b816894b05a9a) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Fix failure when configuration contains BBLAYERS:appendOleksandr Hnatiuk2024-12-121-0/+1
| | | | | | | | | | | | | | | | | | | | | When used with `--newbuilddir` option, it replaces relative paths in BBLAYERS variable with absolute paths by evaluating the final value of the variable, converting paths and saving result by assignment (`BBLAYERS =`) at the end of bblayers.conf. This breaks tests when bblayers.conf contains BBLAYERS:append because :append statements are evaluated after all assignments and we end up with the appended layer added twice - first by evaluating final value of the variable, then again by bitbake evaluating BBLAYERS:append. The error is as follows: AssertionError: Command 'bitbake -e' returned non-zero exit status 1: ERROR: Found duplicated BBFILE_COLLECTIONS 'layer-name', check bblayers.conf or layer.conf to fix it. (From OE-Core rev: 516c8e8f722ed0eb85a715a23b5e68d05abe9275) Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* spdx 3.0: Rework how SPDX aliases are linkedJoshua Watt2024-12-121-13/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The SPDX code needs to be able to look up an Element by its SPDX ID, locating the file that (should) contain the SPDX ID and opening it for parsing. Previously, the code would do this be hashing each Element SPDX ID and Alias, and the creating a symbolic link to the file that contains the element with a name of the hash. This worked well as it was possible to look up any arbitrary SPDX ID or alias by simply hashing it and following the symbolic link to get the file. However, the down side of this approach is that it creates a lot of symbolic links, since it will make one or two per Element in the document. This can be a problem when using SPDX_INCLUDE_SOURCES, for example. This change reworks this strategy so that the only Element that gets a symbolic link based on the hash is the singular SpdxDocument that is create for each file. All other Elements are assigned an alias with a special prefix that encodes the hash of SpdxDocument alias. Thus, when attempting to look up an arbitrary alias, the code sees the special prefix, extract the hash, opens the file based on the symlink with that hash name, then finds the matching Element in the file. This drastically reduces the number of symbolic links by making only one per file. This also means that the custom link extension can be removed since it is now superfluous. (From OE-Core rev: 838d64c09657ac53175737fc4e7fd6f01f3dcf47) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib: oeqa: spdx: Add tests for extra optionsJoshua Watt2024-12-051-14/+104
| | | | | | | | | | | | | | | Adds a test for several of the extra options provided by the SPDX classes. In particular, these are the options that can produce non-reproducible results, so are not enabled by default in OE core. This test takes care to configure the build so that the tests do run in a reproducible manner so that pre-built test objects can be pulled from sstate (From OE-Core rev: 14f534f15f7fe6362723d7f064d39783c5bd758f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Clean up pathnamesRichard Purdie2024-11-221-1/+1
| | | | | | | | | | | | | | | | | There are several problems with these paths. Firstly they contain full system paths which depend upon where the test was run. These are pretty pointless and just take up a lot of space making the results files large. Secondly, they contain the same path twice. The reference and target path will always be the same thing in two different locations. Strip off the prefix and remove the duplication. This does change the output data but that can't really be avoided. It does shrink the results data and makes it more readable. (From OE-Core rev: 81a44de36e864b08687451fd85aeba7c529fd7f7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Drop rawlogsRichard Purdie2024-11-221-6/+0
| | | | | | | | | | | | | The "rawlogs" data consists of a long string of results data which is already in a structured data format. I can't see this is adding much value in duplciating the data but it does create a huge string with a lot of long problematic pathnames and inflates the results data size. I suggest we drop this data as obsolete and not necessary. (From OE-Core rev: 5b2c70fab2ffa409b861d83f048b65d458d03a90) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* qemu/testimage: Ensure GNU tar is in test imagesRichard Purdie2024-11-211-0/+3
| | | | | | | | | | | | | | | | We've seen cases where a ptest (strace) has a 43GB sparse file in the test directory. busybox tar doesn't work well with this. The resulting 1.4GB archive takes hours to extract too. Ensure tar is added to our full images and use the sparse option to collect files for debugging. This stops crazy build hangs. Since tar is GPLv3, we have to exclude it from that test code. We don't boot any of those images so the debug collection code is safe there, at least for now. (From OE-Core rev: fefeb919696b6ac76f0997acfb0f612203ef7f1b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: add a test for bitbake "-e" and "-getvar" differenceYoann Congal2024-11-181-0/+18
| | | | | | | | | | This is a non-regression test for [YOCTO #15638] (From OE-Core rev: 22b508da24e0f7e5ad8ce4e090832bd0829963f0) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rust: Remove passing test case from the exclude list in the Rust OE self-test.Deepesh Varatharajan2024-11-121-1/+0
| | | | | | | | | | | | | | We had previously excluded this test case due to failure in earlier versions. However, with the latest version of Rust, this test case is passing. As a result, we have removed it from the exclude list and added back into the test suite for execution. The rust-1.79 has been successfully tested with the current test inclusion. (From OE-Core rev: a33edee3efe67f81229496e03a098454dc7df051) Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/container: fix IMAGE_FEATURES assignmentRoss Burton2024-11-061-3/+0
| | | | | | | | | | | | | | | | | | | | Assigning IMAGE_FEATURES="" in local.conf doesn't actually do anything useful, as bitbake.conf has IMAGE_FEATURES += EXTRA_IMAGE_FEATURES after the include of local.conf. In this test case, this results in IMAGE_FEATURES still using EXTRA_IMAGE_FEATURES, so the image contains the files installed by the post-install-logging feature. As the intention here is that container-test-image is as minimal as possible, move the IMAGE_FEATURES assignment into the image itself so it actually works, and update the expected file list to remove the post-install-logging files. (From OE-Core rev: c65349e1238ddc6634dfa4759c57e6168a355200) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/reproducibility: restart bitbake for each buildYoann Congal2024-11-051-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since [0], the reproducibility test tries to save logs of failed builds. Using the memory persistent bitbake prevent this by concatenating logs in the single persistent instance (See [YOCTO #15641]). So, force a BitBake server restart for each build to ensure a new log file is created. This fixes this error (seen on AB [1]): 2024-11-02 08:16:00,816 - oe-selftest - ERROR - reproducibleA build failed. Trying to compute built packages differences but the test will fail. 2024-11-02 08:16:00,819 - oe-selftest - INFO - ... ERROR 2024-11-02 08:16:00,819 - oe-selftest - INFO - Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 321, in test_reproducible_builds self.copy_file(variables["BB_CONSOLELOG"], os.path.join(save_dir, "bitbake-%s.log" % name)) File "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 216, in copy_file shutil.copyfile(source, dest) File "/usr/lib/python3.11/shutil.py", line 256, in copyfile with open(src, 'rb') as fsrc: ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-python/build-st/tmp/log/cooker/qemux86-64/20241102071141.log' [0]: https://git.openembedded.org/openembedded-core/commit/?id=c78cc753843d4199443052e8deb0c9c3b7e4b580 [1]: https://valkyrie.yoctoproject.org/#/builders/87/builds/17/steps/40/logs/stdio (From OE-Core rev: e89bbc00ba16574d719b199c01ffbf37646f4f54) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: wic: wic_sector_size add wic-tools pathVince Chang2024-11-011-44/+48
| | | | | | | | | | Add wic-tools to the PATH to avoid failures when running the wic_sector_size test case on a build host that doesn’t have parted. (From OE-Core rev: 013dcdf75669421bc38d699263cb1e8d5b95d398) Signed-off-by: Vince Chang <vince_chang@aspeedtech.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: wic: remove 2>/dev/null to help fail diagnosticVince Chang2024-11-011-21/+23
| | | | | | | | | | | | | | The wic test case previously used '2>/dev/null' to suppress error messages. This commit updates the code to use 'stderr=subprocess.PIPE' when calling runCmd(). Refer: https://lists.openembedded.org/g/openembedded-core/topic/109308684 (From OE-Core rev: bd26d999a0ba1107ee5629a8e238f4fe945e9be5) Signed-off-by: Vince Chang <vince_chang@aspeedtech.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/sstatetests: run CDN mirror check only onceAlexander Kanavin2024-11-011-2/+0
| | | | | | | | | | | The first no-fail check was an attempt to work around the old CDN's instability (and it didn't really help); it should not be necessary with the new CDN, and only delays a-full completion. (From OE-Core rev: 7f75c42b7fcf60a9ca58d3ded9047df675d76dc2) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Add SPDX 3.0 include source case for work-shareHongxu Jia2024-11-011-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | Build gcc and check gcc-14.2.0/README in objset is available $ oe-selftest -r spdx.SPDX30Check.test_gcc_include_source ... 2024-10-26 01:24:57,063 - oe-selftest - INFO - test_gcc_include_source (spdx.SPDX30Check.test_gcc_include_source) 2024-10-26 01:28:24,204 - oe-selftest - INFO - The spdxId of gcc-14.2.0/README in gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/88d5068ffd41e5ea6b4e0dd390b23bf499bb2b6674a41e09eaf2a887eced16c8/sourcefile/42 2024-10-26 01:28:26,369 - oe-selftest - INFO - ... ok 2024-10-26 01:28:33,315 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-10-26 01:28:33,316 - oe-selftest - INFO - Ran 1 test in 216.457s 2024-10-26 01:28:33,316 - oe-selftest - INFO - OK 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS: 2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_gcc_include_source: PASSED (209.31s) 2024-10-26 01:28:45,260 - oe-selftest - INFO - SUMMARY: 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 216.457s 2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0) (From OE-Core rev: ccd6dde301dc8c45c8f901ebd4676b488d638b08) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Reviewed-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducibility: continue testing in case of build failureYoann Congal2024-11-011-9/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current reproducibility test stops all build tasks when a single task fails (default BitBake behavior). This means that a single build failure prevents the reproducibility of other packages from being tested, which is not ideal. To address this, we now use the --continue option of BitBake during test builds, allowing the build process to proceed even when some tasks fail. The failure cases are handled as gracefully as possible. In the event of a build failure: - The entire reproducibility test will be considered a failure. - The complete BitBake log will be saved in the "saved output" directory to facilitate debugging. On the autobuilder, this log should be available at https://autobuilder.yocto.io/pub/repro-fail/. - The last 20 lines of the log, which should show the failing recipes, will be displayed in the oe-selftest console. - If BitBake fails to create the deployment directory, it will be manually created to allow the comparison process to proceed. Here is what the output looks like when testing reproducibility of bash, a failing recipe (hello-fail) and a non-reproducible recipe (hello-norepro): <snip> 2024-10-01 23:09:33,977 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) 2024-10-01 23:10:39,093 - oe-selftest - INFO - Non-reproducible packages will be copied to <snip>/yocto/repro_output/oe-reproducible-20241001-ng43_hzd 2024-10-01 23:10:39,094 - oe-selftest - INFO - Building reproducibleA (sstate allowed)... 2024-10-01 23:18:12,378 - oe-selftest - ERROR - Bitbake failed! but keep going... Log: 2024-10-01 23:18:12,379 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: 2024-10-01 23:18:12,379 - oe-selftest - INFO - 2024-10-01 23:18:12,379 - oe-selftest - INFO - ... (last 20 lines of output) 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: Running task 976 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_package_qa) 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_populate_sysroot: Started 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_populate_sysroot: Succeeded 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: Running setscene task 342 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx_setscene) 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Started 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Succeeded 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: Running setscene task 343 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_spdx_setscene) 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Started 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Succeeded 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded 2024-10-01 23:18:12,379 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) 2024-10-01 23:18:12,380 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 841 didn't need to be rerun and 1 failed. 2024-10-01 23:18:12,380 - oe-selftest - INFO - NOTE: Generating JSON CVE summary 2024-10-01 23:18:12,380 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/log/cve/cve-summary.json 2024-10-01 23:18:12,380 - oe-selftest - INFO - 2024-10-01 23:18:12,380 - oe-selftest - INFO - Summary: 1 task failed: 2024-10-01 23:18:12,380 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile 2024-10-01 23:18:12,380 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.3221257 2024-10-01 23:18:12,380 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. 2024-10-01 23:18:12,380 - oe-selftest - ERROR - reproducibleA build failed. Trying to compute built packages differences but the test will fail. 2024-10-01 23:18:12,382 - oe-selftest - INFO - Building reproducibleB-extended (sstate NOT allowed)... 2024-10-01 23:46:58,451 - oe-selftest - ERROR - Bitbake failed! but keep going... Log: 2024-10-01 23:46:58,463 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: 2024-10-01 23:46:58,463 - oe-selftest - INFO - 2024-10-01 23:46:58,463 - oe-selftest - INFO - ... (last 20 lines of output) 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_ipk: Started 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx: Succeeded 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: Running task 978 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx) 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Started 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Succeeded 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Succeeded 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_ipk: Succeeded 2024-10-01 23:46:58,463 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_rpm: Succeeded 2024-10-01 23:46:58,464 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) 2024-10-01 23:46:58,464 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 2 didn't need to be rerun and 1 failed. 2024-10-01 23:46:58,464 - oe-selftest - INFO - NOTE: Generating JSON CVE summary 2024-10-01 23:46:58,464 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/log/cve/cve-summary.json 2024-10-01 23:46:58,464 - oe-selftest - INFO - 2024-10-01 23:46:58,464 - oe-selftest - INFO - Summary: 1 task failed: 2024-10-01 23:46:58,464 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile 2024-10-01 23:46:58,464 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.4136075 2024-10-01 23:46:58,464 - oe-selftest - INFO - Summary: There were 5 WARNING messages. 2024-10-01 23:46:58,464 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. 2024-10-01 23:46:58,467 - oe-selftest - ERROR - reproducibleB-extended build failed. Trying to compute built packages differences but the test will fail. 2024-10-01 23:46:58,481 - oe-selftest - INFO - Checking deb packages for differences... 2024-10-01 23:46:58,586 - oe-selftest - INFO - Reproducibility summary for deb: same=52 different=1 different_excluded=0 missing=0 total=53 unused_exclusions=[] 2024-10-01 23:46:58,588 - oe-selftest - INFO - Checking ipk packages for differences... 2024-10-01 23:46:58,658 - oe-selftest - INFO - Reproducibility summary for ipk: same=52 different=1 different_excluded=0 missing=0 total=53 unused_exclusions=[] 2024-10-01 23:46:58,659 - oe-selftest - INFO - Checking rpm packages for differences... 2024-10-01 23:46:58,691 - oe-selftest - INFO - Reproducibility summary for rpm: same=52 different=1 different_excluded=0 missing=0 total=53 unused_exclusions=[] 2024-10-01 23:46:58,692 - oe-selftest - INFO - Running diffoscope 2024-10-01 23:46:59,765 - oe-selftest - INFO - ... FAIL 2024-10-01 23:46:59,766 - oe-selftest - INFO - Traceback (most recent call last): File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds self.fail('\n'.join(fails)) AssertionError: Bitbake reproducibleA failure Bitbake reproducibleB-extended failure The following deb packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb The following ipk packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk The following rpm packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm 2024-10-01 23:46:59,769 - oe-selftest - INFO - ====================================================================== 2024-10-01 23:46:59,770 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) 2024-10-01 23:46:59,770 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-10-01 23:46:59,770 - oe-selftest - INFO - Traceback (most recent call last): File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds self.fail('\n'.join(fails)) AssertionError: Bitbake reproducibleA failure Bitbake reproducibleB-extended failure The following deb packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb The following ipk packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk The following rpm packages are different and not in exclusion list: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm 2024-10-01 23:46:59,770 - oe-selftest - INFO - ---------------------------------------------------------------------- 2024-10-01 23:46:59,770 - oe-selftest - INFO - Ran 1 test in 2246.039s 2024-10-01 23:46:59,770 - oe-selftest - INFO - FAILED 2024-10-01 23:46:59,770 - oe-selftest - INFO - (failures=1) 2024-10-01 23:47:03,200 - oe-selftest - INFO - RESULTS: 2024-10-01 23:47:03,200 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2245.79s) 2024-10-01 23:47:03,203 - oe-selftest - INFO - SUMMARY: 2024-10-01 23:47:03,203 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2246.040s 2024-10-01 23:47:03,203 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0) => Test failed but hello-norepro is displayed as non-reproducible. The testresult.json contains: { "oeselftest_debian-12_qemux86-64_20240930000424": { "configuration": { <snip> }, "result": { "reproducible": { "files": { "package_deb": { "different": [ { "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb", "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb" } ], "different_excluded": [], "missing": [], "same": [ <snip> { "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb", "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb" }, <snip> ] }, "package_ipk": { <same as deb> }, "package_rpm": { <same as deb> } } }, "reproducible.ReproducibleTests.test_reproducible_builds": { "duration": 2146.5671875476837, "log": "Traceback (most recent call last):\n File \"<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py\", line 380, in test_reproducible_builds\n self.fail('\\n'.join(fails))\nAssertionError: Bitbake reproducibleA failure\nBitbake reproducibleB-extended failure\nThe following deb packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nThe following ipk packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk\nThe following rpm packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm\n", "status": "FAILED" }, "reproducible.rawlogs": { "log": "DIFFERENT: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nSAME: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb\n<snip>" } } } } => "reproducible.ReproducibleTests.test_reproducible_builds".status is correctly "FAILED" but the reproducibility of bash and hello-norepro is tested. (From OE-Core rev: c78cc753843d4199443052e8deb0c9c3b7e4b580) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Update the BB_HASHSERVE_UPSTREAMJose Quaresma2024-10-312-2/+2
| | | | | | | (From OE-Core rev: ec7b024725fc93b7fef881d41343c8b77d97a57c) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* rust: Remove passing test cases from the exclude list in the Rust OE self-test.Deepesh Varatharajan2024-10-291-3/+0
| | | | | | | | | | | | | | | | | There were some rust tests which were FAILing only for mips architectures. Hence, they were added to the exclusion list to avoid the failures when running rust oe-selftest for mips architectures.The current yocto framework disabled the rust oe-selftest for mips architectures.Hence, these tests have been added back to the test framework. The rust-1.79 has been successfully tested on X86, X86_64, ARM, AARCH64 and PPC Architectures with the current test inclusions. (From OE-Core rev: e1ec1ad75feb1321228bf3b52251e1aecfdb8746) Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-06 01:09:17 +0000