summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: test/fetch: Switch u-boot based test to use our own mirrorRichard Purdie7 days1-1/+1
| | | | | | | | | | The upstream servers are having issues so switch to our own shadow copy of the repo. (Bitbake rev: 8e2d1f8de055549b2101614d85454fcd1d0f94b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cache: bump cache versionSteve Sakoman2025-03-271-1/+1
| | | | | | | | | | | Version bump is needed after "siggen.py: Improve taskhash reproducibility" otherwise there will be taskhash mismatch errors (Bitbake rev: 046871d9fd76efdca7b72718b328d8f545523f7e) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: siggen.py: Improve taskhash reproducibilityPaulo Neves2025-03-191-2/+8
| | | | | | | | | | | | | | | | | file checksums are part of the data checksummed to generate the task hash. The list of file checksums was not ordered. In this commit we make sure the task hash checksum takes a list of checksum data that is ordered by unique file name thus guaranteeing reproducibility. (Bitbake rev: da5f41996687e18b78d9c9845e621d832115aa1e) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48Richard Purdie2025-02-051-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | >From git 2.48 release notes: """ When "git fetch $remote" notices that refs/remotes/$remote/HEAD is missing and discovers what branch the other side points with its HEAD, refs/remotes/$remote/HEAD is updated to point to it. """ This means with git 2.48 onwards, there is a mystery "HEAD" revision appearing in some of our shallow clone tests. We can avoid this by using the same canonicalization as used for the reference revisions. This resolves autobuilder failures on the Fedora 40 workers. (Bitbake rev: e71f1ce53cf3b8320caa481ae62d1ce2900c4670) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Increase timeout to 100s from 30sRichard Purdie2024-12-021-2/+2
| | | | | | | | | | | Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 3f88b005244a0afb5d5c7260e54a94a453ec9b3e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: use persist_data context managersChris Laplante2024-12-021-31/+31
| | | | | | | | | | | | | | | | | | Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 9789c55ecc90ba074596061fa16e90d3e8accb02) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> (cherry picked from commit 61f803c7d92a012b62837b0cdae4789a394b260e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: persist_data: close connection in SQLTable __exit__Chris Laplante2024-12-021-0/+1
| | | | | | | | | | | | | | | A Connection's __exit__ does not close the connection, but rather deals with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 97856a56cbf09401b34d6c9bb5444c07a88772ce) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> (cherry picked from commit b0fba0f5af555ff970406b5cf3bf8fb113d953d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser: Fix handling of string AST nodes with older Python versionsPhilip Lorenz2024-11-151-13/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Commits 4591011449212c8e494ea42348acb2d27a82a51b and 6c19b6cf105ac321ec89da1a876a317020c45ab7 unconditionally changed codeparser to rely on CPython 3.8 semantics. However, kirkstone continues to support CPython versions >= 3.6.0 and as such string AST nodes were no longer correctly identified. Fix this by continuing to use `ast.Str` for Python versions < 3.8.0 and only using the new code path for more recent versions. Detecting which version of the AST API to use seems to be non-trivial so the Python feature version is used instead. Instances of this issue can be identified when executing bitbake with debug logging: while parsing MACHINE_ARCH, in call of d.getVar, argument ''TUNE_PKGARCH'' is not a string literal As a consequence of these parsing issues, bitbake may assume that task inputs haven't changed and as such erroneously reuse sstate objects when it shouldn't. (Bitbake rev: fb73c495c45d1d4107cfd60b67a5b4f11a99647b) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | decouple from gnome gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 0c30e9aadd30fc6f0dcf811eb8340687b52eb00b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: e4ec4267e4c0818a1682f8a1a4bf3d1419e509a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-11-021-3/+5
| | | | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 15582daed9a18330bcf1ad316a57d46571bbf7c6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-11-021-28/+14
| | | | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: ff2dfda55258d8034ea748d87222e51124a03f02) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake: doc/user-manual: Update the BB_HASHSERVE_UPSTREAMJose Quaresma2024-11-021-1/+1
| | | | | | | | (Bitbake rev: c092f7e6c5e07a829173b25e591ab0326e9dcb67) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-11-021-2/+1
| | | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 53264dc14554890b3a2afc83cb1749cf10d86854) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: hashserv: tests: Omit client in slow server start testJoshua Watt2024-10-101-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | On Fedora 39 and Fedora 40 hosts, this version of the hash server exhibits different behavior on exit when using Unix Domain sockets. Instead of closing the client connections and exiting immediately, the server will wait until all clients have disconnected before exiting. It is unknown why this changed, or why it only affects Unix Domain sockets and not TCP sockets. Because of this behavior change, the Slow Server Start test is failing on these hosts. This test is primarily concerned with ensuring that the server will actually exit, even if it gets a termination signal before it enters its main loop, and doesn't really care about clients. As such, modify the test so that a client is not pre-connected to the server. This allows the server to actually exit so that the test can verify the signal behavior. The latest version of the hash equivalence server (on master) does not exhibit this behavior. Speculation is that the more standardized usage of asyncio allows the server to properly exit, even if clients are still connected. Regardless, this patch is only intended for the older versions, and should not be applied to master (Bitbake rev: eb5c1ce6b1b8f33535ff7b9263ec7648044163ea) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-10-101-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 07081a94997142746f7d345c27bc6805231d025d) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman <steve@sakoman.com> (cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Move files into place atomicallyRichard Purdie2024-10-101-4/+7
| | | | | | | | (Bitbake rev: aaf9c1b8673f5b181255a08eee4e9a342863e896) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak to work on Fedora40Richard Purdie2024-10-101-3/+3
| | | | | | | | | | | | | | On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. (Bitbake rev: 3147ebb0edcca8f7f850aabd86ef010ab593496f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-10-101-1/+4
| | | | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: 477f90f9515cb7cd093920cee5518f662cef2e05) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart: Improve performance for VariableHistoryRobert Yang2024-08-231-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: ec2a99a077da9aa0e99e8b05e0c65dcbd45864b1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak test to match upstream repo url change Upstream ↵Steve Sakoman2024-06-241-1/+1
| | | | | | | | | changed their urls, update our test to match. (Bitbake rev: 734b0ea3dfe45eb16ee60f0c2c388e22af4040e0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: parse: Improve/fix cache invalidation via mtimeRichard Purdie2024-05-291-3/+6
| | | | | | | | | | | | | | | | | | | | We have been seeing obscure failures in devtool, particularly on newer autobuilder workers where it appears the cache is assumed to be valid when it shouldn't be. We're using the 'seconds' granulation mtime field which is not really a good way of telling if a file has changed. We can switch to the "ns" version which is better however also add in inode number and size as precautions. We already have all this data and tuples are fast so there isn't really any cost to do so. This hopefully fixes [YOCTO #15318]. (Bitbake rev: 5a90927f31c4f9fccbe5d9d07d08e6e69485baa8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-03-012-6/+90
| | | | | | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae) (Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2024-03-011-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (cherry picked from commit d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) (Bitbake rev: 2a6fd774405a58f3ef2953a0dc3eca063e0cf890) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2024-03-011-16/+24
| | | | | | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (cherry picked from commit 70f848631450bd723c223227c21c60e815ee033d) (Bitbake rev: 1d5d4a46c772f44ab652f95b37f508b890828e67) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake/lib/bs4/tests/test_tree.py: python 3.12 regexAdrian Freihofer2024-03-011-1/+1
| | | | | | | | | | | | | Python 3 interprets string literals as Unicode strings, and therefore \s is treated as an escaped Unicode character which is not correct. Declaring the RegEx pattern as a raw string instead of unicode is required for Python 3. (Bitbake rev: 7efed7bc09c56e41d3074a26388581a62f145625) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake/codeparser.py: address ast module deprecations in py 3.12Alexander Kanavin2024-03-011-10/+10
| | | | | | | | | | | | | | Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s (Bitbake rev: 6c19b6cf105ac321ec89da1a876a317020c45ab7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser: replace deprecated ast.Str and 's'Chris Laplante2024-03-011-2/+2
| | | | | | | | | | These have been deprecated since 3.8 (Bitbake rev: 4591011449212c8e494ea42348acb2d27a82a51b) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toaster/toastergui: Bug-fix verify given layer path only if ↵Alassane Yattara2024-02-071-5/+8
| | | | | | | | | | import/add local layer (Bitbake rev: a4c516ef5e72b2d77ac5ff7e86c5ee2190ebc42f) Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: asyncrpc: Add context manager APIJoshua Watt2024-01-103-22/+33
| | | | | | | | | | | | | | | | | | Adds context manager API for the asyncrcp client class which allow writing code that will automatically close the connection like so: with hashserv.create_client(address) as client: ... Rework the bitbake-hashclient tool and PR server to use this new API to fix warnings about unclosed event loops when exiting (Bitbake rev: ee090484cc25d760b8c20f18add17b5eff485b40) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: lib/bb: Add workaround for libgcc issues with python 3.8 and 3.9Richard Purdie2024-01-041-0/+7
| | | | | | | | | | | | | | | | | | | | | With python 3.8 and 3.9, we see intermittent errors of: libgcc_s.so.1 must be installed for pthread_cancel to work Aborted (core dumped) which seem related to: https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work https://bugs.ams1.psf.io/issue42888 These tend to occur on debian 11 and ubuntu 20.04. Workaround this by ensuring libgcc is preloaded in all cases. (Bitbake rev: 2c6183594279e2e9d03f11155ad969448869c863) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data: Add missing dependency handling of remove operatorInsu Park2024-01-042-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | A recipe variable handles its dependencies even on the "contains" variables within the "inline Python expressions" like bb.utils.filter(). And it also handles those in the append operator correctly, but the problem is that it does not so in the remove operator. Fix it by adding the missing dependencies every time the remove operator has been handled. Also add a test case to check if the override operators handle dependencies correctly. (Bitbake rev: 23639edfbbb3fced7606dce211db8a31c5766585) Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Cherry-picked from master: b90520eedb1dbc7f6a3928d089fe74fafb864eb5 - Conflicts in data.py are resolved as the master branch moved handle_contains() and handle_remove() out of the try block. - The test code in codeparser.py are modified as the master branch added three more arguments to the build_dependencies(). Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toastergui: verify that an existing layer path is givenMarta Rybczynska2023-12-211-6/+7
| | | | | | | | | | | | | | Verify that an existing layer path was given when adding a new layer. Manually using the shell for globbing is unnecessary, use the glob function instead for cleaner code. (Bitbake rev: de1cf0e31f3836a449cbd17490a2f6b6a86b17f9) Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: show number of currently running bitbake threads when ↵Martin Jansa2023-12-111-1/+1
| | | | | | | | | | | | | | | | | | pressure changes * it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads (Bitbake rev: 42a1c9fe698a03feb34c5bba223c6e6e0350925b) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: show more pressure dataMartin Jansa2023-12-111-8/+10
| | | | | | | | | | | | | | * with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once (Bitbake rev: 1050145c3004861ebede4777fd4fbd89d0470716) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue.py: fix PSI check logicChen Qi2023-12-111-4/+5
| | | | | | | | | | | | | | | | | | | The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. (Bitbake rev: 76889ff0a8938a3d77603d2af176aa9e264df839) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Add pressure change loggingRichard Purdie2023-12-111-0/+4
| | | | | | | | | | | It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. (Bitbake rev: b5f77e8159ad321f31999af8304f082a2c56b537) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-12-111-4/+9
| | | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: 2b2997ab6bdda730e4b638f416311a73e0c42156) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: Fix disk space monitoring on cephfsSamantha Jalabert2023-11-091-3/+4
| | | | | | | | | | | | | Error occured while running bitbake on cephfs: WARNING: The free inode of path is running low (-0.001K left) ERROR: Immediately halt since the disk space monitor action is "HALT"! (Bitbake rev: a7f6c3e67bd9170e93b2b94676e84018faf0df91) Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: convert deferral messages from bb.note to bb.debugDenys Dmytriyenko2023-11-091-3/+3
| | | | | | | | | | | | | | | | | Using multiconfig to target baremetal pieces of the system and building corresponding toolchains for them results in hundreds and hundreds of "Deferring %s after %s" and "Deferred task %s now buildable". To clean up the output and to reduce risk of missing important warnings, convert these notice messages to debug messages. (Bitbake rev: 3505d8d8c02b041946670ab6bc5751e54fe292ff) Signed-off-by: Denys Dmytriyenko <denis@denix.org> Signed-off-by: Denys Dmytriyenko <denys@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tinfoil: Do not fail when logging is disabled and full config is usedPeter Kjellerstedt2023-11-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | If Tinfoil is initialized with setup_logging = False and Tinfoil.prepare() is called with config_only = False, then it fails because self.localhandlers is only initialized when setup_logging = True. This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`: Traceback (most recent call last): File ".../bitbake/bin/bitbake-getvar", line 41, in <module> tinfoil.prepare(quiet=2) File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare for handler in self.localhandlers: AttributeError: 'Tinfoil' object has no attribute 'localhandlers'. Did you mean: 'oldhandlers'? (Bitbake rev: e452c6d7ba5bb4f78a1d2bfb742794efdf171dbc) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 616101ddb630e2c9975022068b52a87c4cf647f6) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-getvar: Make --quiet work with --recipePeter Kjellerstedt2023-11-091-2/+3
| | | | | | | | | | | | | Initializing Tinfoil with setup_logging = False only has an effect when recipe parsing is not needed. To make it work regardless of if --recipe is used, manipulate the quiet parameter to Tinfoil.prepare() instead. (Bitbake rev: 161ab0d5bab74732e12d490cee50e14295be0a9f) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 71ee69a20f21f3d37f4f060a7d8e87d9f1dc6aa1) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: SECURITY.md: add fileMarta Rybczynska2023-10-241-0/+24
| | | | | | | | | | | Add a SECURITY.md file with hints for security researchers and other parties who might report potential security vulnerabilities. (Bitbake rev: 6c1ffa9091d0c53a100e8c8c15122d28642034bd) Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copiesEtienne Cordonnier2023-10-182-9/+9
| | | | | | | | | | | | | | | | | | | | | declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 9993a89e5b97dda5f3657e5a7cc3a4fa94ff7111) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-getvar: Add a quiet command line argumentPaulo Neves2023-10-181-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | bitbake-getvar does not have a way to silence bitbake server's logger and that makes the tool hard to use for text processing. This is especially true when one wants to get a bitbake value to be piped to some other utility and instead we get uncontrolled logging messages or warnings together with bitbake's variable value. Example without quiet: bitbake-getvar --value MACHINE NOTE: Starting bitbake server... qemux86-64 With quiet: bitbake-getvar --value MACHINE --quiet qemux86-64 (Bitbake rev: d677daa0770c66010fc4ed0af5bbbb94752ba590) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit af354e975d0b4c26d0e91e3c82946b093bc11b45) Signed-off-by: Markus Niebel <Markus.Niebel@ew.tq-group.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-layers: initialize tinfoil before registering command line ↵Alexander Kanavin2023-08-041-2/+2
| | | | | | | | | | | | | | | | | arguments Plugins may want to use it (e.g. the layers-setup plugin that would want to discover writer sub-plugins with it), and so it makes sense to make tinfoil available a bit eariler. (Bitbake rev: 41b6684489d0261753344956042be2cc4adb0159) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2f6c7523a622f59ddf84a1a196927492bc5fa7a2) Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-07-121-4/+12
| | | | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 907416ee1062f87f5844ab0638b54616abfc1a22) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2023-04-264-6/+21
| | | | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: 0c6f86b60cfba67c20733516957c0a654eb2b44c) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-04-111-2/+6
| | | | | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-03-141-5/+0
| | | | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: a29ffcc12df24737e702e198aeb8f6892884932b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-06 23:00:02 +0000