summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: hashserv: tests: Omit client in slow server start testJoshua Watt2024-10-101-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | On Fedora 39 and Fedora 40 hosts, this version of the hash server exhibits different behavior on exit when using Unix Domain sockets. Instead of closing the client connections and exiting immediately, the server will wait until all clients have disconnected before exiting. It is unknown why this changed, or why it only affects Unix Domain sockets and not TCP sockets. Because of this behavior change, the Slow Server Start test is failing on these hosts. This test is primarily concerned with ensuring that the server will actually exit, even if it gets a termination signal before it enters its main loop, and doesn't really care about clients. As such, modify the test so that a client is not pre-connected to the server. This allows the server to actually exit so that the test can verify the signal behavior. The latest version of the hash equivalence server (on master) does not exhibit this behavior. Speculation is that the more standardized usage of asyncio allows the server to properly exit, even if clients are still connected. Regardless, this patch is only intended for the older versions, and should not be applied to master (Bitbake rev: eb5c1ce6b1b8f33535ff7b9263ec7648044163ea) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-10-101-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 07081a94997142746f7d345c27bc6805231d025d) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman <steve@sakoman.com> (cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Move files into place atomicallyRichard Purdie2024-10-101-4/+7
| | | | | | | | (Bitbake rev: aaf9c1b8673f5b181255a08eee4e9a342863e896) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak to work on Fedora40Richard Purdie2024-10-101-3/+3
| | | | | | | | | | | | | | On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. (Bitbake rev: 3147ebb0edcca8f7f850aabd86ef010ab593496f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-10-101-1/+4
| | | | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: 477f90f9515cb7cd093920cee5518f662cef2e05) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart: Improve performance for VariableHistoryRobert Yang2024-08-231-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: ec2a99a077da9aa0e99e8b05e0c65dcbd45864b1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak test to match upstream repo url change Upstream ↵Steve Sakoman2024-06-241-1/+1
| | | | | | | | | changed their urls, update our test to match. (Bitbake rev: 734b0ea3dfe45eb16ee60f0c2c388e22af4040e0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: parse: Improve/fix cache invalidation via mtimeRichard Purdie2024-05-291-3/+6
| | | | | | | | | | | | | | | | | | | | We have been seeing obscure failures in devtool, particularly on newer autobuilder workers where it appears the cache is assumed to be valid when it shouldn't be. We're using the 'seconds' granulation mtime field which is not really a good way of telling if a file has changed. We can switch to the "ns" version which is better however also add in inode number and size as precautions. We already have all this data and tuples are fast so there isn't really any cost to do so. This hopefully fixes [YOCTO #15318]. (Bitbake rev: 5a90927f31c4f9fccbe5d9d07d08e6e69485baa8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-03-012-6/+90
| | | | | | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae) (Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2024-03-011-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (cherry picked from commit d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) (Bitbake rev: 2a6fd774405a58f3ef2953a0dc3eca063e0cf890) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2024-03-011-16/+24
| | | | | | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (cherry picked from commit 70f848631450bd723c223227c21c60e815ee033d) (Bitbake rev: 1d5d4a46c772f44ab652f95b37f508b890828e67) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake/lib/bs4/tests/test_tree.py: python 3.12 regexAdrian Freihofer2024-03-011-1/+1
| | | | | | | | | | | | | Python 3 interprets string literals as Unicode strings, and therefore \s is treated as an escaped Unicode character which is not correct. Declaring the RegEx pattern as a raw string instead of unicode is required for Python 3. (Bitbake rev: 7efed7bc09c56e41d3074a26388581a62f145625) Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake/codeparser.py: address ast module deprecations in py 3.12Alexander Kanavin2024-03-011-10/+10
| | | | | | | | | | | | | | Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s (Bitbake rev: 6c19b6cf105ac321ec89da1a876a317020c45ab7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser: replace deprecated ast.Str and 's'Chris Laplante2024-03-011-2/+2
| | | | | | | | | | These have been deprecated since 3.8 (Bitbake rev: 4591011449212c8e494ea42348acb2d27a82a51b) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toaster/toastergui: Bug-fix verify given layer path only if ↵Alassane Yattara2024-02-071-5/+8
| | | | | | | | | | import/add local layer (Bitbake rev: a4c516ef5e72b2d77ac5ff7e86c5ee2190ebc42f) Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: asyncrpc: Add context manager APIJoshua Watt2024-01-103-22/+33
| | | | | | | | | | | | | | | | | | Adds context manager API for the asyncrcp client class which allow writing code that will automatically close the connection like so: with hashserv.create_client(address) as client: ... Rework the bitbake-hashclient tool and PR server to use this new API to fix warnings about unclosed event loops when exiting (Bitbake rev: ee090484cc25d760b8c20f18add17b5eff485b40) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: lib/bb: Add workaround for libgcc issues with python 3.8 and 3.9Richard Purdie2024-01-041-0/+7
| | | | | | | | | | | | | | | | | | | | | With python 3.8 and 3.9, we see intermittent errors of: libgcc_s.so.1 must be installed for pthread_cancel to work Aborted (core dumped) which seem related to: https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work https://bugs.ams1.psf.io/issue42888 These tend to occur on debian 11 and ubuntu 20.04. Workaround this by ensuring libgcc is preloaded in all cases. (Bitbake rev: 2c6183594279e2e9d03f11155ad969448869c863) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data: Add missing dependency handling of remove operatorInsu Park2024-01-042-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | A recipe variable handles its dependencies even on the "contains" variables within the "inline Python expressions" like bb.utils.filter(). And it also handles those in the append operator correctly, but the problem is that it does not so in the remove operator. Fix it by adding the missing dependencies every time the remove operator has been handled. Also add a test case to check if the override operators handle dependencies correctly. (Bitbake rev: 23639edfbbb3fced7606dce211db8a31c5766585) Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Cherry-picked from master: b90520eedb1dbc7f6a3928d089fe74fafb864eb5 - Conflicts in data.py are resolved as the master branch moved handle_contains() and handle_remove() out of the try block. - The test code in codeparser.py are modified as the master branch added three more arguments to the build_dependencies(). Signed-off-by: Insu Park <insu0.park@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toastergui: verify that an existing layer path is givenMarta Rybczynska2023-12-211-6/+7
| | | | | | | | | | | | | | Verify that an existing layer path was given when adding a new layer. Manually using the shell for globbing is unnecessary, use the glob function instead for cleaner code. (Bitbake rev: de1cf0e31f3836a449cbd17490a2f6b6a86b17f9) Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: show number of currently running bitbake threads when ↵Martin Jansa2023-12-111-1/+1
| | | | | | | | | | | | | | | | | | pressure changes * it might be a bit confusing as it shows number of threads before making the decision to start more tasks and also it can show only a few tasks running, but not because of pressure when there just aren't many tasks left or wait for their dependencies to be finished first * example output: NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads (Bitbake rev: 42a1c9fe698a03feb34c5bba223c6e6e0350925b) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: show more pressure dataMartin Jansa2023-12-111-8/+10
| | | | | | | | | | | | | | * with latest bitbake I'm seeing very low number of bitbake tasks executed in parallel, probably due to pressure regulation show the values this is based on in the note * also simplify a bit by counting the pressure and exceeds signs only once (Bitbake rev: 1050145c3004861ebede4777fd4fbd89d0470716) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue.py: fix PSI check logicChen Qi2023-12-111-4/+5
| | | | | | | | | | | | | | | | | | | The current calculation is not correct because if tdiff is less than 1.0, it's not taken into consideration when calculating the current pressure. Also, make it clear that the 1.0s is the psi accumulation cycle, which might be changed in the future. We have this cycle because it could largely avoid the 0 result issue, that is, if the interval between checks are too small, the result might be 0. With this accumulation logic, which has been there but let's make it clear, this 0 result problem could be mitigated. (Bitbake rev: 76889ff0a8938a3d77603d2af176aa9e264df839) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Add pressure change loggingRichard Purdie2023-12-111-0/+4
| | | | | | | | | | | It is currently hard to tell when bitbake is throttling task execution due to system pressure changes. Add notes to the console output to make this clearer, only generating output when the values change. (Bitbake rev: b5f77e8159ad321f31999af8304f082a2c56b537) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: fix PSI check calculationChen Qi2023-12-111-4/+9
| | | | | | | | | | | | | | | | The current PSI check calculation does not take into consideration the possibility of the time interval between last check and current check being much larger than 1s. In fact, the current behavior does not match what the manual says about BB_PRESSURE_MAX_XXX, even if the value is set to upper limit, 1000000, we still get many blocks on new task launch. The difference between 'total' should be divided by the time interval if it's larger than 1s. (Bitbake rev: 2b2997ab6bdda730e4b638f416311a73e0c42156) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: Fix disk space monitoring on cephfsSamantha Jalabert2023-11-091-3/+4
| | | | | | | | | | | | | Error occured while running bitbake on cephfs: WARNING: The free inode of path is running low (-0.001K left) ERROR: Immediately halt since the disk space monitor action is "HALT"! (Bitbake rev: a7f6c3e67bd9170e93b2b94676e84018faf0df91) Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: convert deferral messages from bb.note to bb.debugDenys Dmytriyenko2023-11-091-3/+3
| | | | | | | | | | | | | | | | | Using multiconfig to target baremetal pieces of the system and building corresponding toolchains for them results in hundreds and hundreds of "Deferring %s after %s" and "Deferred task %s now buildable". To clean up the output and to reduce risk of missing important warnings, convert these notice messages to debug messages. (Bitbake rev: 3505d8d8c02b041946670ab6bc5751e54fe292ff) Signed-off-by: Denys Dmytriyenko <denis@denix.org> Signed-off-by: Denys Dmytriyenko <denys@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tinfoil: Do not fail when logging is disabled and full config is usedPeter Kjellerstedt2023-11-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | If Tinfoil is initialized with setup_logging = False and Tinfoil.prepare() is called with config_only = False, then it fails because self.localhandlers is only initialized when setup_logging = True. This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`: Traceback (most recent call last): File ".../bitbake/bin/bitbake-getvar", line 41, in <module> tinfoil.prepare(quiet=2) File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare for handler in self.localhandlers: AttributeError: 'Tinfoil' object has no attribute 'localhandlers'. Did you mean: 'oldhandlers'? (Bitbake rev: e452c6d7ba5bb4f78a1d2bfb742794efdf171dbc) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 616101ddb630e2c9975022068b52a87c4cf647f6) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-getvar: Make --quiet work with --recipePeter Kjellerstedt2023-11-091-2/+3
| | | | | | | | | | | | | Initializing Tinfoil with setup_logging = False only has an effect when recipe parsing is not needed. To make it work regardless of if --recipe is used, manipulate the quiet parameter to Tinfoil.prepare() instead. (Bitbake rev: 161ab0d5bab74732e12d490cee50e14295be0a9f) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 71ee69a20f21f3d37f4f060a7d8e87d9f1dc6aa1) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: SECURITY.md: add fileMarta Rybczynska2023-10-241-0/+24
| | | | | | | | | | | Add a SECURITY.md file with hints for security researchers and other parties who might report potential security vulnerabilities. (Bitbake rev: 6c1ffa9091d0c53a100e8c8c15122d28642034bd) Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-worker/runqueue: Avoid unnecessary bytes object copiesEtienne Cordonnier2023-10-182-9/+9
| | | | | | | | | | | | | | | | | | | | | declaring queue=b"" creates an object of types bytes(). bytes() is an immutable object, and therefore doing "self.queue = self.queue + r" creates a new object containing "self.queue" concatenated with "r". On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker, so those copies significantly slow down the initialization of the bitbake-worker. Rather use bytearray() which a mutable type, and use extend() to avoid copies. In my test setup, byterray.extend() is 10.000 times faster than copying the queue, for a queue size of 180MB. (Bitbake rev: 9993a89e5b97dda5f3657e5a7cc3a4fa94ff7111) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-getvar: Add a quiet command line argumentPaulo Neves2023-10-181-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | bitbake-getvar does not have a way to silence bitbake server's logger and that makes the tool hard to use for text processing. This is especially true when one wants to get a bitbake value to be piped to some other utility and instead we get uncontrolled logging messages or warnings together with bitbake's variable value. Example without quiet: bitbake-getvar --value MACHINE NOTE: Starting bitbake server... qemux86-64 With quiet: bitbake-getvar --value MACHINE --quiet qemux86-64 (Bitbake rev: d677daa0770c66010fc4ed0af5bbbb94752ba590) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit af354e975d0b4c26d0e91e3c82946b093bc11b45) Signed-off-by: Markus Niebel <Markus.Niebel@ew.tq-group.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-layers: initialize tinfoil before registering command line ↵Alexander Kanavin2023-08-041-2/+2
| | | | | | | | | | | | | | | | | arguments Plugins may want to use it (e.g. the layers-setup plugin that would want to discover writer sub-plugins with it), and so it makes sense to make tinfoil available a bit eariler. (Bitbake rev: 41b6684489d0261753344956042be2cc4adb0159) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2f6c7523a622f59ddf84a1a196927492bc5fa7a2) Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix deferred task/multiconfig race issueRichard Purdie2023-07-121-4/+12
| | | | | | | | | | | | | | | | | If there are several multiconfigs in play for example a non-multiconfig with a task with one hash and then three multiconfigs for the same task, different architectures but the same hash (different to the non-mc), the three mcs will be deferred until after the non-mc task but then will all run together and race against each other. Change the code to re-enable deferred tasks one at a time. This way, if they do race, they won't run in parallel against each other. (Bitbake rev: 907416ee1062f87f5844ab0638b54616abfc1a22) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2023-04-264-6/+21
| | | | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: 0c6f86b60cfba67c20733516957c0a654eb2b44c) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/git: Fix local clone url to make it work with repoRobert Yang2023-04-111-2/+6
| | | | | | | | | | | | | | | | | | | | | The "git clone /path/to/git/objects_symlink" couldn't work after the following change: https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56 But repo command manages the git repo as symlinks, so check whether the objects is an symlink to fix the problem: * Nothing is changed if git/objects is not a symlink * Use "git clone file://" when git/objects is a symlink (Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Drop dubious exception handling codeRichard Purdie2023-03-141-5/+0
| | | | | | | | | | | | | This code appears to be dangerous, it swallows exceptions, turning them into "handled" versions which then show no errors to the user. This is a pretty poor user experience and I can't see why this code should be swallowing such things. Drop the worst bits of code. (Bitbake rev: a29ffcc12df24737e702e198aeb8f6892884932b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve early exception handlingRichard Purdie2023-03-141-9/+1
| | | | | | | | | | | | | | | | | | Martin Jansa reported that if you put a syntax error into an imported module such as qa.py in OE, no error is shown. Part of the issue appears to be that the catch_parse_error() decorator only catches certain exceptions and SyntaxError isn't one of them. As far as I can tell we should remove all the special cases and use the more advanced code in all cases, not just expansion errors. I confirmed this now prints a proper error message for a qa.py syntax error. (Bitbake rev: e66012bfda60ffe1658473e25879aa67909ae65f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Remove incorrect SystemExit usageRichard Purdie2023-03-141-1/+1
| | | | | | | | | | | | | Calling SystemExit doesn't work well with server/client usage since the string isn't printed to the right place. Use bb.fatal() instead which prints the right log output and raises and handled exception which then shows correctly on the UI. (Bitbake rev: b9ae7164d9e744e8eb9aaff79218f57233a449b7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Allow to_boolean to support int valuesRichard Purdie2023-03-141-0/+3
| | | | | | | | | | | | | | | | | | Some variables may be set as: X = 1 as well the more usual X = "1" so add support to to_boolean to handle this case. (Bitbake rev: e7df13a61911b7431802af2b4d7472b2aaf346fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: show SRCREV and git repo in error message about fixed ↵Martin Jansa2023-02-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SRCREV * in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to be one character shorter and because fetcher uses: if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]): to decide which SRCREV values are fixed SRCREVs this one was considered as tag or branch name, because it was only 39 chars long The original error message wasn't very helpful as it doesn't show which repo or which SRCREV was considered missing: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) with SRCPV included in PV as error recomments it's a bit better: bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16 The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T with this change the first error will read: do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None) (Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: break on first dependent task differenceSchmidt, Adriaan2023-02-171-0/+1
| | | | | | | | | | | | | | | | | compare_sigfiles() recursively calculates differences on all dependent tasks with changed hashes. This is done in arbitrary/alphabetical order, and only the last of those results is returned, while everything else is discarded. This changes the behavior to instead return the first difference and not calculate any more, which significantly speeds up diffs of tasks with many dependencies. (Bitbake rev: 89f13cd4a927a73de98998c27082c63b07671525) Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: Make PEP8 compliantMarius Kriegerowski2023-02-171-22/+27
| | | | | | | | | | | | | | | This ignores flake8 rules: * E402 module level import not at top of file * E501 line too long (Bitbake rev: 60e05043f83c73a34cd154193e5c40d18a3ed3db) Signed-off-by: Marius Kriegerowski <marius.kriegerowski@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e8b176de448dc387c7a578c92b52aef28591038f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Update md5 to better report errors with hashlibMark Hatle2023-02-172-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. (Bitbake rev: d26ed38c233583b35ad1451e521f07d9c2329f4e) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> (cherry picked from commit 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix inefficient string concatenationEtienne Cordonnier2023-02-171-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython has an optimization for statements in the form "a = a + b" or "a += b". It seems that this line does not get optimized, because it has a form a = a + b + c: data = data + "./" + f.split("/./")[1] For that reason, it does a copy of data for each iteration, potentially copying megabytes of data for each iteration. Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds instead of 45 seconds on my test setup where SRC_URI points to a big directory. Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython: "do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b" However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup: data = ''.join((data, "./", f.split("/./")[1])) I have changed the other lines to also use += for consistency only, however those were in the form a = a + b and were optimized already. Co-authored-by: JJ Robertson <jrobertson@snap.com> (Bitbake rev: 590ae6fde9da75db3a368e5c0d47920696c33ebf) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2023-02-042-3/+4
| | | | | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2023-02-042-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> (cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: include SSL certificate paths in export_proxiesRoss Burton2023-01-301-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. Also remove the return value as nothing ever checked it. [ YOCTO #15000 ] (Bitbake rev: ed0dcc40f80c48839bac20298013d70043858a4e) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix regression in gitsm submodule path parsingPavel Zhukov2022-12-091-1/+1
| | | | | | | | | | | | | | | | | | | Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression in submodules path parsing. As the result gitsm fetcher fails on each submodule which name begins from the name of the parent repo which is totally valid usecase [Yocto #14045] [1] Fix the code to error out only if submodule's name is equal to parent name but not if it's part of it. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4 (Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix race issues around hash equivalence and sstate reuseRichard Purdie2022-11-241-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We identified a use case where a native recipe (autoconf-native) was rebuilt with no change in output yet the sstate for do_package tasks wasn't being used. The issue is that do_package tasks have a hard dependency on pseudo-native:do_populate_sysroot. That task was one of the many tasks being rehashed when autoconf-native's hash was changed. If update_tasks processed a recipe before it had processed pseudo-native, that recipe would be marked as not possible from sstate and would run the full tasks. The fix is to split the processing into two passes, first to handle the existing covered/notcovered updates, then in the second pass, check whether there are "harddep" issues. This defers the do_package tasks until after pseudo-native is installed from sstate as expected and everything works well again. (Bitbake rev: 72a3afd99e8b785cb2a2f687e71a58e08cdd9c74) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-271-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: c90d57497b9bcd237c3ae810ee8edb5b0d2d575a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-07 05:52:05 +0000