summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: siggen: Add parallel unihash exist APIJoshua Watt2024-02-191-0/+32
| | | | | | | | | Adds API to query if unihashes are known to the server in parallel (Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add parallel query APIJoshua Watt2024-02-191-34/+87
| | | | | | | | | | | | | | | | Implements a new API called get_unihashes() that allows for querying multiple unihashes in parallel. The API is also reworked to make it easier for derived classes to interface with the new API in a consistent manner. Instead of overriding get_unihash() to add custom handling for local hash calculating (e.g. caches) derived classes should now override get_cached_unihash(), and return the local unihash or None if there isn't one. (Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add Client Pool objectJoshua Watt2024-02-192-1/+78
| | | | | | | | | | | | Adds an abstract base class that can be used to implement a pool of client connections. The class implements a thread that runs an async event loop, and allows derived classes to schedule work on the loop and wait for the work to be finished. (Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: taskexp_ncurses: ncurses version of taskexp.pyDavid Reyna2024-02-181-0/+1511
| | | | | | | | | | | | | | | | * Create an ncurses version of the GTK app "taskexp.py". * Add these additional features: - Sort tasks in recipes by their dependency order - Print individual and/or recipe-wide dependencies to a file - Add a wild card filter - Show the target recipes on BOLD * Provide a GUI self test * Provide a non-ncurses self test for ptest (Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve setcene performance when encoutering many 'hard' ↵Richard Purdie2024-02-131-1/+11
| | | | | | | | | | | | | | | | | | dependencies "bitbake world -n --setscene-only" shows poor performance as the numbers of tasks increases. Analysys shows this is due to the "deferred" hard dependencies being repeatedly processed which doesn't allow much forward progress in overall task execution. To avoid this, mark when it has been done and don't reprocess until dependencies are updated. We have to be careful as we've seen bugs where these dependencies aren't processed at the right time. They also have to be reproceseed during task migration/rehashing so the code has to "self heal" the data structures. (Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise taskname lookups in next_buildable_taskRichard Purdie2024-02-131-3/+3
| | | | | | | | | | A quick profile of bitbake world showed 147 million calls to taskname_from_tid(). The next_buildable_task function is performance senstive so move the call inside the if block to reduce the number of calls and speed the code up. (Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve performance for executing tasksRichard Purdie2024-02-131-10/+21
| | | | | | | | | | | | | Now that runqueue performance profiling works again we can see a lot of time is lost in build_taskdepdata. Whilst we can't compute that in advance, we can compute the individual entries. Therefore put a cache in place to compute those and save overhead in starting up tasks. (Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process/server: Fix typoRichard Purdie2024-02-101-1/+1
| | | | | | | | Ensure the message matches the filenames the code actually uses. (Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: always use the custom user agentThomas Perrot2024-02-101-3/+4
| | | | | | | | | | | | | | | | | | | Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Add profile logging for main loopRichard Purdie2024-02-101-0/+16
| | | | | | | | | When the idle/main loop was added, we didn't include profiling information for it. There is a performance issue in there, add logging for it. (Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: support temporary AWS credentialsToni Lammi2024-02-091-0/+1
| | | | | | | | | | | | | Support AWS_SESSION_TOKEN which is used in temporary AWS credentials. Fixes [YOCTO #15384]. (Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb) Signed-off-by: Toni Lammi <toni.lammi@kone.com> Reported-by: Toni Lammi <toni.lammi@tl-software.fi> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch.py: use real subversion repositoryMartin Jansa2024-02-081-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * github no longer provides support for subversion clients: https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients it was shut down on 2024-01-08: https://github.blog/2023-01-20-sunsetting-subversion-support/ and this test was now failing with: ====================================================================== ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn) ---------------------------------------------------------------------- Traceback (most recent call last): File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn fetcher.download() File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download raise FetchError("Unable to fetch URL from any source.", u) bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source. Stdout: Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2 Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output: A trunk/README.md U trunk Checked out revision 2. svn: warning: W205011: Error handling externals definition for 'trunk/bitbake': svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git' svn: E205011: Failure occurred processing one or more externals definitions in the rare cases where subversion was still installed on the host running bitbake-selftest :). to avoid this use still alive repository from https://svn.apache.org/ and pick something rather small and only the trunk subdirectory which is fast to fetch: svn co https://svn.apache.org/repos/asf/serf/trunk takes just 2 sec here adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of "trunk/README" (Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: fix walking the task dependencies and show better ↵Martin Jansa2024-02-081-6/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | error * when comparing 2 tmp/stamps/*do_configure.sigdata* I got TypeError("filename must be a str or bytes object, or a file") but both files I was comparing were Zstandard compressed data (v0.8+), Dictionary ID: None according to "file" with TypeError catched to show which file it failed to open I got better error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata" file now and after a while you might notice that it's not just the expected file, but a dict with 'path', 'sstate', 'time'. Fix that in bitbake-diffsigs but keep the TypeError and add OSError in case it will eventually walk on file which isn't zstd compressed pipecompress throws CompressionError. ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file Traceback (most recent call last): File "bitbake/bin/bitbake-diffsigs", line 192, in <module> output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles recout = recursecb(dep, a[dep], b[dep]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles raise err File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/compress/zstd.py", line 12, in open return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap raise TypeError("filename must be a str or bytes object, or a file") TypeError: filename must be a str or bytes object, or a file * if I replace zstd file with just plaintext it fails with: $ echo foo > foo $ echo foo > bar $ bitbake-diffsigs foo bar zstd: /*stdin*\: unsupported format ERROR: Process died with 1 sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'> with this change it shows the name of the file which it failed to uncompress: $ bitbake-diffsigs foo bar zstd: /*stdin*\: unsupported format ERROR: Failed to open sigdata file 'foo': Process died with 1 ERROR: Process died with 1 sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'> (Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Version bump for inherit_defer additionRoss Burton2024-01-261-1/+1
| | | | | | | | | | We've added a new statement, inherit_defer, so bump the version so this can be checked. (Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/BBHandler: Add inherit_defer supportRichard Purdie2024-01-182-1/+31
| | | | | | | | | | | | | | | | | | | | | | Add support for an inherit_defer statement which works as inherit does but is only evaulated at the end of parsing. This allows conditional expressions to be evaulated 'late' meaning changes to PACKAGECONFIG from bbappends can be applied for example. This addresses a usability/confusion issue users have with the current conditional inherit mechanism since they don't realise the condition has to be expanded immediately with the current model. There is a commented out warning we could enable in future which warns about the use of a conditional inherit that isn't deferred. There is a behaviour difference in the placement of the inherit, particularly around variables set using ?= which means wen swapping from one to the other, caution must be used as values can change. (Bitbake rev: 5c2e840eafeba1f0f754c226b87bfb674f7bea29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall all bugRichard Purdie2024-01-121-18/+24
| | | | | | | | | | | | | Where chains of RDEPENDS are multiple levels deep, the runall code was not accounting for this and recursing deeply enough to gather all dependencies. Fix this by iterating over the result until no more dependencies are found. Tested-by: Jonas Gorski <jonas.gorski@bisdn.de> Reported-by: Jonas Gorski <jonas.gorski@bisdn.de> (Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: rework 'bitbake -S printdiff' logicAlexander Kanavin2024-01-101-15/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously printdiff code would iterate over tasks that were reported as invalid or absent, trying to follow dependency chains that would reach the most basic invalid items in the tree. While this works in tightly controlled local builds, it can lead to bizarre reports against industrial-sized sstate caches, as the code would not consider whether the overall target can be fulfilled from valid sstate objects, and instead report missing sstate signature files that perhaps were never even created due to hash equivalency providing shortcuts in builds. This commit reworks the logic in two ways: - start the iteration over final targets rather than missing objects and try to recursively arrive at the root of the invalid object dependency. A previous version of this patch relied relies on finding the most 'recent' signature in stamps or sstate in a different function later, and recursively comparing that to the current signature, which is unreliable on real world caches. - if a given object can be fulfilled from sstate, recurse only into its setscene dependencies; bitbake wouldn't care if dependencies for the actual task are absent, and neither should printdiff I wrote a recursive function for following dependencies, as doing recursive algorithms non-recursively can result in write-only code, as was the case here. [YOCTO #15289] (Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/knotty: properly handle exceptions when calling runCommand()Mark Asselstine2024-01-101-10/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | In runCommand() the send() and recv() can fail and raise BrokenPipeError and EOFError exceptions when the bitbake-server is unexpectedly terminated. In these cases a python traceback is currently dumped. Similarly updateFromServer() which calls runCommand() can also raise these and other exceptions, and currently lacks proper exception handling resulting in python traceback. We wrap calls to runCommand() and updateFromServer() in a try/except block as well as improve the exception handling for updateToServer(). This along with the earlier commit which added text to the BrokenPipeError and EOFError exceptions in runCommand() to indicate a bitbake-server termination may have occurred, should improve the user's ability to understand and handle these errors. An easy way to trigger each of the runCommand() exceptions is to 'kill -9' bitbake-server before (causes EOFError) or after (causes BrokenPipeError) the "Loading Cache" stage. (Bitbake rev: 804d366ee3ddc0f37f0a6c712c8d42db45b119bc) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: catch and expand multiprocessing connection exceptionsMark Asselstine2024-01-101-2/+8
| | | | | | | | | | | | | | | | | | | | | | | Doing builds on systems with limited resources, or with high demand package builds such as chromium it isn't uncommon for the OOM Killer to be triggered and for bitbake-server to be selected as the process to be killed. When the bitbake-server does terminate unexpectedly due to the OOM Killer or otherwise, this currently results in a generic python traceback with little indication as to what has failed. Here we trap and raise the exceptions while extending the exception text in runCommand() to make it clear that this is most likely caused by the bitbake-server unexpectedly terminating. Callers of runCommand() should be updated to properly handle the BrokenPipeError and EOFError exceptions to avoid printing a python traceback, but even if they don't, the added text in the exceptions should provide some hints as to what might have caused the failure. (Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast: Fix EXPORT_FUNCTIONS bugRichard Purdie2024-01-102-4/+103
| | | | | | | | | | | | | | | | If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton and a standard funciton definition for the function that is exported, the export function can sometimes overwrite the standard one. The issue is that the internal flag the code uses isn't ovweritten if the variable is giving a new value. Fix the issue by using a comment in the code that is injected so that we know if it is ours or not. Also add some testing for EXPORT_FUNCTIONS, not perfect but a start. (Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/event: Add test_lineno_in_eventhandlerRobert Yang2024-01-101-0/+24
| | | | | | | | | Add test_lineno_in_eventhandler to test lineno in eventhandler. (Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: event: Inject empty lines to make code match lineno in ↵Robert Yang2024-01-101-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | filename So that we can get the correct error messages. * In python 3.10.9, the error message was: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 4 # SPDX-License-Identifier: MIT ^^^^^ SyntaxError: invalid syntax This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4. * Now the error message and lineno is correct: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 256 an error line ^^^^^ SyntaxError: invalid syntax And no impact on parsing time: * Before: $ rm -fr cache tmp; time bitbake -p real 0m27.254s * Now: $ rm -fr cache tmp; time bitbake -p real 0m27.200s (Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Version bump for find_siginfo chanagesRichard Purdie2024-01-051-1/+1
| | | | | | | | Bump the version to 2.7.1 for the find_siginfo changes. (Bitbake rev: 03995e16bf7186f5368f772f617d563f4d280641) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: prioritize local stamps over sstate signatures in ↵Alexander Kanavin2024-01-051-0/+3
| | | | | | | | | | | | | | | | | | | printdiff Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. (Bitbake rev: c8574b796dabb69699c70540dd95a44d8f7388ab) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure version of siggen is verifiedRichard Purdie2024-01-053-0/+12
| | | | | | | | | | Since we need to change the form of the siggen function, we need to add versioning and some verison checks. This means if a newer bitbake is used with older metadata we can detect it. (Bitbake rev: 721556568413508213d22c29985e305a45a8d68a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs/runqueue: adapt to reworked find_siginfo()Alexander Kanavin2024-01-051-5/+5
| | | | | | | | | | In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. (Bitbake rev: 5439aca056c84ab4410aaf24bdb68e896191d8e1) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Post release version bump to 2.7.0Richard Purdie2024-01-051-1/+1
| | | | | | | | Bump to a development version post release. (Bitbake rev: 28364c08f36c778a5cb2e3f20ceb052370ef153c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: add debugging for find_siginfo() callsAlexander Kanavin2024-01-051-0/+4
| | | | | | | (Bitbake rev: 52f5503e8cf048331134233fb681db6dc736ae38) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/codeparser.py: address ast module deprecations in py 3.12Alexander Kanavin2024-01-021-10/+10
| | | | | | | | | | | | | Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s (Bitbake rev: de8ba2770d9a1a94af3d084f9540da7e2fae6022) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add workaround for libgcc issues with python 3.8 and 3.9Richard Purdie2023-12-301-0/+7
| | | | | | | | | | | | | | | | | | | | With python 3.8 and 3.9, we see intermittent errors of: libgcc_s.so.1 must be installed for pthread_cancel to work Aborted (core dumped) which seem related to: https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work https://bugs.ams1.psf.io/issue42888 These tend to occur on debian 11 and ubuntu 20.04. Workaround this by ensuring libgcc is preloaded in all cases. (Bitbake rev: c6666f6cfafd55e1c980239a7c5ff908f1a69196) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Remove tie between rqexe and starts_workerRichard Purdie2023-12-171-15/+8
| | | | | | | | | | | We've been moving to try and separate several pieces of runqueue. Allow start_worker to operate separately to rqexe since they don't need to be tied. This allows rqexe to be available to print_diff for future improvements. (Bitbake rev: 834e452243ff2eea6e8e2e7f4935b5233ffb4b00) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix mkdir with PosixPathRichard Purdie2023-12-171-1/+1
| | | | | | | | | Avoid: Exception: AttributeError: 'PosixPath' object has no attribute 'find' (Bitbake rev: 0b37fe89ba12549109905b6d0e6d07d342162436) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Added new feature to import eventlogs from command line ↵Marlon Rodriguez Garcia2023-12-122-1/+87
| | | | | | | | | | | | | | | | | | | | | | | | into toaster using replay functionality Added a new button on the base template to access a new template. Added a model register the information on the builds and generate access links Added a form to include the option to load specific files Added jquery and ajax functions to block screen and redirect to build page when import eventlogs is trigger Added a new button on landing page linked to import build page, and set min-height of buttons in landing page for uniformity Removed test assertion to check command line build in content, because new button contains text Updated toaster_eventreplay to use library Fix test in test_layerdetails_page Rebased from master This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read the files created by bitbake Exclude listing of files that don't contain the allvariables definitions used to replay builds This part of the feature should be revisited. Over a long period of time, the BB_DEFAULT_EVENTLOG will exponentially increase the size of the log file and cause bottlenecks when importing. (Bitbake rev: ab96cafe03d8bab33c1de09602cc62bd6974f157) Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Do not create directories with ${ in the namePavel Zhukov2023-12-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In some cases ${ may not be expanded in the WORKDIR (one of the cases is undefined variable) and causes cryptic failures [1]. Guard this by erroring out if directory name contains ${ Fixes: [Yocto #15255] [1] ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_hardcode_path(d) 0003: File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path 0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)) 0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir) 0652: 0653: # If the fixmefn is empty, remove it.. *** 0654: if os.stat(fixmefn).st_size == 0: 0655: os.remove(fixmefn) 0656: else: 0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd)) 0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath' (Bitbake rev: e91c256ec076e70cf8a18e369fe7862e50618c48) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Move to version 2.6.1 to mark runqueue changesRichard Purdie2023-12-061-1/+1
| | | | | | (Bitbake rev: 651a6dcf6f8ff33a4e9290a37c23e4f243974ac3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/toaster: Fix assertEquals deprecation warningsRichard Purdie2023-12-061-1/+1
| | | | | | | | | Fix: DeprecationWarning: Please use assertEqual instead (Bitbake rev: dd990ea6843685927954101feb729f3faa3a16d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve inter setscene task dependency handlingRichard Purdie2023-12-061-13/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The way the code currently handles dependencies between setscene tasks is fairly poor, basically by deleting chunks of dependencies and adding reversed dependency relationships. This was once the best way to handle things but now a lot of the surrounding code has changed and this approach is suboptimal and can be improved. This change firstly adds debug logging for "hard" setscene task dependencies since previously the codepaths were missing from logs making them very hard to read. The changes to the setscene dependency graph are removed entirely this these altered graphs were a significant source of problems. Instead, if a hard dependency is run into, we mark the hard dependency as buildable and defer the task until the hard dependencies are met. The code now also skips the check_dependencies() code for hard dependencies since previously that code was having to list all possible hard dependencies. We don't need to do that as we can safely assume hard dependencies are required. With these changes to runqueue's behaviour, we stand some chance of being able to fix other bugs in OE-Core related to useradd for example. (Bitbake rev: 367789b53c1c22ec26e0f4836cdf2bdd9c7d84fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.Charlie Johnston2023-12-021-2/+5
| | | | | | | | | | | | | | | | | The GCP fetcher was calling bb.fetch2.check_network_access with "gsutil stat" as the command, but then never actually ran that command to check if the file exists. In cases where the file did not exist in a gs:// premirror, this would lead to an unhandled exception from do_fetch when the GCP python API tried to perform the download. This change resolves that issue by adding a runfetchcmd to call gsutil. (Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229) Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests: Fix duplicate test_underscore_override testRichard Haar2023-12-021-10/+10
| | | | | | | | | Found a duplicate test, added _2 suffix to one, 74 tests now pass up from 73. (Bitbake rev: ae2a19dadb4f3065b8731a61f45f29e6a70af402) Signed-off-by: Richard Haar <rh@richhaar.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid eventlog variable listing lockupsRichard Purdie2023-11-271-25/+11
| | | | | | | | | | | | | | | If the event log is enabled and parsing the metadata triggers log messages, the event code and deadlock. Iterating the variables inside the event handling code causes this. SOURCE_DATE_EPOCH triggers a python function which calls bb.debug() and can trigger a lockup as one example. Move the code around and add it to the BuildStarted events explictly. This does mean runs without builds no longer get variables added to the eventlog however we can look into a more targetted version of data if/as/where neded. (Bitbake rev: 4135a617ae16d509362b5bf56378139cdc0876d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/ncurses: Add missing function call to avoid tracebackRichard Purdie2023-11-271-0/+3
| | | | | | | | | The ncurses UI wasn't working due to a missing function call. Add it to avoid a traceback when starting builds. (Bitbake rev: db8f36b69a68de2179e5685cf24a42ec10d68257) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/aws: forward env-vars used in gitlab-ci K8sFelix Moessbauer2023-11-231-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds the following variables to the allow-list, which are used in the "IAM roles for AWS when using the GitLab chart": - AWS_ROLE_ARN - AWS_WEB_IDENTITY_TOKEN_FILE These variables are set in the CI job environment and are needed to access the sstate cache artifacts in a connected S3 bucket. [1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com> (Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: add missing destsuffix and subpath parameters in ↵Julien Stephan2023-11-231-0/+8
| | | | | | | | | | | | docstrings Docstring for git fetcher is missing destsuffix and subpath parameters, so add them (Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: git: Update Git-LFS download and testsDesone Burns2023-11-232-28/+47
| | | | | | | | | | | | | | | | | | | | When downloading a Git repository containing an LFS, the Git hooks are not always initialized correctly to perform the download. This change updates the Git downloader to run the "git lfs install" command in order to "smudge" the LFS files. The tests have also been updated to do checks for situations in which git lfs is not installed, as the application was required to be installed for any of the tests to run previously. The Git LFS functionality was working to some degree previously, but this change also updates the fetcher to allow LFS downloads for nobranch SRC_URIs. (Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217) Signed-off-by: Desone Burns II <dburns@seegrid.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Simplify virtualfn2realfn()Peter Kjellerstedt2023-11-231-4/+2
| | | | | | | | | Also correct the description of variant2virtual(). (Bitbake rev: d766e9bd22ec6392fbf1694eea5032b9d09f1949) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Be consistent with what type bb_data representsPeter Kjellerstedt2023-11-231-6/+5
| | | | | | | | | | | | | | | | In _parse_recipe(), bb_data was initially a datastore object, but then after the call to bb.parse.handle() it was all of a sudden turned into a dict of variant:datastore pairs. At the same time, parseRecipeVariants() used bb_data for datastore objects, while parseRecipe() used it for the dicts of variant:datastore pairs. Change these functions to consistently use "bb_data" for datastore objects and use "datastores" for the dicts of variant:datastore pairs. (Bitbake rev: ed0f94c037476efd465db87726b2480406076beb) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Make parseRecipeFile() handle virtual recipes correctlyPeter Kjellerstedt2023-11-231-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running `devtool search gcc` would result in errors like this for virtual recipes: ERROR: When reparsing .../meta/recipes-devtools/gcc/libgcc-initial_13.2.bb:do_populate_sysroot, the basehash value changed from b1cd809ed98cef9db0fb1b17d34c4083e739c336f9d5619b89715b0294d81af5 to 44c2f92781dc4a20e98b7bb4724e204e64b101905fa75e71241a574b725997dc. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Snone ERROR: Then: ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Sprintdiff The reason was the newly introduced :layer-<layername> override, which is used, e.g., in meta/classes-global/insane.bbclass to add the patch-status QA test only for the meta layer: ERROR_QA:append:layer-core = " patch-status" When tinfoil parsed the recipes using the parseRecipeFile() function, it failed to properly identify the layername for virtual recipes, which resulted in the error above. The correct thing to do is to make parseRecipeFile() call bb.cache.virtualfn2realfn() to convert the virtual filename into a real filename and virtual class. (Bitbake rev: da2aed134412f5de04d7b540f92d735983ad0108) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid sideeffects for autorev from getAllKeysWithFlagsRichard Purdie2023-11-231-0/+7
| | | | | | | | | | | | | If we expand the variable AUTOREV in OE-Core, it triggers side effects in the fetcher. The situation isn't ideal and needs improvement but this breaks and is blocking enabling BB_DEFAULT_EVENTLOG. Hack around the issue for now so we unblock things until we can work out a better plan for how to improve AUTOREV support. (Bitbake rev: cb9b6530f3d12c56a8b48847af2e7461924205d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-09 19:11:42 +0000