summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: test/fetch: Switch u-boot based test to use our own mirrorRichard Purdie6 days1-1/+1
| | | | | | | | | | The upstream servers are having issues so switch to our own shadow copy of the repo. (Bitbake rev: 86e7a58c0e9904c33b1b94dddc3d683c2da22d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: ast: Change deferred inherits to happen per recipeRichard Purdie2025-06-111-9/+11
| | | | | | | | | | | | | | | | | | | | | | | | | Currently deferred inherits get processed once for all class extensions as a minor speed optimisation. Unfortunately this limits our options for being able to report deferred classes to our code. There are two challenges with using our deferred classes in OE at present. One is that PACKAGECONFIG values don't work well with class overrides like class-native if there are deferred classes based on PACKAGECONFIG, such as python support. The second is that toolchain selection is proving problematic to implement due to interactions between the toolchain deferred inherit, the class extensions and class overrides being very late. By changing deferred inherits to be recipe extension specific, we open the way to generate events and "peek" at where things will end up, allowing the class overrides to be set earlier. The class extension code is updated to use a deferred inherit for the class extension inheriting so that it is still inherited last. (Bitbake rev: 139f61fe9eec221745184a14b3618d2dfa650b91) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gcp.py: remove slow calls to gsutil statEtienne Cordonnier2025-06-111-7/+7
| | | | | | | | | | | | | | | | The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however: - in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False and does not throw an exception in case the file does not exist. - Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat. - I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable. - add a try/except block in download() instead of the extra call to gsutil [RP: Tweak to avoid import until needed so google module isn't required for everyone] (Bitbake rev: 59df5390381792aba4f3f5185000adf5109267fb) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: Avoid deprecation warningRichard Purdie2025-06-051-1/+1
| | | | | | | | | | | | >From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated. In future Python versions they will be keyword-only parameters." Avoid the warning. (Bitbake rev: 65b744bec9756ee2f43adbfa33c14899638e2b9f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: toaster/tests/buildtest: Switch to new CDNRichard Purdie2025-06-051-1/+1
| | | | | | | | | | | The project is switching the way handle our CDN provision of sstate objects, update the URL accordingly. (Bitbake rev: aef3536b6e4cdbff6d3a171e183811509370b9f9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3372524cb961d95993b27fe4a8d794cdb7255e09) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive eventsRichard Purdie2025-03-152-1/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which were problematic to debug. The failure was inside lock_timeout and once that was identified and the backtrace obtained, the problem becomes clearer: File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal retval = function(self, data, False) File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand self.cooker.updateCache() File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache self.parser = CookerParser(self, mcfilelist, total_masked) File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__ self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array) File "X/bitbake/lib/bb/cache.py", line 772, in __init__ loaded += c.prepare_cache(progress) File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache loaded = self.load_cachefile(progress) File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile progress(cachefile.tell() + previous_progress) File "X/bitbake/lib/bb/cache.py", line 751, in progress bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize), File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers _ui_handlers[h].event.send(event) File "X/bitbake/lib/bb/cooker.py", line 117, in send str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\') File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__ _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg sw(msg.message, msg.category, msg.filename, msg.lineno, File "X/bitbake/lib/bb/main.py", line 113, in _showwarning warnlog.warning(s) File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning self._log(WARNING, msg, args, **kwargs) File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log self.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle self.callHandlers(record) File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers hdlr.handle(record) File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle self.emit(record) File "X/bitbake/lib/bb/event.py", line 778, in emit fire(record, None) File "X/bitbake/lib/bb/event.py", line 234, in fire fire_ui_handlers(event, d) File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers with bb.utils.lock_timeout(_thread_lock): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack()) or put in simpler terms, whilst sending an event(), an unrelated warning message happens to be triggered from asyncio: /usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f0e797d3100> which triggers a second event() which can't be sent as we're already in the critcal section and already hold the lock. That warning is due to the version of asyncio used on Ubuntu 22.04 with python 3.10 and that comined with timing issues explains why we don't see it on other python versions or distros. We can't handle the second event as the lock is there to serialise the events. Instead, we queue the event and then process the queue later. Add a new version of lock_timeout which allows us to handle the situation more gracefully. (Bitbake rev: 696c2c1ef095f8b11c7d2eff36fae50f58c62e5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2c590ff1aff89d23b25ce808650f200013a1e6af) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Add signal blocking for lock_timeoutRichard Purdie2025-03-151-0/+2
| | | | | | | | | | | | We never want to exit whilst holding these locks as it deadlocks all python threads. Add signal blocking around the lock critical part so a signal shouldn't cause such an exit. (Bitbake rev: 5cc1dde1ffd9382b3a9e4cc182be067defba16a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a097755c671e2b530dea6200a94b39fa9dca246c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Tweak lock_timeout logicRichard Purdie2025-03-151-1/+1
| | | | | | | | | | | | We should really try and take the lock in the try/finally block so that in some rare cases such as badly timed interrupt/signal, we always release the lock. (Bitbake rev: 5dbbca616a71047fc3573a2be49846d5c9cdd15a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: utils: Print information about lock issue before exitingRichard Purdie2025-03-151-0/+1
| | | | | | | | (Bitbake rev: d41fad7fadde59dd63647bbbeb074604314bf310) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit cdf6c51a064f8f335c3262b7f102618996f1a229) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart.py: clear expand_cache in _setvar_update_overridevarsChen Qi2025-03-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the end of this function, self.overrides is cleared, which means we'll construct a new self.overrides after this call. And a new self.overrides will give out different expand_cache, so the old expand_cache should also be cleared to avoid any wrong value. Currently, there's a problem revealed by recent recipe specific virtual provider patch. If we enable multilib and set "OVERRIDES:prepend" in local.conf, things don't work. Here's the error message: ERROR: Nothing PROVIDES 'lib32-gcc-cross-x86_64' Below are reproduce steps: 1. Add in local.conf the following lines: MACHINE ?= "qemux86-64" require conf/multilib.conf MULTILIBS ?= "multilib:lib32" DEFAULTTUNE:virtclass-multilib-lib32 ?= "core2-32" OVERRIDES:prepend = "some-override:" (Note that using :append and :remove also reproduces the issue.) 2. bitbake -n lib32-sysstat (bitbake -n core-image-minimal also reproduces the issue) The expandWithRefs calls getVar, which fills expand_cache. So when setting OVERRIDES:prepend, this will fill the expand_cache. When overridevars are updated, if we don't clear expand_cache, we'll retrieve wrong values. Previously, things happened to work because there's a call to expand PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc, which in turn expands TARGET_VENDOR. Now what we expand is PREFERRED_PROVIDER_ virtual/cross-cc, so the problem is revealed. (Bitbake rev: 7375d32e8c1af20c51abec4eb3b072b4ca58b239) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart.py: simple clean upChen Qi2025-03-081-4/+1
| | | | | | | | | | | | | When var matches __setvar_regexp__, which has the value as below, __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$') the keyword will not be __doc__ or __module__ and var will always contain ":". (Bitbake rev: 9cac695951d0fd587063b250bfc8ef0f87acad18) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart.py: remove unnecessary ? from __expand_var_regexp__Chen Qi2025-03-081-1/+1
| | | | | | | | | | | The non-greedy modifier suffix ? is not necessary here because } is not in the character set [a-zA-Z0-9\-_+./~:]. (Bitbake rev: aae570b7f051fc5deee7a1712a02ed92498a4461) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake-diffsigs: fix handling when finding only a single sigfileEnrico Jörns2025-03-081-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the following error when calling 'bitbake-dumpsig' or 'bitbake-diffsigs' when having only a single sigfile available: | Traceback (most recent call last): | File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 171, in <module> | files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1]) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 83, in find_siginfo_task | sig2 = latestsigs[1] | ~~~~~~~~~~^^^ | IndexError: list index out of range Handle this by adding (and returning) the path for the second sigfile only if one is found. This way it will work for both diffsigs and dumpsig use case. The calling argparse code already deals with find_siginfo_task() returning only a single file. For 'bitbake-dumpsig' it will just dump the single sigfile, for 'bitbake-diffsigs' it will emit a proper error message again: | ERROR: Only one matching sigdata file found for the specified task (systemd configure) (cherry picked from commit 25057d33e9131f3214a06bbb316c916c744f8f03) (Bitbake rev: 4e443aeab9096b41c9e5ba41cd21027ecaa20285) Signed-off-by: Enrico Jörns <ejo@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Patrick Vogelaar <patrick.vogelaar@belden.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bblayers/query: Fix using "removeprefix" string methodJoerg Schmidt2025-02-211-1/+2
| | | | | | | | | | | | | | | | | | The minimum Python version required for Yocto 5.0 is 3.8 which causes failure in poky/bitbake/lib/bblayers/query.py when listing layers by using command "bitbake-layers show-recipes -f --bare --mc MC" for the given multiconfig MC. The reason for that failure is the use of "removeprefix" string method which got introduced in Python 3.9. This patch replaces the "removeprefix" method with an equivalent solution supported by Python 3.8. (Bitbake rev: f6635b32ab553e812fe7c05998ef05834fccd0f3) Signed-off-by: Joerg Schmidt <joerg.schmidt@garmin.com> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48Richard Purdie2025-01-251-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | >From git 2.48 release notes: """ When "git fetch $remote" notices that refs/remotes/$remote/HEAD is missing and discovers what branch the other side points with its HEAD, refs/remotes/$remote/HEAD is updated to point to it. """ This means with git 2.48 onwards, there is a mystery "HEAD" revision appearing in some of our shallow clone tests. We can avoid this by using the same canonicalization as used for the reference revisions. This resolves autobuilder failures on the Fedora 40 workers. (Bitbake rev: aa0e540fc31a1c26839efd2c7785a751ce24ebfb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mcChris Laplante2025-01-244-19/+43
| | | | | | | | | | | | | | | | | | | Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. (Bitbake rev: 7dde14582bfd104c6da26e3f5ecf2ef37a1494ce) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: ui/knotty: respect NO_COLOR & check for tty; rename print_hyperlink ↵Chris Laplante2025-01-241-5/+9
| | | | | | | | | => format_hyperlink (Bitbake rev: 72a334d5d5763476d5421a902fabb303f3f84bd8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: ui/knotty: print log paths for failed tasks in summaryChris Laplante2025-01-241-3/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When tasks fail, it's very frustrating to have to scroll up to find the log path(s). Many of us have the muscle memory to navigate to the 'temp' directories under tmp/work/, but new users do not. This change enhances the final summary to include log paths (reported via bb.build.TaskFailed events). Here's an example: NOTE: Tasks Summary: Attempted 856 tasks of which 853 didn't need to be rerun and 3 failed. Summary: 3 tasks failed: virtual:native:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-linux/ncurses-native/6.5/temp/log.do_fetch.1253462 /home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/core2-64-poky-linux/ncurses/6.5/temp/log.do_fetch.1253466 virtual:nativesdk:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch log: /home/chris/repos/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-ncurses/6.5/temp/log.do_fetch.1253467 Summary: There were 3 WARNING messages. Summary: There were 6 ERROR messages, returning a non-zero exit code. Each log is rendered as a clickable hyperlink in the terminal. See https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda (Bitbake rev: 9c020cc314bfd0702bb1d457d94925c6e9613268) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix scenetask processing performance issueRichard Purdie2024-12-091-1/+5
| | | | | | | | | | | | | Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: f40a3a477d5241b697bf2fb030dd804c1ff5839f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Optimise setscene loop processingRichard Purdie2024-12-091-2/+8
| | | | | | | | | | | | Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 28569e9796d4b34d7b77b4f79074ab7854850386) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Fix performance of multiconfigs with large overlapRichard Purdie2024-12-091-2/+4
| | | | | | | | | | | | | | | | | | | | | There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: db083cfe9e33c9fd7ffeead7b8c6023a5d581976) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2: use persist_data context managersChris Laplante2024-12-061-31/+31
| | | | | | | | | | | | | | | | | Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s. See https://docs.python.org/3/whatsnew/3.13.html#sqlite3 The previous commit fixed persist_data's context manager to close the connection, but we were never actually using `with` in the first place. This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: persist_data: close connection in SQLTable __exit__Chris Laplante2024-12-061-0/+1
| | | | | | | | | | | | | | A Connection's __exit__ does not close the connection, but rather deals with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager This change is not necessary on 'master' because persist_data was removed. (Bitbake rev: c9ea354c2df3d94b9ebe7810e83df9efb6007013) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch/wget: Increase timeout to 100s from 30sRichard Purdie2024-12-061-2/+2
| | | | | | | | | | | Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: c7f282cd27edfd78830b61db586ed669808893a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: Remove custom exception backtrace formattingJoshua Watt2024-12-065-121/+25
| | | | | | | | | | | | | | | | | | | | | | | | Removes the code in bitbake to show custom backtrace formatting for exceptions. In particular, the bitbake exception code prints function arguments, which while helpful is a security problem when passwords and other secrets can be passed as function arguments. As it turns out, the handling of the custom serialized exception stack frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass formatted exceptions"), which changed the events to pass a preformatted stacktrack list of strings, but the passing of the serialized data was never removed. Change all the code to use the python traceback API to format exceptions instead of the custom code; conveniently traceback.format_exception() also returns a list of stack trace strings, so it can be used as a drop in replacement for bb.exception.format_exception() (Bitbake rev: c25e7ed128b9fd5b53d28d678238e2f3af52ef8b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-11-281-2/+1
| | | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 70bd343fb273ad174e56d08c8b80c5594501e030) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | decouple from gnome gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 377eba2361850adfb8ce7e761ef9c76be287f88c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome ↵Richard Purdie2024-11-021-1/+1
| | | | | | | | | | | | | | gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 12e2b11e4038138ad1a02b9d4d36d0904fbc820d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-11-021-3/+5
| | | | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 6894f68204130713d3651fceacadc7f8061174a8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-11-021-28/+14
| | | | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: 76de94d46ce270c4e485f8178725973adf0f42eb) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bitbake: doc/user-manual: Update the BB_HASHSERVE_UPSTREAMJose Quaresma2024-11-021-1/+1
| | | | | | | | (Bitbake rev: 220f202471cd03c7c8f08649845c59e0601bc07d) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: asyncrpc: Use client timeout for websocket open timeoutJoshua Watt2024-10-301-1/+5
| | | | | | | | | | | | The default connection timeout for websockets is 10 seconds, so use the provided client timeout instead (which defaults to 30 seconds). (Bitbake rev: 270c0c8bdc802c04fd3021c082d58cf8e8d68d82) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 23681775e5941e54ebead469addf708fca1e6beb) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: data_smart: Improve performance for VariableHistoryRobert Yang2024-08-261-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS $ bitbake -p -P Check profile.log.processed, the record() cost more than 20 seconds, it is less than 1 second when multiconfig is not enabled, and there would be the following error when more muticonfigs are enabled: Timeout while waiting for a reply from the bitbake server Don't change the type of loginfo['detail'] or re-assign it can make record() back to less than 1 second, this won't affect COW since loginfo is a mutable type. The time mainly affected by two factors: 1) The number of enabled layers, nearly 1 second added per layer when the number is larger than 50. 2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer when the layers number is larger than 50. (Bitbake rev: d251668d9a7a8dd25bd8767efb30d6d9ff8b1ad3) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser: Skip non-local functions for module dependenciesRichard Purdie2024-07-261-0/+5
| | | | | | | | | | | | | | | | If modules do something like "from glob import glob" then we end up checksumming the glob code. That leads to bugs as the code can change between different python versions for example, leading to checksum instability. We should ignore functions not from the current file as implemented by this change. (Bitbake rev: 11d83170922a2c6b9db1f6e8c23e533526984b2c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e6f862864539d6f6a0bea3e4479e0dd40ff3091) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: codeparser/data: Ensure module function contents changing is ↵Richard Purdie2024-07-262-12/+18
| | | | | | | | | | | | | | | | | | | | | | accounted for Currently, if a pylib function changes contents, the taskhash remains unchanged since we assume the functions have stable output. This is probably a poor assumption so take the code of the function into account in the taskhashes. This avoids certain frustrating build failures we've been seeing in automated testing. To make this work we have to add an extra entry to the python code parsing cache so that we can store the hashed function contents for efficiency as in the python module case, that isn't used as the key to the cache. The cache version changes since we're adding data to the cache. (Bitbake rev: 4bf332ccac283ca3440e81d8c781fcc23fe10b98) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b2c3438ebe62793ebabe2c282534893908d520b4) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Fix failure path for files that are empty or don't existRichard Purdie2024-06-241-9/+9
| | | | | | | | | | | | | | | When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: 8714a02e13477a9d97858b3642e05f28247454b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak to work on Fedora40Richard Purdie2024-06-241-3/+3
| | | | | | | | | | | | | | On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. (Bitbake rev: 5fa596b3f2e49fbd144347b9800c76de72a29e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-06-241-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 47678142e26bb76d1351886060deff5e75039bc9) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: tests/fetch: Tweak test to match upstream repo url changeRichard Purdie2024-06-141-1/+1
| | | | | | | | | | Upstream changed their urls, update our test to match. (Bitbake rev: 0791e66a47185d6f202af6be0d39e36a8e41850a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dc391b86540ec5e0a0f1d811c776a22d443b1c06) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: siggen: Enable batching of unihash queriesJoshua Watt2024-06-061-3/+6
| | | | | | | | | | | Uses the batching API of the client to reduce the effect of latency when making multiple queries to the server (Bitbake rev: fdac31655fb5441139f70e50292dbb5774cfdafe) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: hashserv: client: Add batch stream APIJoshua Watt2024-06-062-9/+172
| | | | | | | | | | | | | | Changes the stream mode to do "batch" processing. This means that the sending and reciving of messages is done simultaneously so that messages can be sent as fast as possible without having to wait for each reply. This allows multiple messages to be in flight at once, reducing the effect of the round trip latency from the server. (Bitbake rev: f99a17023b972d0d90dccb111f983655af6ccb87) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: bb: Use namedtuple for Task dataJoshua Watt2024-06-062-18/+39
| | | | | | | | | | | | | Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: bc7c44affe8e3fa94e92c3bcb8ad85bf11963779) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Improve rehash get_unihash parallelismRichard Purdie2024-06-061-2/+10
| | | | | | | | | | Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: 682fb42420eebf18f8a98bd3992baf9034dac5ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Process unihashes in parallel at initRichard Purdie2024-06-062-9/+10
| | | | | | | | | | | | | | | Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. (Bitbake rev: e0486054c7a4c637446c18608e9983cc8dc4d7fe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: siggen/runqueue: Report which dependencies affect the taskhashjoshua Watt2024-06-062-4/+6
| | | | | | | | | | | | | | Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) (Bitbake rev: 853423661779023763a87462b623b6e9ff2798b2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Allow rehash loop to exit in case of interruptsRichard Purdie2024-06-061-1/+3
| | | | | | | | | | | The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. (Bitbake rev: ca2f63d84b90cbd99936c9d7815d912d9019a594) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: runqueue: Add timing warnings around slow loopsRichard Purdie2024-06-061-0/+22
| | | | | | | | | | | | | | With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. (Bitbake rev: 9ee503c79936b13f1d45f9e43211f77a528cdbfa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: fetch2/gcp: Add missing runfetchcmd importSven Schwermer2024-05-291-0/+1
| | | | | | | | | | This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7) Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: parse: Improve/fix cache invalidation via mtimeRichard Purdie2024-05-291-4/+8
| | | | | | | | | | | | | | | | | | | | We have been seeing obscure failures in devtool, particularly on newer autobuilder workers where it appears the cache is assumed to be valid when it shouldn't be. We're using the 'seconds' granulation mtime field which is not really a good way of telling if a file has changed. We can switch to the "ns" version which is better however also add in inode number and size as precautions. We already have all this data and tuples are fast so there isn't really any cost to do so. This hopefully fixes [YOCTO #15318]. (Bitbake rev: 2aa519fec30ecf0e55202d6421fbba2f0320e211) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67) Signed-off-by: Steve Sakoman <steve@sakoman.com>
* bitbake: cooker: Handle ImportError for websocketsjoshua Watt2024-05-291-1/+2
| | | | | | | | | | | | | | Handles ImportError when creating a hash equivalence to ping the server. This notifies user earlier with a more precise error if websockets can't be used, and also prevents passing a known bad upstream value to the local server (Bitbake rev: 93190565fd1251e8f47d9a6291739f8b8ed5ec87) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74) Signed-off-by: Steve Sakoman <steve@sakoman.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-04 07:14:19 +0000