summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: ast: Fix EXPORT_FUNCTIONS bugRichard Purdie2024-01-102-4/+103
| | | | | | | | | | | | | | | | If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton and a standard funciton definition for the function that is exported, the export function can sometimes overwrite the standard one. The issue is that the internal flag the code uses isn't ovweritten if the variable is giving a new value. Fix the issue by using a comment in the code that is injected so that we know if it is ours or not. Also add some testing for EXPORT_FUNCTIONS, not perfect but a start. (Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/event: Add test_lineno_in_eventhandlerRobert Yang2024-01-101-0/+24
| | | | | | | | | Add test_lineno_in_eventhandler to test lineno in eventhandler. (Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: event: Inject empty lines to make code match lineno in ↵Robert Yang2024-01-101-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | filename So that we can get the correct error messages. * In python 3.10.9, the error message was: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 4 # SPDX-License-Identifier: MIT ^^^^^ SyntaxError: invalid syntax This is hard to debug since the error line number 4 is incorrect, but nothing is wrong with the code in line 4. * Now the error message and lineno is correct: ERROR: Unable to register event handler 'defaultbase_eventhandler': File "/path/to/poky/meta/classes-global/base.bbclass", line 256 an error line ^^^^^ SyntaxError: invalid syntax And no impact on parsing time: * Before: $ rm -fr cache tmp; time bitbake -p real 0m27.254s * Now: $ rm -fr cache tmp; time bitbake -p real 0m27.200s (Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Version bump for find_siginfo chanagesRichard Purdie2024-01-051-1/+1
| | | | | | | | Bump the version to 2.7.1 for the find_siginfo changes. (Bitbake rev: 03995e16bf7186f5368f772f617d563f4d280641) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: prioritize local stamps over sstate signatures in ↵Alexander Kanavin2024-01-051-0/+3
| | | | | | | | | | | | | | | | | | | printdiff Even with the reworked printdiff code, sstate which is heavily used in parallel can throw races at the tests: if a new matching, but otherwise unrelated sstate signature appears between writing out local stamps and listing matching sstate files, then that signature will be deemed 'the latest' and the actual local stamp will be discarded. This change ensures the scenario does not happen. It also makes use of the reworked find_siginfo(), particularly the 'sstate' entry in returned results. (Bitbake rev: c8574b796dabb69699c70540dd95a44d8f7388ab) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure version of siggen is verifiedRichard Purdie2024-01-053-0/+12
| | | | | | | | | | Since we need to change the form of the siggen function, we need to add versioning and some verison checks. This means if a newer bitbake is used with older metadata we can detect it. (Bitbake rev: 721556568413508213d22c29985e305a45a8d68a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs/runqueue: adapt to reworked find_siginfo()Alexander Kanavin2024-01-051-5/+5
| | | | | | | | | | In particular having 'time' explicitly used as a sorting key should make it more clear how the entries are being sorted. (Bitbake rev: 5439aca056c84ab4410aaf24bdb68e896191d8e1) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Post release version bump to 2.7.0Richard Purdie2024-01-051-1/+1
| | | | | | | | Bump to a development version post release. (Bitbake rev: 28364c08f36c778a5cb2e3f20ceb052370ef153c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/runqueue: add debugging for find_siginfo() callsAlexander Kanavin2024-01-051-0/+4
| | | | | | | (Bitbake rev: 52f5503e8cf048331134233fb681db6dc736ae38) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/codeparser.py: address ast module deprecations in py 3.12Alexander Kanavin2024-01-021-10/+10
| | | | | | | | | | | | | Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s (Bitbake rev: de8ba2770d9a1a94af3d084f9540da7e2fae6022) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add workaround for libgcc issues with python 3.8 and 3.9Richard Purdie2023-12-301-0/+7
| | | | | | | | | | | | | | | | | | | | With python 3.8 and 3.9, we see intermittent errors of: libgcc_s.so.1 must be installed for pthread_cancel to work Aborted (core dumped) which seem related to: https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work https://bugs.ams1.psf.io/issue42888 These tend to occur on debian 11 and ubuntu 20.04. Workaround this by ensuring libgcc is preloaded in all cases. (Bitbake rev: c6666f6cfafd55e1c980239a7c5ff908f1a69196) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Remove tie between rqexe and starts_workerRichard Purdie2023-12-171-15/+8
| | | | | | | | | | | We've been moving to try and separate several pieces of runqueue. Allow start_worker to operate separately to rqexe since they don't need to be tied. This allows rqexe to be available to print_diff for future improvements. (Bitbake rev: 834e452243ff2eea6e8e2e7f4935b5233ffb4b00) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix mkdir with PosixPathRichard Purdie2023-12-171-1/+1
| | | | | | | | | Avoid: Exception: AttributeError: 'PosixPath' object has no attribute 'find' (Bitbake rev: 0b37fe89ba12549109905b6d0e6d07d342162436) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Added new feature to import eventlogs from command line ↵Marlon Rodriguez Garcia2023-12-122-1/+87
| | | | | | | | | | | | | | | | | | | | | | | | into toaster using replay functionality Added a new button on the base template to access a new template. Added a model register the information on the builds and generate access links Added a form to include the option to load specific files Added jquery and ajax functions to block screen and redirect to build page when import eventlogs is trigger Added a new button on landing page linked to import build page, and set min-height of buttons in landing page for uniformity Removed test assertion to check command line build in content, because new button contains text Updated toaster_eventreplay to use library Fix test in test_layerdetails_page Rebased from master This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read the files created by bitbake Exclude listing of files that don't contain the allvariables definitions used to replay builds This part of the feature should be revisited. Over a long period of time, the BB_DEFAULT_EVENTLOG will exponentially increase the size of the log file and cause bottlenecks when importing. (Bitbake rev: ab96cafe03d8bab33c1de09602cc62bd6974f157) Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Do not create directories with ${ in the namePavel Zhukov2023-12-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In some cases ${ may not be expanded in the WORKDIR (one of the cases is undefined variable) and causes cryptic failures [1]. Guard this by erroring out if directory name contains ${ Fixes: [Yocto #15255] [1] ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_hardcode_path(d) 0003: File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path 0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)) 0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir) 0652: 0653: # If the fixmefn is empty, remove it.. *** 0654: if os.stat(fixmefn).st_size == 0: 0655: os.remove(fixmefn) 0656: else: 0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd)) 0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath' (Bitbake rev: e91c256ec076e70cf8a18e369fe7862e50618c48) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Move to version 2.6.1 to mark runqueue changesRichard Purdie2023-12-061-1/+1
| | | | | | (Bitbake rev: 651a6dcf6f8ff33a4e9290a37c23e4f243974ac3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/toaster: Fix assertEquals deprecation warningsRichard Purdie2023-12-061-1/+1
| | | | | | | | | Fix: DeprecationWarning: Please use assertEqual instead (Bitbake rev: dd990ea6843685927954101feb729f3faa3a16d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve inter setscene task dependency handlingRichard Purdie2023-12-061-13/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The way the code currently handles dependencies between setscene tasks is fairly poor, basically by deleting chunks of dependencies and adding reversed dependency relationships. This was once the best way to handle things but now a lot of the surrounding code has changed and this approach is suboptimal and can be improved. This change firstly adds debug logging for "hard" setscene task dependencies since previously the codepaths were missing from logs making them very hard to read. The changes to the setscene dependency graph are removed entirely this these altered graphs were a significant source of problems. Instead, if a hard dependency is run into, we mark the hard dependency as buildable and defer the task until the hard dependencies are met. The code now also skips the check_dependencies() code for hard dependencies since previously that code was having to list all possible hard dependencies. We don't need to do that as we can safely assume hard dependencies are required. With these changes to runqueue's behaviour, we stand some chance of being able to fix other bugs in OE-Core related to useradd for example. (Bitbake rev: 367789b53c1c22ec26e0f4836cdf2bdd9c7d84fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.Charlie Johnston2023-12-021-2/+5
| | | | | | | | | | | | | | | | | The GCP fetcher was calling bb.fetch2.check_network_access with "gsutil stat" as the command, but then never actually ran that command to check if the file exists. In cases where the file did not exist in a gs:// premirror, this would lead to an unhandled exception from do_fetch when the GCP python API tried to perform the download. This change resolves that issue by adding a runfetchcmd to call gsutil. (Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229) Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests: Fix duplicate test_underscore_override testRichard Haar2023-12-021-10/+10
| | | | | | | | | Found a duplicate test, added _2 suffix to one, 74 tests now pass up from 73. (Bitbake rev: ae2a19dadb4f3065b8731a61f45f29e6a70af402) Signed-off-by: Richard Haar <rh@richhaar.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid eventlog variable listing lockupsRichard Purdie2023-11-271-25/+11
| | | | | | | | | | | | | | | If the event log is enabled and parsing the metadata triggers log messages, the event code and deadlock. Iterating the variables inside the event handling code causes this. SOURCE_DATE_EPOCH triggers a python function which calls bb.debug() and can trigger a lockup as one example. Move the code around and add it to the BuildStarted events explictly. This does mean runs without builds no longer get variables added to the eventlog however we can look into a more targetted version of data if/as/where neded. (Bitbake rev: 4135a617ae16d509362b5bf56378139cdc0876d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/ncurses: Add missing function call to avoid tracebackRichard Purdie2023-11-271-0/+3
| | | | | | | | | The ncurses UI wasn't working due to a missing function call. Add it to avoid a traceback when starting builds. (Bitbake rev: db8f36b69a68de2179e5685cf24a42ec10d68257) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/aws: forward env-vars used in gitlab-ci K8sFelix Moessbauer2023-11-231-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds the following variables to the allow-list, which are used in the "IAM roles for AWS when using the GitLab chart": - AWS_ROLE_ARN - AWS_WEB_IDENTITY_TOKEN_FILE These variables are set in the CI job environment and are needed to access the sstate cache artifacts in a connected S3 bucket. [1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com> (Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: git: add missing destsuffix and subpath parameters in ↵Julien Stephan2023-11-231-0/+8
| | | | | | | | | | | | docstrings Docstring for git fetcher is missing destsuffix and subpath parameters, so add them (Bitbake rev: 7cea6c2efefbb0f789f654c04a61d9ba770c148b) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: git: Update Git-LFS download and testsDesone Burns2023-11-232-28/+47
| | | | | | | | | | | | | | | | | | | | When downloading a Git repository containing an LFS, the Git hooks are not always initialized correctly to perform the download. This change updates the Git downloader to run the "git lfs install" command in order to "smudge" the LFS files. The tests have also been updated to do checks for situations in which git lfs is not installed, as the application was required to be installed for any of the tests to run previously. The Git LFS functionality was working to some degree previously, but this change also updates the fetcher to allow LFS downloads for nobranch SRC_URIs. (Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217) Signed-off-by: Desone Burns II <dburns@seegrid.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Simplify virtualfn2realfn()Peter Kjellerstedt2023-11-231-4/+2
| | | | | | | | | Also correct the description of variant2virtual(). (Bitbake rev: d766e9bd22ec6392fbf1694eea5032b9d09f1949) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Be consistent with what type bb_data representsPeter Kjellerstedt2023-11-231-6/+5
| | | | | | | | | | | | | | | | In _parse_recipe(), bb_data was initially a datastore object, but then after the call to bb.parse.handle() it was all of a sudden turned into a dict of variant:datastore pairs. At the same time, parseRecipeVariants() used bb_data for datastore objects, while parseRecipe() used it for the dicts of variant:datastore pairs. Change these functions to consistently use "bb_data" for datastore objects and use "datastores" for the dicts of variant:datastore pairs. (Bitbake rev: ed0f94c037476efd465db87726b2480406076beb) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Make parseRecipeFile() handle virtual recipes correctlyPeter Kjellerstedt2023-11-231-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Running `devtool search gcc` would result in errors like this for virtual recipes: ERROR: When reparsing .../meta/recipes-devtools/gcc/libgcc-initial_13.2.bb:do_populate_sysroot, the basehash value changed from b1cd809ed98cef9db0fb1b17d34c4083e739c336f9d5619b89715b0294d81af5 to 44c2f92781dc4a20e98b7bb4724e204e64b101905fa75e71241a574b725997dc. The metadata is not deterministic and this needs to be fixed. ERROR: The following commands may help: ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Snone ERROR: Then: ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Sprintdiff The reason was the newly introduced :layer-<layername> override, which is used, e.g., in meta/classes-global/insane.bbclass to add the patch-status QA test only for the meta layer: ERROR_QA:append:layer-core = " patch-status" When tinfoil parsed the recipes using the parseRecipeFile() function, it failed to properly identify the layername for virtual recipes, which resulted in the error above. The correct thing to do is to make parseRecipeFile() call bb.cache.virtualfn2realfn() to convert the virtual filename into a real filename and virtual class. (Bitbake rev: da2aed134412f5de04d7b540f92d735983ad0108) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid sideeffects for autorev from getAllKeysWithFlagsRichard Purdie2023-11-231-0/+7
| | | | | | | | | | | | | If we expand the variable AUTOREV in OE-Core, it triggers side effects in the fetcher. The situation isn't ideal and needs improvement but this breaks and is blocking enabling BB_DEFAULT_EVENTLOG. Hack around the issue for now so we unblock things until we can work out a better plan for how to improve AUTOREV support. (Bitbake rev: cb9b6530f3d12c56a8b48847af2e7461924205d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add support for BB_DEFAULT_EVENTLOGRichard Purdie2023-11-231-8/+17
| | | | | | | | | | Currently it is only possible to specify an eventlog on the bitbake commandline. Add a variable that can be used in bitbake.conf so that we can log data by default more easily. (Bitbake rev: ee174b231897a53cdde0f68769518342e53210cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: utils: remove spaces on empty linesJulien Stephan2023-11-201-8/+8
| | | | | | | | (Bitbake rev: 979ff8b4228e4c6f90985e9a3faa5fc9f3922218) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Refactor StaleSetSceneTasks event out of ↵Richard Purdie2023-11-161-18/+18
| | | | | | | | | | | | | | build_scenequeue_data The event at the end of the function doesn't really belong there, move it out so the function becomes cleaner. It also avoids having to pass a number of parameters around, another sign this is the correct thing to do. This continues refactoring started a while ago to improve and modularise the code. (Bitbake rev: f712b927a84e8b8deaa8c907e8f9bcdec681f2aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Move 'cantskip' into sqdataRichard Purdie2023-11-161-7/+7
| | | | | | | | | Conceptually, this belongs in sqdata so this is the next step in refactoring this code, continuing what was started a while ago. (Bitbake rev: 2cf267cb657cd1864db43955cc010abba7a64093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add option to set log level when running as a processJoshua Watt2023-11-142-2/+5
| | | | | | | | | | | | | | | | When running an asyncrpc server as a subprocess, it is often desired to run it with a lower logging level since the normal logging of clients connecting and disconnecting is not desired. As such, add an option to set the logging level of the server when running as a subprocess and set the level to WARNING when starting a local hashserver or PRserver (Bitbake rev: 61dac7b99ad6d2a858f85d8ed1b5524d558be6c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: client: Add disconnect APIJoshua Watt2023-11-091-1/+7
| | | | | | | | | | Adds an API to explicitly disconnect a client. This can be useful for testing the auto-reconnect behavior of clients (Bitbake rev: cb2fec7eaca20608faf4b8ff11ec8590ac7f2229) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: client: Prevent double closing of loopJoshua Watt2023-11-091-4/+6
| | | | | | | | | | Invalidate the loop in the client close() call so that it is not closed twice (which is an error in the asyncio code) (Bitbake rev: ef22f8ee82c242383625f078baafb629e45dad7e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add InvokeErrorJoshua Watt2023-11-094-4/+22
| | | | | | | | | | | | | Adds support for Invocation Errors (that is, errors raised by the actual RPC call instead of at the protocol level) to propagate across the connection. If a server RPC call raises an InvokeError, it will be sent across the connection and then raised on the client side also. The connection is still terminated on this error. (Bitbake rev: 50ee68175e7cf20a32bfbb176db2c47d7859da04) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add SQLalchemy backendJoshua Watt2023-11-091-2/+9
| | | | | | | | | | | Adds an SQLAlchemy backend to the server. While this database backend is slower than the more direct sqlite backend, it easily supports just about any SQL server, which is useful for large scale deployments. (Bitbake rev: e0b73466dd7478c77c82f46879246c1b68b228c0) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Prefix log messages with client infoJoshua Watt2023-11-091-3/+18
| | | | | | | | | | Adds a logging adaptor to the asyncrpc clients that prefixes log messages with the client remote address to aid in debugging (Bitbake rev: f4d64ce73c2449c008ff5d9b32376a2893ef7195) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Add context manager APIJoshua Watt2023-11-091-0/+13
| | | | | | | | | | | | | | | | Adds context manager API for the asyncrcp client class which allow writing code that will automatically close the connection like so: with hashserv.create_client(address) as client: ... Rework the bitbake-hashclient tool and PR server to use this new API to fix warnings about unclosed event loops when exiting (Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add websocket connection implementationJoshua Watt2023-11-093-2/+106
| | | | | | | | | | | | | | | Adds support to the hash equivalence client and server to communicate over websockets. Since websockets are message orientated instead of stream orientated, and new connection class is needed to handle them. Note that websocket support does require the 3rd party websockets python module be installed on the host, but it should not be required unless websockets are actually being used. (Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Abstract socketsJoshua Watt2023-11-095-228/+298
| | | | | | | | | | | | | Rewrites the asyncrpc client and server code to make it possible to have other transport backends that are not stream based (e.g. websockets which are message based). The connection handling classes are now shared between both the client and server to make it easier to implement new transport mechanisms (Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix errors when using -S printdiffRichard Purdie2023-11-091-2/+2
| | | | | | | | | | | | | | | | | When running printdiff, messages like: "Can't find a task we're supposed to have written out?" might be shown. This happens when hashequivalence is enabled and is due to the use of "hash" when "unihash" should be used (which defaults back to hash). Changing this fixed various errors oe-selftest highlighted. Also print the task ID when erroring about a missing task as it aids debugging. (Bitbake rev: e7783bbe0559abb91f5038cf83575c0f25138af2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option for setscene tasksRichard Purdie2023-11-091-0/+1
| | | | | | | | | | | If --runall is used with setscene tasks, it will still skip them if it can. By marking the tasks as targets, this avoids that skipping and means --runall deploy_source_date_epoch works as expected for example. (Bitbake rev: 8c6cd1e8dd224bc1cc92736cbb7e113fd1fb5aae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: add header with length of messageEtienne Cordonnier2023-11-061-12/+22
| | | | | | | | | | | | | | | | | | | | | | | | | The IPC mechanism between runqueue.py and bitbake-worker is currently not scalable: The data is sent with the format <tag>pickled-data</tag>, and bitbake-worker has no information about the size of the message. Therefore, the bitbake-worker is calling select() and read() in a loop, and then calling "self.queue.find(b"</" + item + b">")" for each chunk received. This does not scale, because queue.find has a linear complexity relative to the size of the queue, and workerdata messages get very big e.g. for builds which reference a lot of files in SRC_URI. The number of chunks varies, but on my test system a lot of chunks of 65536 bytes are sent, and each iteration takes 0.1 seconds, making the transfer of the "workerdata" data very slow (on my test setup 35 seconds before this fix, and 1.5 seconds after this fix). This commit adds a 4 bytes header after <tag>, so that bitbake-worker knows how many bytes need to be received, and does not need to constantly search the whole queue for </tag>. (Bitbake rev: 595176d6be95a9c4718d3a40499d1eb576b535f5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: fix lfs fetch with destsuffix paramRagesh Nair2023-10-231-2/+2
| | | | | | | | | | | | | | | Pre-fetching of LFS blobs now also works if a destsuffix parameter is supplied. This also fixes issues with Yocto Kirkstone (and newer) builds, where fetching the LFS blobs is retried unsuccessfully during do_unpack, as network access is blocked for any task other than do_fetch. (Bitbake rev: e411dc07d032be6811d0393c50a06fc28e669b24) Signed-off-by: Ragesh Nair <ragesh.nair@duagon.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: add missing 'import os'Chris Laplante2023-10-231-0/+1
| | | | | | | | (Bitbake rev: 302969885d37a76edec3aa79181e98f8d7e28021) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: set has 'add', not 'append' methodChris Laplante2023-10-231-1/+1
| | | | | | | | | | Discovered via pylint (Bitbake rev: 1a353cda696b7f59386ad2d78a57005b90a37da4) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-19 15:35:56 +0000