summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local()Robert Yang2024-07-131-26/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch can make the following settings much more faster: BB_GIT_SHALLOW = "1" BB_GENERATE_MIRROR_TARBALLS = "1" * The previous implementation was: - Make a full clone for the repo from local ud.clonedir - Use git-make-shallow to remove unneeded revs It was very slow for recipes which have a lot of SRC_URIs, for example vulkan-samples and docker-compose, the docker-compose can't be done after 5 hours. $ bitbake vulkan-samples -cfetch Before: 12 minutes Now: 2 minutes $ bitbake docker-compose -cfetch Before: More than 300 minutes Now: 15 minutes * The patch uses git shallow fetch to fetch the repo from local ud.clonedir: - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev Then the git repo will be shallow, and git-make-shallow is not needed any more. And git shallow fetch will download less commits than before since it doesn't need "rev^" to parse the dependencies, the previous code always need 'rev^'. (Bitbake rev: a5a569c075224fe41707cfa9123c442d1fda2fbf) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout errory75zhang2024-07-041-1/+6
| | | | | | | | | | | | | | | | * to avoid wrong http response in checkstatus function: in wget checkstatus() we are using 'HTTPConnectionCache' to share connections 1. state_file1(exists on http server) use shared connection <shared1> to send request 2. http_server recieved request of state_file1, but delayed by some reason to sent respone 3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1> 4. state_file2(not exists on http server) get shared connection <shared1> and send request 5. http_server finally send 200 response for state_file1 6. state_file2 recived 200 response and thought it was exists on http_server (Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd) Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/clearcase: remove True option to getVar calls in clearcase modulePeter Marko2024-07-021-2/+2
| | | | | | | | | | | | | Layer cleanup similar to https://git.openembedded.org/openembedded-core/commit/?id=26c74fd10614582e177437608908eb43688ab510 It was probably not found before beacause of the extra "d" parameter. That seem to be a bug as getVar does not support that. (Bitbake rev: 720189b810995c5737853458b7eb3779ca0df37e) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Skip non-local functions for module dependenciesRichard Purdie2024-07-011-0/+5
| | | | | | | | | | | | | | If modules do something like "from glob import glob" then we end up checksumming the glob code. That leads to bugs as the code can change between different python versions for example, leading to checksum instability. We should ignore functions not from the current file as implemented by this change. (Bitbake rev: 1e6f862864539d6f6a0bea3e4479e0dd40ff3091) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data: Ensure module function contents changing is ↵Richard Purdie2024-07-012-12/+18
| | | | | | | | | | | | | | | | | | | | accounted for Currently, if a pylib function changes contents, the taskhash remains unchanged since we assume the functions have stable output. This is probably a poor assumption so take the code of the function into account in the taskhashes. This avoids certain frustrating build failures we've been seeing in automated testing. To make this work we have to add an extra entry to the python code parsing cache so that we can store the hashed function contents for efficiency as in the python module case, that isn't used as the key to the cache. The cache version changes since we're adding data to the cache. (Bitbake rev: b2c3438ebe62793ebabe2c282534893908d520b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Remove invalid symlink for bb_cache.datRobert Yang2024-06-251-1/+1
| | | | | | | | | | | | | The bb_cache.dat might be an invalid symlink when error happens, then os.path.exists(symlink) would return False for it, the invalid symlink wouldn't be removed and os.symlink can't update it any more. Use os.path.islink(symlink) can fix the problem. (Bitbake rev: 1387d7b9ee3f270488f89b29f36f9f240e44accc) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also ↵Martin Jansa2024-06-181-7/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in get_unihashes * avoids long trace when BB_HASHSERVE points to non-existent socket file, e.g.: BB_HASHSERVE = "unix:///OE/no-socket.sock" or when running the build before starting the bin/bitbake-hashserv. * now it shows just warnings like it did in kirkstone many of them, e.g. 6 just for rebuilding zlib-native, but better than long trace for nonexistent socket file: WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/no-socket.sock: [Errno 2] No such file or directory for existing file, but before starting bin/bitbake-hashserv: WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/hashserv.sock: [Errno 111] Connection refused ERROR: An uncaught exception occurred in runqueue############################################################### | ETA: 0:00:00 Traceback (most recent call last): File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None): if typ == ADDR_TYPE_UNIX: > c.connect_unix(*a) elif typ == ADDR_TYPE_WS: File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'): self.loop.run_until_complete(self.client.connect_unix(path)) > self.loop.run_until_complete(self.client.connect()) File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client. py:150> exception=FileNotFoundError(2, 'No such file or directory')>): > return future.result() File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect(): if self.socket is None: > self.socket = await self._connect_sock() await self.setup_connection() File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock: sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) > sock.connect(os.path.basename(path)) finally: FileNotFoundError: [Errno 2] No such file or directory ERROR: Running idle function Traceback (most recent call last): File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None): if typ == ADDR_TYPE_UNIX: > c.connect_unix(*a) elif typ == ADDR_TYPE_WS: File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'): self.loop.run_until_complete(self.client.connect_unix(path)) > self.loop.run_until_complete(self.client.connect()) File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client. py:150> exception=FileNotFoundError(2, 'No such file or directory')>): > return future.result() File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect(): if self.socket is None: > self.socket = await self._connect_sock() await self.setup_connection() File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock: sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) > sock.connect(os.path.basename(path)) finally: FileNotFoundError: [Errno 2] No such file or directory Summary: There were 2 ERROR messages, returning a non-zero exit code. 1605616 09:29:05.369352 Parse cache valid 1605616 09:30:14.500863 Registering idle function <function BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7f43988c09a0> 1605616 09:30:14.500927 Removing idle function <bound method Command.runAsyncCommand of <bb.command.Command object at 0x7f43a961c350>> 1605616 09:30:14.573274 Exception Traceback (most recent call last): File "/OE/build/oe-core/bitbake/lib/bb/server/process.py", line 435, in idle_thread_internal retval = function(self, data, False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/cooker.py", line 1487, in buildTargetsIdle retval = rq.execute_runqueue() ^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1651, in execute_runqueue return self._execute_runqueue() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1567, in _execute_runqueue if self.rqdata.prepare() == 0: ^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1290, in prepare unihashes = bb.parse.siggen.get_unihashes(ready) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 713, in get_unihashes with self.client() as client: File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 595, in client self._client = hashserv.create_client(self.server, **self.get_hashserv_creds()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 88, in create_client raise e File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client c.connect_unix(*a) File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in connect_unix self.loop.run_until_complete(self.client.connect()) File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in connect self.socket = await self._connect_sock() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock sock.connect(os.path.basename(path)) FileNotFoundError: [Errno 2] No such file or directory broke the idle_thread, exiting 1605616 09:30:14.673756 Exiting (socket: True) 1605616 09:30:14.683153 Exiting as we could obtain the lock sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/OE/build/oe-core/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'> sys:1: ResourceWarning: unclosed <socket.socket fd=17, family=1, type=1, proto=0> ResourceWarning: Enable tracemalloc to get the object allocation traceback (Bitbake rev: 550c86969e5a137ffef61b08a520a4855232fb1c) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-06-071-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid save_unitaskhashesRichard Purdie2024-06-071-3/+0
| | | | | | | | | | The save comes with an IO overhead which can slow down the rehash loop in bitbake a lot. We only needed to do this when recipes were doing unihash cache copying. Now they aren't doing that, drop this IO pain point. (Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop copy_unihashes functionRichard Purdie2024-06-071-6/+0
| | | | | | | | | | The code in OE-Core using this has been replaced with something more fit for purpose. Drop these function calls as they were never a great idea in the first place and cause IO slowdown for runqueue needing to sync the cache. (Bitbake rev: 2c8fa57778c4bd2a5c48a60b701ac57de4289cb2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Tweak test to match upstream repo url changeRichard Purdie2024-06-051-1/+1
| | | | | | | | Upstream changed their urls, update our test to match. (Bitbake rev: dc391b86540ec5e0a0f1d811c776a22d443b1c06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Use client timeout for websocket open timeoutJoshua Watt2024-06-041-1/+5
| | | | | | | | | | The default connection timeout for websockets is 10 seconds, so use the provided client timeout instead (which defaults to 30 seconds). (Bitbake rev: 23681775e5941e54ebead469addf708fca1e6beb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Tweak to work on Fedora40Richard Purdie2024-05-311-3/+3
| | | | | | | | | | | | On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1 and python only binds to one of the addresses, leading to test failures. Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying to test things other than the host networking. (Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix failure path for files that are empty or don't existRichard Purdie2024-05-311-9/+9
| | | | | | | | | | | | | When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Batch unihash_exists checksJoshua Watt2024-05-311-7/+7
| | | | | | | | | | Similar to looking up unihashes, use the batch API when checking if a unihash exists to speed up lookups (Bitbake rev: 0ac521ff37b578f7487bca0eccc7dc9e5974991b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Remove ClientPoolJoshua Watt2024-05-312-76/+2
| | | | | | | | | | | | Batching support on the client side has proven to be a much more effective way of dealing with server latency than multiple client connections and is also much nicer on the server, so drop the client pool support from asyncrpc and the hash server (Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop client pool supportJoshua Watt2024-05-311-38/+15
| | | | | | | | | | Drops support for client pools, since batching support in the client code has proven to be much more effective (Bitbake rev: 85dafaf8e070459f7de7bfb37300d8b60a27002e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop older python version compatibility codeRichard Purdie2024-05-314-29/+6
| | | | | | | | | | | | | | cooker: We can call multiprocessing close() unconditionally and tweak a comment give 3.8 is now the minimum version. lib/bb: We can drop the logger addition code only needed before 3.6 asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the conditional code. (Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve handling errors during parsing when profilingRichard Purdie2024-05-311-4/+5
| | | | | | | | | We've seeing profiling tracebacks when parse errors occur during profiling. Try and avoid these but not processing invalid profiles. (Bitbake rev: 171bd9dd575307fbd61b5179ad86131d76add067) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: server: Add support for SO_REUSEPORTJoshua Watt2024-05-301-7/+27
| | | | | | | | | | | | | | SO_REUSEPORT is a socket option that allows multiple servers to listen on the same TCP port, and the kernel will automatically load balance the connections between them. This is particularly helpful for the hash server since it runs in a single thread. To take advantage of a multi-core server, multiple servers can be started in parallel with this option (up to 1 per CPU) and the kernel will load balance between them. (Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Enable batching of unihash queriesJoshua Watt2024-05-301-3/+6
| | | | | | | | | | Uses the batching API of the client to reduce the effect of latency when making multiple queries to the server (Bitbake rev: a54734b4ac2ddb3bce004e576cf74d6ad6caf62a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb: Use namedtuple for Task dataJoshua Watt2024-05-282-18/+39
| | | | | | | | | | | | Task dependency data is becoming unwieldy with the number of indices it contains. Convert it to use a named tuple instead, which allows members to be indexed by a named property or an index (which allows it to retain backward compatibility). (Bitbake rev: 26446cca4d22734c3f1b328a205c169dadb7e494) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve rehash get_unihash parallelismRichard Purdie2024-05-281-2/+10
| | | | | | | | | Improve the rehash code to query unihashes in parallel since this is more efficient on slower links. (Bitbake rev: c1949d5350342eaaf6ab988d7bfba99496d55523) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Process unihashes in parallel at initRichard Purdie2024-05-282-9/+10
| | | | | | | | | | | | | | Improve the runqueue init code to call unihash queries in parallel since this is faster and more efficient, particularly on slower links with longer round trip times. The call to the function from cooker is unneeded since that function calls prepare() and hence this functionality will already have run, so drop that obsolete call. (Bitbake rev: 721c97a115a7a4bf21955be79391bd6e0099f40e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Allow rehash loop to exit in case of interruptsRichard Purdie2024-05-281-1/+3
| | | | | | | | | | The initial hash serve loop exits in the case where interrupts are present but probably checks a bit too often. Tweak that and also allow the slow rehash loop to break on interrupt, improving bitbake Ctrl+C response. (Bitbake rev: 4534365591fd17bcc2b684900863b67bc69519ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add timing warnings around slow loopsRichard Purdie2024-05-281-0/+22
| | | | | | | | | | | | | With hashserve enabled, there are two slow paths/loops, one at initial runqueue generation and also during the rehash process when new outhashes are found. Add timing information at the hashserve log level for when these loops take longer than 30s or 60s overall. This will leave evidence in the logs when things are running particularly slowly. (Bitbake rev: 6c357ede08e0b2a93bdaad2c1d631994faf2b784) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ui/buildinfohelper: Add exception treatment to fix missing target_fileMarlon Rodriguez Garcia2024-05-281-1/+4
| | | | | | | | | | | Based on the discution on https://lists.yoctoproject.org/g/toaster/message/6157 in some cases the value for Target_file could be missing and is needed to bypass it to finish build. (Bitbake rev: c60f6d20911632b41473f7c8577949be2f99ad80) Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: The fetcher shouldn't have any knowledge of SRichard Purdie2024-05-231-1/+1
| | | | | | | | | | | | | I don't know why there is hardcoded knowledge of S in the fetcher but there shouldn't be and the OE unpack changes highlight this doing things it shouldn't. Drop the S reference and use rootdir which is the only place it should be touching. (Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests.codeparser: add tests for shell expansionsAntonin Godard2024-05-211-0/+40
| | | | | | | | | | | Tests quotes around `` and $() expansions, nested and multiple expansions, and that escaped quotes are treated as characters by the parser. (Bitbake rev: d98130cb4d500c495bc692c56dde3e019f36320a) Signed-off-by: Antonin Godard <antoningodard@pm.me> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: remove redundant list conversionAntonin Godard2024-05-211-1/+1
| | | | | | | (Bitbake rev: 89712949de9476e4674864a8dcd6862fefe92eae) Signed-off-by: Antonin Godard <antoningodard@pm.me> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: support shell substitutions in quotesAntonin Godard2024-05-211-7/+22
| | | | | | | | | | | | | | | | | | | | | | | | | The current shell substitution mechanism only works without quotes. For example: var1=$(cmd1 ...) Will work and add `cmd1` to the correspondind `run.do_*` file. However, although quite common, this syntax is not supported: var1="$(cmd1 ...)" This commit adds this feature by adding a step to process_words() to check whether we are dealing with quotes first, and by iterating on what's between them to detect new shell substitution candidates. These candidates are tested and parsed like before in the next step. The original `part` being part of the candidates means the syntax var1=$(cmd1 ...) is still valid. (Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e) Signed-off-by: Antonin Godard <antoningodard@pm.me> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure generateTaskDepTreeData fails for NoProviderRichard Purdie2024-05-171-3/+3
| | | | | | | | | | If an invalid provider is requested, error out early rather than trying to build partial runqueue data structures as the taskdep UI will have exited after seeing the bad provider. (Bitbake rev: a478087998cb794cc4e31189b3ce07973d3949bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Improve/fix cache invalidation via mtimeRichard Purdie2024-05-161-4/+8
| | | | | | | | | | | | | | | | | | We have been seeing obscure failures in devtool, particularly on newer autobuilder workers where it appears the cache is assumed to be valid when it shouldn't be. We're using the 'seconds' granulation mtime field which is not really a good way of telling if a file has changed. We can switch to the "ns" version which is better however also add in inode number and size as precautions. We already have all this data and tuples are fast so there isn't really any cost to do so. This hopefully fixes [YOCTO #15318]. (Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc/client: Fix websockets minimum version for python 3.10Richard Purdie2024-05-161-1/+3
| | | | | | | | | | | | python 3.10 support is only available in websockets 10.0 and later: https://github.com/python-websockets/websockets/commit/08d8011132ba038b3f6c4d591189b57af4c9f147 Update the version for this case. This avoids failures on Ubuntu 22.04. (Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: update to version 2.9.1Richard Purdie2024-05-081-1/+1
| | | | | | | | This allow the use of new siggen API (Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: Check websockets versionjoshua Watt2024-05-081-0/+22
| | | | | | | | | | | Checks that the minimum version of the websockets module is present, and if not raises an ImportError. This allows the user to get earlier feedback if using websockets is going to succeed (Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle ImportError for websocketsjoshua Watt2024-05-081-1/+2
| | | | | | | | | | | | Handles ImportError when creating a hash equivalence to ping the server. This notifies user earlier with a more precise error if websockets can't be used, and also prevents passing a known bad upstream value to the local server (Bitbake rev: aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue: Report which dependencies affect the taskhashjoshua Watt2024-05-082-4/+6
| | | | | | | | | | | | | Report which task dependencies in BB_TASKDEPDATA are included in the taskhash. This allows tasks to identify which tasks dependencies may change without the task re-running. Knowing this information is important for tasks that want to transfer information from dependencies (such as SPDX) (Bitbake rev: a313b4f07727e8187526157ba039911c3f73dd46) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix mirroring issue with svnKari Sivonen2024-05-082-1/+12
| | | | | | | | | | | | Add return false to supports_checksum for svn fetcher which fhis fixes MIRROR usage for svn uris. Also add a testcase. [YOCTO #15473] (Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f) Signed-off-by: Kari Sivonen <kari.sivonen@live.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Handle conflict between cwd and cleandirsRichard Purdie2024-05-081-0/+2
| | | | | | | | | | If the cwd of the task is also a cleandirs, you would see warnings from bitbake about being unable to obtain cwd during the task execution. Tweak the code to detect this and avoid the warnings. (Bitbake rev: 6c7fd60c10955b0f23f64b25b5b4e154eb22a8f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: add upstream latest version check functionAlexander Kanavin2024-05-082-0/+25
| | | | | | | | | | | This is actually rather easy: crate web API provides a json with all the versions, for example: https://crates.io/api/v1/crates/cargo-c/versions (Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Use hash client to ping upstream serverjoshua Watt2024-05-021-4/+3
| | | | | | | | | | | | | | The cooker attempts to connect to the upstream hash equivalent server to warn the user early if it is misconfigured. However, this was making the assumption that it was a raw TCP connection and failed when attempting to use a websocket upstream server. Fix this by creating an hash client and using the ping API to check the server instead of using a raw socket. (Bitbake rev: 5e84c13a6c594ed34c341849806657ddda206714) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gcp: Add missing runfetchcmd importSven Schwermer2024-04-201-0/+1
| | | | | | | | | This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: 97ffe14311407f6e705ec24b70870ab32f0637b9) Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.9.0 development version postreleaseRichard Purdie2024-04-161-1/+1
| | | | | | (Bitbake rev: 67a1aa8dbb3cb3a30fa7d697431ebb30323e4f28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump to version 2.8.0Richard Purdie2024-04-161-1/+1
| | | | | | (Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Capture SSL environment for hashserverJoshua Watt2024-04-161-24/+70
| | | | | | | | | | | | | | Now that the bitbake hash server supports SSL connections, we need to capture a few environment variables which can affect the ability to connect via SSL. Note that the variables are only put in place to affect the environment while actually invoking the server [RP: Tweak to use BB_ORIGENV as well] [RP: Tweak to handle os.environ restore correctly] (Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler: Handle unclosed functions correctlyRichard Purdie2024-04-141-1/+7
| | | | | | | | | | | | | | | | | A function accidentally defined as: somefunction() { : } which is unclosed due to the space at the end, would currently silently cause breakage. Have the parser throw and error for this. [YOCTO #15470] (Bitbake rev: a7dce72da6be626734486808f1b731247697e638) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: asyncrpc: include parse_address from hashservMichael Opdenacker2024-04-141-0/+23
| | | | | | | | | | | | | | Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients. (Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Suggested-by: Joshua Watt <JPEWhacker@gmail.com> Cc: Tim Orling <ticotimo@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add support for hashserve credentialsJoshua Watt2024-03-231-5/+16
| | | | | | | | | | Adds support for hashserver credentials to be specified in the SignatureGenerator (Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-24 00:42:42 +0000