summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch2/wget: handle HTTP 308 Permanent RedirectRoss Burton2024-12-201-5/+37
| | | | | | | | | | | | | | | urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be worked around without copying the entire redirect_request() method. When we can depend on Python 3.13, FixedHTTPRedirectHandler can be removed. (Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch: add npmsw test case for bundled dependenciesStefan Herbrechtsmeier2024-12-201-0/+26
| | | | | | | | | | | | | | | The npm package lock and shrinkwrap file list bundled dependencies which are supplied together with the parent dependency. The bundled dependencies are marked by a flag. The flag and thereby test depends on the lock file version. The old lock file version uses a `bundled` flag and stores dependencies in the `dependencies` list. The new lock file version uses an `inBundle` flag and stores dependencies in the `packages` list. (Bitbake rev: 34fd8ea6abe755e04220fe70b082aa620ae15f86) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: ignore bundled dependencies in packagesStefan Herbrechtsmeier2024-12-201-0/+2
| | | | | | | | | | | | The npm package lock and shrinkwrap file list bundled dependencies which are supplied together with the parent dependency. The flag depends on the lock file version. Add the missing `inBundle` flag to packages list processing. (Bitbake rev: 042d7121ddc93c18306bf03d26ea774f66379a90) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch: do not skip plain npmsw testsStefan Herbrechtsmeier2024-12-201-6/+0
| | | | | | | | | | | The npmsw fetcher doesn't use the npm command. Remove the `@skipIfNoNpm()` decorator from plain npmsw tests which don't use the npm fetcher. (Bitbake rev: 48f9964a19b8e2e3a73667c9889908eceecf7719) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: fetch: adapt npmsw tests to fixed unpack behaviorStefan Herbrechtsmeier2024-12-201-8/+5
| | | | | | | | | | Adapt the npmsw tests to commit 84f102954e10 ("fetch/npmsw: The fetcher shouldn't have any knowledge of S"). (Bitbake rev: 8194178291d9b47efb2079fe81b704ba6171efc1) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Add include_all conf file directiveRichard Purdie2024-12-172-0/+24
| | | | | | | | | | | | | | | | | | | | In some cases it would be helpful to be able to have an include file in a standard location which is included in all layers that are added to the system. The intent is for these to provide configuration tweaks of specific types so that a given file pattern can be adopted more widely for such configuration. The code will search for any named configuration file within BBPATH, so a configuration directive of: include_all conf/distro/include/myinc.conf would include the myinc.conf file in that subpath if present in any directory in BBPATH. Multiple files will be included if present. (Bitbake rev: d01d5593e7829ac60f37bc23cb87dc6917026471) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mcChris Laplante2024-12-173-12/+36
| | | | | | | | | | | | | | | | | | Previously, the cooker skiplist was shared across multiconfigs (including default ''). If you had a recipe that was incompatible with several multiconfigs for different reasons, then the displayed reason (i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might vary across invocations of bitbake. This was caused by the random order in which recipes are parsed under different multiconfig contexts, with each skip reason overwriting the previously assigned reason. I hit this specificially when using COMPATIBLE_MACHINE, but COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too. (Bitbake rev: c51f01a35ed9a928402eab0899598b5c59602eef) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/tests: use bb.build.listtasks() instead of __BBTASKSRoss Burton2024-12-131-1/+1
| | | | | | | | | | | | | Use the new listtasks() function (introduced in 185c4b) to avoid accessing a private variable. Also use assertSequenceEqual() directly as we don't really care about the actual type returned, only that it's a sequence. (Bitbake rev: 452289efc6d3608ceca321286fd5bf417a6e1bed) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: parse: add support for flagged fragment variablesAlexander Kanavin2024-12-132-3/+10
| | | | | | | | | | | | | | | | | It was suggested that using flags in fragment content to mark fragment-specific metadata (such as descriptions and dependencies) is prone to quiet regressions when fragments are renamed or moved, and it clutters the fragment content. With this change allowed fragment metadata variables must be explicitly listed in a variable that is given to addfragment directive, and parser will add the flag containing the fragment name to them. (Bitbake rev: ed9a3ca9426500511feb77f41a146953dbfe9af7) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: add support for 'addfragments' directiveAlexander Kanavin2024-12-082-0/+42
| | | | | | | | | | | | | | | | It takes two parameters: - location prefix for fragments - name of variable that holds the list of enabled fragments, each of them prefixed by layer id Implementation of this directive essentially expands the fragment list obtained from the variable into absolute fragment paths and hands them to the implementation of 'require'. (Bitbake rev: f687746703e7b096c5480668fd4f49bd4951182b) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse/ConfHandler: allow / in variable flagsAlexander Kanavin2024-12-081-1/+1
| | | | | | | | | | | | This is beneficial for config fragments, as their names are specified via flags, and those names can include slashes: BB_CONF_FRAGMENT_SUMMARY[init/systemd] = "This fragment enables systemd as an init manager" (Bitbake rev: 80805988492b35593067230f68782a5687c8f557) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: use enum for cooker state to improve readabilityChris Laplante2024-12-063-34/+32
| | | | | | | | | enum was introduced in Python 3.4 (Bitbake rev: 35b71a94f8757fcca830f972a42edab1dd000c16) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpcserver: fix some warningsChris Laplante2024-12-061-2/+5
| | | | | | | | | | | + Unnecessary parens + Missing imports + Instance attribute defined outside of __init__ (Bitbake rev: 0622b3df9a55ed66009f3bed47fbf52bb2122056) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: some small cleanups while I'm in hereChris Laplante2024-12-061-3/+10
| | | | | | | | | | | | + missing imports + indentation fix + missing call to super __init__ + redundant parenthesis (Bitbake rev: 89613f4ba31377528d1934cc591fe33218bd916a) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: print an error if MACHINE is not setChris Laplante2024-12-061-2/+12
| | | | | | | | | | | | | | When the user forgets to set MACHINE, bitbake just exits without printing anything. This is because BB_CONSOLELOG ends up with an unexpanded '${MACHINE}', which bb.utils.mkdirhier tries to report using bb.fatal. But bb.fatal utilizes the very logging infrastructure that this code was trying to setup. (Bitbake rev: 7d3f3655b2f610f76898c84b8b97ef2e26529c41) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix scenetask processing performance issueRichard Purdie2024-11-291-1/+5
| | | | | | | | | | | Analysis shows that "bitbake core-image-ptest-all" spends a lot of time in scenequeue_updatecounters and much of it is rebuilding a set which doens't change. Reorder the code to avoid that performance glitch. (Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise setscene loop processingRichard Purdie2024-11-291-2/+8
| | | | | | | | | | Rather than looping through things we looped through on the previous execution, start looping where we left off for setscene processing. This gives speed improvements depending on the kind of build being executed. (Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix performance of multiconfigs with large overlapRichard Purdie2024-11-291-2/+4
| | | | | | | | | | | | | | | | | | | There have been complaints about the performance of large multiconfig builds for a while. The key missing data point was that the builds needed to have large overlaps in sstate objects. This can be simulated by building the same things with just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of deferred tasks. The issue is that the expensive checks in the setscene loop were hit every time through runqueue's execute function before the check on deferred tasks. This leads to task execution starvation as that only happens once per iteration. Move the skip check earlier in the function which speeds things up enormously and should improve performance of such builds for users. (Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix unpack of a single zstd filePascal Eberhard2024-11-291-1/+1
| | | | | | | | | | | | | | | | do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when uncompressing a single 'file.zst', zstd command line returns the error 'unexpected end of file', and the input file 'file.zst' becomes empty. The command line is 'zstd --decompress --stdout file.zst > file.zst'. So the issue is that the input and output filenames are the same. Fix the output filename when using zstd for a single file. (Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8) Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Remove custom exception backtrace formattingJoshua Watt2024-11-285-119/+25
| | | | | | | | | | | | | | | | | | | | | | | Removes the code in bitbake to show custom backtrace formatting for exceptions. In particular, the bitbake exception code prints function arguments, which while helpful is a security problem when passwords and other secrets can be passed as function arguments. As it turns out, the handling of the custom serialized exception stack frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass formatted exceptions"), which changed the events to pass a preformatted stacktrack list of strings, but the passing of the serialized data was never removed. Change all the code to use the python traceback API to format exceptions instead of the custom code; conveniently traceback.format_exception() also returns a list of stack trace strings, so it can be used as a drop in replacement for bb.exception.format_exception() (Bitbake rev: 2cda75a185aaf8f657f072dac34f8cef9d75f63a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/build: add a function to list the tasks in a datastoreRoss Burton2024-11-281-0/+6
| | | | | | | | | | | | | | | There's no easy way to list all of the tasks in a recipe, you can either look at __BBTASKS (internal variable, shouldn't be used) or iterate all items in the datastore looking for variables with the 'task' flag set (which is slow). Solve this problem by adding a bb.build.listtasks() function that returns an immutable copy of the __BBTASSK variable. (Bitbake rev: 185c4b803962b20ba65a7d885dfe1a14e68736ef) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Increase timeout to 100s from 30sRichard Purdie2024-11-211-2/+2
| | | | | | | | | | Testing shows the worst case CDN response time can be up to 100s. The wget fetcher is used for accessing sstate from the CDN so increase our timeouts there to match our worst case repsonse times. (Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/cooker: Increase default pipe sizeRichard Purdie2024-11-211-1/+1
| | | | | | | | | | | | | | | | The default pipe size is 64kb on builds, which can be inefficient for larger log files from workers. Increase the pipe size to 512kb since build systems have decent amounts of memory and this is a more efficient way of batching the data. Tweak the default read sizes to match the pipe size for efficiency. Since the contstant is only present in python 3.10 onwards, add some compatibility code. (Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Drop support for BB_DANGLINGAPPENDS_WARNONLYRichard Purdie2024-11-211-7/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | DANGLINGAPPENDS_WARNONLY is a very poorly designed interface and is commonly abused. The challenge is that once it is set, by any layer, it applies everywhere. Some layers rely on this to get notification they need to update bbappend files and having the behaviour change from inclusion of an antisocial layer is not good. In addition, showing warnings as an accepted thing on the console devalues them and trains the user to ignore them. I want to steer us away from this mindset. We could extend the functionality and make it apply only to certain layers, or only to certain appends but then we've basically re-invented BBMASK. Given all the above, we should drop support for BB_DANGLINGAPPENDS_WARNONLY and direct anyone with issues to BBMASK instead. https://lists.openembedded.org/g/openembedded-architecture/message/2029 [YOCTO #14870] (Bitbake rev: fca9c9e3cb6f8e9f99bf51dc5e8a8d83f4c84c69) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: add new "finalizeData" APIYoann Congal2024-11-182-0/+14
| | | | | | | | | | | | Create a new tinfoil API "finalizeData" which does key expantion and runs the anonymous functions to allow bitbake-getvar to have the same output as "bitbake -e". (Bitbake rev: 63db64ca98564420a378f14c337074ae06da8c63) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: fix needconfig for revalidateCachesYoann Congal2024-11-071-1/+1
| | | | | | | | | | Fixes what looked like a type since parseConfiguration.needconfig is already set just after parseConfiguration() definition. (Bitbake rev: d371df029ece3b9e600a89d08337c437a8ddbf63) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Decrease idle/main loop frequencyRichard Purdie2024-11-071-2/+2
| | | | | | | | | | The idle and main loops have socket select calls to know when to execute. This means we can increase the normal timeout frequency since this is just a fall back and have some small efficiency gains. (Bitbake rev: 8d8e17af8619c976819170c9d5d9a686a666c317) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Don't send heartbeats when no idle functionsRichard Purdie2024-11-071-1/+1
| | | | | | | | | If there are no idle functions present, don't sent heartbeat events. These are only meant to happen while builds are active. (Bitbake rev: 9a2d5e63b07c3912838781776c61f0f1ac9640e1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Merge a function to simplfy codeRichard Purdie2024-11-071-26/+16
| | | | | | | | | Keeping this code separate just makes the code harder to understand, merge them. (Bitbake rev: e5ac26a0e1779df1da3277bf48899c8f7642f1f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid dumpsigs idle loop blockingRichard Purdie2024-11-071-29/+44
| | | | | | | | | | | | | | | | | | | | | We're seeing some failures on hosts where slow "idle" loop iterations are causing bitbake server timeouts. These seem to happen particularly in the dump_signatures() function within runqueue. That isn't entirely surprising since it creates a pool of threads to execute work an at best can take around 10s to execture and return control backto the main loop. On a slow system, it is understandable this can take longer, particularly as these functions are creating large chunks of IO. Since the work is being done in threads, we can launch them, return to idle and check on the results periodically as they complete. This should hopefully address some of the remaining timeout issues we see on the autobuilder in oe-selftest sstate tests. (Bitbake rev: e66f1b643b4b77404ba31f2704cda5af9bf00a57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/compress: use lz4 instead of lz4cJustin Bronder2024-11-052-4/+4
| | | | | | | | | | | | | | | | | | | lz4c has been considered deprecated by upstream since at least 2018 [1] and has been disabled by default recently [2]. openSUSE Tumbleweed is no longer packaging the deprecated version and others will likely follow. Luckily, we're not using any legacy-specific options and, going back as far as Ubuntu 16.04/Fedora 25, both lz4 and lz4cli are installed as part of the same package 1. https://github.com/lz4/lz4/pull/553 2. https://github.com/lz4/lz4/pull/1479 (Bitbake rev: 907472034b344e4eb73cfd43059a413469f52e1c) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Sort pn-buildlistRobert Yang2024-11-051-4/+5
| | | | | | | | | So that we can compare the different pn-buildlist easily. (Bitbake rev: 529043117a7c62feb45bc891658a412cc8dd7e3f) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to ↵Richard Purdie2024-10-291-1/+1
| | | | | | | | | | | | decouple from gnome gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 91e268b11ed683bd197026f9b36001f6d54ee05c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome ↵Richard Purdie2024-10-281-1/+1
| | | | | | | | | | | | gitlab GNOME gitlab has occasional downtime which impacts bitbake-selftest and causes autobuilder failures. Switch to our own mirror for test purposes to avoid those issues. (Bitbake rev: 008808755ed6cfeb6c41273e69ce718f0833c26c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git: Clean broken symlinkRobert Yang2024-10-281-1/+1
| | | | | | | | | | | The file may be a symlink when mirror from local disk, and the symlink will be broken when the linked file is removed, use os.path.islink() to check and remove the broken symlink. (Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: git: Clean shallow mirror tarballRobert Yang2024-10-281-0/+5
| | | | | | | | | | | | | | | Fixed: BB_GIT_SHALLOW = "1" BB_GENERATE_SHALLOW_TARBALLS = "1" $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without this fix. (Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add clean functionRobert Yang2024-10-281-0/+13
| | | | | | | | | | | | Fixed: $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix. (Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-10-281-3/+5
| | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-10-281-28/+14
| | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()Alexander Kanavin2024-10-221-6/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This eliminates the last usage of 'fake mozilla' in bitbake, and it's then truthful everywhere about presenting itself, or wget (when that is used). I understand this will make people nervous so I want to provide an extended decription. 1. How was this tested? - bitbake-selftest -k FetchCheckStatusTest (tests a few hardcoded URIs, all passed) - bitbake -k -c checkuri world (runs checkstatus() over all recipes in oe-core, and all passed again - this hopefully goes a long way to reassure everyone that hosts around the world and various CDNs typically do not have a problem with user-agent strings they haven't seen before or bitbake user-agent specifically) 2. What about that removed cloudflare comment? I digged into git history, and I think it is not fully accurate. First, 'fake mozilla' agent is used only for checkstatus() - in actual fetching with wget it is not. And that has not been a problem for anyone. Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here: https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903 At that point it did not have to be specifically 'mozilla', the commit message indicates that any User-Agent would have been ok. Mozilla was simply copied from upstream version check for convenience. Later on, the string was updated to a more recent Mozilla: https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4 The claim in the added comment become something else: that User-Agent *must* a browser, without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok. 3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent? Please see point one. It's not impossible but I think it's highly unlikely. I do think we should rather tell servers the truth, and learn where the actual issues are. Then we can consider options - whether that would be pretending to be wget, or allowing user-agent to be configured. We should also add such servers to bitbake-selftest so we know what they are. (Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: use default wget user-agent in upstream version checksAlexander Kanavin2024-10-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has been set to various mozilla versions from this point onwards without explanation or specific examples: https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6 However, sourceforge has recently banned the current 'fake mozilla' version we've been supplying, resulting in broken version checks: alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk ... --2024-10-12 11:53:23-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. HTTP request sent, awaiting response... 403 Forbidden 2024-10-12 11:53:24 ERROR 403: Forbidden. gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin <alex.kanavin@gmail.com> On the other hand, using wget's default user-agent works just fine: --2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] (Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add GitHub Codespaces supportOliver Kästner2024-10-141-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker containers to work on GitHub repositories in VS Code within the browser or using a locally running instance of VS Code [1]. Users can define access permissions for Codespaces containers, allowing them to clone or work with other private repositories [2]. To do this, a git credential helper is injected into the container at /.codespaces/bin/gitcredential_github.sh, which contains the following lines: #!/bin/sh echo protocol=https echo host=github.com echo path= echo username=PersonalAccessToken echo password=$GITHUB_TOKEN The `GITHUB_TOKEN` environment variable is automatically set when the Codespaces container is created. If authorized by the user, it has the specified permissions. It is thus required to be able to fetch private git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS` to make it available for invocations of git. [1] https://docs.github.com/en/codespaces/overview [2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces (Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f) Signed-off-by: Oliver Kästner <git@oliver-kaestner.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Improve multiple lock handlingRichard Purdie2024-10-111-0/+1
| | | | | | | | | | We should release locks in the opposite order to the order we locked them in to avoid potentially problematic behaviour. This hasn't been observed with our current usage and is just a preventative improvement. (Bitbake rev: 4fbef87adaca2df8c1e877aec5588c2e5466f252) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: exceptions: Add hack to workaround python 3.13 failuresRichard Purdie2024-10-091-1/+5
| | | | | | (Bitbake rev: 95a48f6e4b2f60b7b477bac7db5bde10e10c5a05) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/exceptions: Drop obsolete 2.6 codeRichard Purdie2024-10-091-7/+1
| | | | | | | | | In python 3 we should always have named tuples in this context. Drop the obsolete exception handling. (Bitbake rev: c11c9f18dad1cf4cb21e3bd873338ecc95652062) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Remove itRichard Purdie2024-10-092-400/+0
| | | | | | | | | It was never a great solution to persisting data and there are much better ones now. The last user has been replaced so drop the code and tests. (Bitbake rev: 681a7516e9f7027e0be6f489c54a7a5e19fa9f06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: checksum/fetch2: Switch from persist_data to a standard cache fileRichard Purdie2024-10-094-15/+58
| | | | | | | | | | | | | | | The sqlite connection handling is causing problems with python 3.13. The connection can be closed at gc time which causing warnings and those can appear at 'random' points and break output, causing weird failures in different tinfoil tools and other tests. Using sqlite as an IPC was never a great idea so drop that usage entirely and just use the standard cache mechanism we already have for other situations. (Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid deprecation warningRichard Purdie2024-10-091-1/+1
| | | | | | | | | | | >From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated. In future Python versions they will be keyword-only parameters." Avoid the warning. (Bitbake rev: 016d872dc3c36792af5c84da1d35a5c86996d795) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: COW: Fix hardcoded magic numbers and work with python 3.13Richard Purdie2024-10-091-4/+6
| | | | | | | | | The COW tests started failing on python 3.13. Looks like it is time to fix the FIXME and drop the magic numbers! (Bitbake rev: 2e6608cec508b3b9bab3530f83e70665ff638182) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-10-011-2/+1
| | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 9ef4f2a30127455562e38e869793a723eed6c07c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>