summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: git: Clean shallow mirror tarballRobert Yang2024-10-281-0/+5
| | | | | | | | | | | | | | | Fixed: BB_GIT_SHALLOW = "1" BB_GENERATE_SHALLOW_TARBALLS = "1" $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without this fix. (Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add clean functionRobert Yang2024-10-281-0/+13
| | | | | | | | | | | | Fixed: $ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix. (Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Remove downloads/tmpdir when failedRobert Yang2024-10-281-3/+5
| | | | | | | | | The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix. (Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add call_process_submodules() to remove duplicated codeRobert Yang2024-10-281-28/+14
| | | | | | | | | There are 14 lines can be removed, and can make it easy to maintain. (Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()Alexander Kanavin2024-10-221-6/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This eliminates the last usage of 'fake mozilla' in bitbake, and it's then truthful everywhere about presenting itself, or wget (when that is used). I understand this will make people nervous so I want to provide an extended decription. 1. How was this tested? - bitbake-selftest -k FetchCheckStatusTest (tests a few hardcoded URIs, all passed) - bitbake -k -c checkuri world (runs checkstatus() over all recipes in oe-core, and all passed again - this hopefully goes a long way to reassure everyone that hosts around the world and various CDNs typically do not have a problem with user-agent strings they haven't seen before or bitbake user-agent specifically) 2. What about that removed cloudflare comment? I digged into git history, and I think it is not fully accurate. First, 'fake mozilla' agent is used only for checkstatus() - in actual fetching with wget it is not. And that has not been a problem for anyone. Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here: https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903 At that point it did not have to be specifically 'mozilla', the commit message indicates that any User-Agent would have been ok. Mozilla was simply copied from upstream version check for convenience. Later on, the string was updated to a more recent Mozilla: https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4 The claim in the added comment become something else: that User-Agent *must* a browser, without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok. 3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent? Please see point one. It's not impossible but I think it's highly unlikely. I do think we should rather tell servers the truth, and learn where the actual issues are. Then we can consider options - whether that would be pretending to be wget, or allowing user-agent to be configured. We should also add such servers to bitbake-selftest so we know what they are. (Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: use default wget user-agent in upstream version checksAlexander Kanavin2024-10-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has been set to various mozilla versions from this point onwards without explanation or specific examples: https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6 However, sourceforge has recently banned the current 'fake mozilla' version we've been supplying, resulting in broken version checks: alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk ... --2024-10-12 11:53:23-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. HTTP request sent, awaiting response... 403 Forbidden 2024-10-12 11:53:24 ERROR 403: Forbidden. gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin <alex.kanavin@gmail.com> On the other hand, using wget's default user-agent works just fine: --2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/ Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] (Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add GitHub Codespaces supportOliver Kästner2024-10-141-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker containers to work on GitHub repositories in VS Code within the browser or using a locally running instance of VS Code [1]. Users can define access permissions for Codespaces containers, allowing them to clone or work with other private repositories [2]. To do this, a git credential helper is injected into the container at /.codespaces/bin/gitcredential_github.sh, which contains the following lines: #!/bin/sh echo protocol=https echo host=github.com echo path= echo username=PersonalAccessToken echo password=$GITHUB_TOKEN The `GITHUB_TOKEN` environment variable is automatically set when the Codespaces container is created. If authorized by the user, it has the specified permissions. It is thus required to be able to fetch private git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS` to make it available for invocations of git. [1] https://docs.github.com/en/codespaces/overview [2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces (Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f) Signed-off-by: Oliver Kästner <git@oliver-kaestner.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: checksum/fetch2: Switch from persist_data to a standard cache fileRichard Purdie2024-10-091-13/+20
| | | | | | | | | | | | | | | The sqlite connection handling is causing problems with python 3.13. The connection can be closed at gc time which causing warnings and those can appear at 'random' points and break output, causing weird failures in different tinfoil tools and other tests. Using sqlite as an IPC was never a great idea so drop that usage entirely and just use the standard cache mechanism we already have for other situations. (Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid deprecation warningRichard Purdie2024-10-091-1/+1
| | | | | | | | | | | >From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated. In future Python versions they will be keyword-only parameters." Avoid the warning. (Bitbake rev: 016d872dc3c36792af5c84da1d35a5c86996d795) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use quote from shlex, not pipesRichard Purdie2024-10-011-2/+1
| | | | | | | | | | | The pipes module is removed in python 3.13. It was already using the quote function from shlex so use that directly instead. The module already imports shlex too so it is an easy substitution. (Bitbake rev: 9ef4f2a30127455562e38e869793a723eed6c07c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: don't try to preserve all attributes when unpacking filesRoss Burton2024-09-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When copying files as part of the unpack we currently use cp -p, which is a shortcut for --preserve=mode,ownership,timestamps. We do want to preserve timestamps, because some fetchers set these explicitly. We don't care about ownership. If the files are owned by us then they ill remain owned by us, and if they're not then the attempt to change ownership will be silently ignored. In a shared DL_DIR where files have group ownership this group access isn't relevant in the single-user build tree. We do want to preserve executable bits in the mode, but cp always does this. The difference between --preserve=mode and no --preserve is that the mode isn't preserved exactly (no sticky bits, no suid, umask is applied) but this also isn't a relevant difference in a build tree. Also expand the arguments to be clearer about what options are being passed. The impetus for this is that coreutils 9.4 includes a change in gnulib[1] and will now try to preserve permission-based xattrs if asked to preserve the mode. This can result in cp failing when copying a file from a NFSv4 server with ACLs stored in xattrs to a non-NFS directory where those xattrs cannot be written: cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported The error comes from the kernel refusing to write a system.nfs4_acl xattr to a file on ext4. This situation doesn't appear on all systems with coreutils 9.4, at the time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This is because /etc/xattr.conf is used to determine which xattrs describe permissions, and Fedora 40 has removed the NFSv4 attributes[2]. Also, use long-form options to make the cp command clearer. [1] https://github.com/coreutils/gnulib/commit/eb6a8a4dfb [2] https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch [ YOCTO #15596 ] (Bitbake rev: 2f35dac0c821ab231459922ed98e1b2cc599ca9a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gomod: Support URIs with only a hostnamePeter Kjellerstedt2024-09-171-2/+6
| | | | | | | | | | | | | | | | When calculating the module name for a gomod URI with only a hostname, e.g.: gomod://go.opencensus.io;version=v0.24.0;sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c the non-existing path would actually be treated as "/", which resulted in a trailing slash being added to the module name preventing the unpack method from correctly locating the go.mod file. (Bitbake rev: f0e02e1de4d649e647e4ab61341042dd38d0eeb0) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "fetch2/gitsm: use configparser to parse .gitmodules"Martin Jansa2024-09-161-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit a4f42e396e2942fde94b8b4944487c1c45f7a295. Unfortunately configparser is a bit more strict and fails to parse e.g.: https://github.com/espressif/esp-idf/blob/e104dd7f27d2e73ab0e9b614dd7b9295099069bf/.gitmodules [submodule "components/bt/controller/lib_esp32"] path = components/bt/controller/lib_esp32 url = ../../espressif/esp32-bt-lib.git The url is using 8 spaces while path 1 tab causing: Exception: configparser.NoOptionError: No option 'url' in section: 'submodule "components/bt/controller/lib_esp32"' It was fixed in: https://github.com/espressif/esp-idf/commit/62ca8e2fb478a1cdc0e47003025265cd0d840395#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584 but mcuboot is using a bit older esp-idf revision in: https://github.com/ATmobica/mcuboot/blame/main/.gitmodules and mcuboot is then used as submodule in: https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules so it might take a while for the fix to be propagated everywhere. Not sure how common these issues are, but configparser parses "url = ../../espressif/esp32-bt-lib.git" as 2nd line of "path" value (because it's indented differently) while git submodule and old gitsm implementation parses it as separate path and url keys. (Bitbake rev: d9eb2650256292a524a0c0bb1c0562d87bedcbc4) Signed-off-by: Martin Jansa <martin.jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: use configparser to parse .gitmodulesRoss Burton2024-09-121-11/+13
| | | | | | | | | | .gitmodules is basically ini-style, so use configparser instead of manually parsing by hand. (Bitbake rev: a4f42e396e2942fde94b8b4944487c1c45f7a295) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add gomodgit fetcherChristian Lindeberg2024-09-102-2/+139
| | | | | | | | | | | | | | | Add a go module fetcher for downloading module dependencies to the module cache directly from a git repository. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..." (Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c) Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add gomod fetcherChristian Lindeberg2024-09-102-1/+131
| | | | | | | | | | | | | | | Add a go module fetcher for downloading module dependencies to the module cache from a module proxy. The fetcher can be used with the go-mod class in OE-Core. A module dependency can be specified with: SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..." (Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67) Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: allow the '@' character in package namesLeonard Göhrs2024-08-251-3/+4
| | | | | | | | | | | | | | | | | | | The '@types/ramda' [1] npm package has recently gained a dependency on the 'types-ramda' [2] npm package. Both have the same version number. The name mangling results in the tarballs of both packages sharing the same name, but different contents. Fix that by accepting '@' as valid character in the package name, resulting in one package named @types-ramda and one called types-ramda. [1]: https://www.npmjs.com/package/@types/ramda [2]: https://www.npmjs.com/package/types-ramda (Bitbake rev: 7c9573cb6ea2081bc585eb65267f3124fd4d7e43) Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gcp.py: remove slow calls to gsutil statEtienne Cordonnier2024-08-251-7/+7
| | | | | | | | | | | | | | | The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however: - in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False and does not throw an exception in case the file does not exist. - Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat. - I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable. - add a try/except block in download() instead of the extra call to gsutil [RP: Tweak to avoid import until needed so google module isn't required for everyone] (Bitbake rev: dd120f630e9ddadad95fe83728418335a14d3c3b) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: allow packages not declaring a registry versionEnguerrand de Ribaucourt2024-08-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We fetch npm dependencies from the npm-shrinkwrap.json file. They can point to a package on the NPM registry with a version field, or to a git/http/file URL with the resolved field. Such packages are allowed not to declare a registry version field because they may not have been published to the NPM registry. The previous implementation refuses to fetch such packages and throws an error. The resolved field contains the exact source, including the revision, wich we can use to pass as SRC_URI to the git/http/file fetcher. The integrity field is also mandatory for HTTP tarballs which will ensure reproducibility. So even if the version field is not present, we are still fetching a precise revision of the package. Another commit published along this stack is also required in the npm class to support these packages. v5: - improve commit message v3: - Split bitbake npmsw.py modification in another commit Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> (Bitbake rev: 209982b5a3efc8081e65b4326bf9b64eef7f0ba0) Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npmsw: fix fetching git revisions not on masterEnguerrand de Ribaucourt2024-08-231-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The NPM package.json documentation[1] states that git URLs may contain a commit-ish suffix to specify a specific revision. When running `npm install`, this revision will be looked for on any branch of the repository. The bitbake implementation however translates the URL stored in package.json into a git URL to be fetch by the bitbake git fetcher. The bitbake fetcher git.py, enforces the branch to be master by default. If the revision specified in the package.json is not on the master branch, the fetch will fail while the package.json is valid. To fix this, append the ";nobranch=1" suffix to the revision in the git URL to be fetched. This will make the bitbake git fetcher ignore the branch and respect the behavior of `npm install``. This can be tested with the following command: $ devtool add --npm-dev https://github.com/seapath/cockpit-cluster-dashboard.git -B version Which points to a project which has a package.json with a git URL: ```json "devDependencies": { "cockpit-repo": "git+https://github.com/cockpit-project/cockpit.git#d34cabacb8e5e1e028c7eea3d6e3b606d862b8ac" } ``` In this repo, the specified revision is on the "main" branch, which would fail without this fix. [1] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#git-urls-as-dependencies Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> (Bitbake rev: 37a35adf7882f231c13643dbf9168497c6a242a1) Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Enforce default remote name to "origin"Yuri D'Elia2024-07-131-1/+1
| | | | | | | | | | | | Enforce the default remote name to "origin", as assumed in numerous places. This prevents build failures in case the system/user configuration sets this to a different value. (Bitbake rev: 1d7360031164f04887c792fb0b2dd86c6ccfcc23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local()Robert Yang2024-07-131-26/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch can make the following settings much more faster: BB_GIT_SHALLOW = "1" BB_GENERATE_MIRROR_TARBALLS = "1" * The previous implementation was: - Make a full clone for the repo from local ud.clonedir - Use git-make-shallow to remove unneeded revs It was very slow for recipes which have a lot of SRC_URIs, for example vulkan-samples and docker-compose, the docker-compose can't be done after 5 hours. $ bitbake vulkan-samples -cfetch Before: 12 minutes Now: 2 minutes $ bitbake docker-compose -cfetch Before: More than 300 minutes Now: 15 minutes * The patch uses git shallow fetch to fetch the repo from local ud.clonedir: - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev Then the git repo will be shallow, and git-make-shallow is not needed any more. And git shallow fetch will download less commits than before since it doesn't need "rev^" to parse the dependencies, the previous code always need 'rev^'. (Bitbake rev: a5a569c075224fe41707cfa9123c442d1fda2fbf) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: checkstatus: drop shared connecton when catch Timeout errory75zhang2024-07-041-1/+6
| | | | | | | | | | | | | | | | * to avoid wrong http response in checkstatus function: in wget checkstatus() we are using 'HTTPConnectionCache' to share connections 1. state_file1(exists on http server) use shared connection <shared1> to send request 2. http_server recieved request of state_file1, but delayed by some reason to sent respone 3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1> 4. state_file2(not exists on http server) get shared connection <shared1> and send request 5. http_server finally send 200 response for state_file1 6. state_file2 recived 200 response and thought it was exists on http_server (Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd) Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/clearcase: remove True option to getVar calls in clearcase modulePeter Marko2024-07-021-2/+2
| | | | | | | | | | | | | Layer cleanup similar to https://git.openembedded.org/openembedded-core/commit/?id=26c74fd10614582e177437608908eb43688ab510 It was probably not found before beacause of the extra "d" parameter. That seem to be a bug as getVar does not support that. (Bitbake rev: 720189b810995c5737853458b7eb3779ca0df37e) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibilityRudolf J Streif2024-06-071-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some distributions (namely Fedora Core 40) have started replacing wget with wget2. There are some changes to wget2 that make it incompatible with wget: 1. ftp/ftps is not supported anymore 2. progress 'dot' is not yet supported 3. Relative paths in -P and -O are not correctly dealt with Item 1: Is already dealt with since Scarthgap by only adding the option --passive-ftp when the URL specifies ftp/sftp. While that won't help if ftp/sftp is actually required it at least does not break http/https downloads. Item 2: While not supported it at least does not break the operation. Item 3: If there are relative path components in -P or -O then wget2 only deals with them correctly if there is one, and only one, relative path component at the beginning of the path: -P ./downloads works -P ../downloads works -P ../../downloads does not work -P ./../downloads does not work -P /home/user/downloads/../downloads does not work In cases where there are more than one relative path component at the beginning of the path and/or one or more reltaive path component somewhere in the middle or end of the path, wget2 aborts with the message Internal error: Unexpected relative path: '<path>') Such can happen if DL_DIR includes relative path components e.g. DL_DIR = "${TOPDIR}/../../downloads". This patch canonicalizes DL_DIR before it is passed to wget. (Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9) Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix failure path for files that are empty or don't existRichard Purdie2024-05-311-9/+9
| | | | | | | | | | | | | When we intercepted the file download to a temp file, we broke the exist/size checks which need to happen before the rename. Correct the ordering. For some reason, python 3.12 exposes this problem in the selftests differently to previous versions. (Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/npmsw: The fetcher shouldn't have any knowledge of SRichard Purdie2024-05-231-1/+1
| | | | | | | | | | | | | I don't know why there is hardcoded knowledge of S in the fetcher but there shouldn't be and the OE unpack changes highlight this doing things it shouldn't. Drop the S reference and use rootdir which is the only place it should be touching. (Bitbake rev: 84f102954e10a3390fca9c26d5c3c639e952a2c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix mirroring issue with svnKari Sivonen2024-05-081-0/+3
| | | | | | | | | | | | Add return false to supports_checksum for svn fetcher which fhis fixes MIRROR usage for svn uris. Also add a testcase. [YOCTO #15473] (Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f) Signed-off-by: Kari Sivonen <kari.sivonen@live.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/crate: add upstream latest version check functionAlexander Kanavin2024-05-081-0/+9
| | | | | | | | | | | This is actually rather easy: crate web API provides a json with all the versions, for example: https://crates.io/api/v1/crates/cargo-c/versions (Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gcp: Add missing runfetchcmd importSven Schwermer2024-04-201-0/+1
| | | | | | | | | This adds the missing import. This bug was introduced with 1ab1d36c. (Bitbake rev: 97ffe14311407f6e705ec24b70870ab32f0637b9) Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftpsRob Woolley2024-04-111-1/+4
| | | | | | | | | | | | | | | Fedora 40 introduces wget2 as a drop-in replacement for wget. This rewrite does not currently have support for FTP. This causes the wget fetcher to fail complaining about an unrecognized option. Making --passive-ftp conditional based on the protocol used in the SRC_URI limits the scope of the problem. It also gives us an opportunity to build the older wget as a host tool. (Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a) Signed-off-by: Rob Woolley <rob.woolley@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Install Git LFS in local repository configDerek Erdmann2024-03-221-1/+1
| | | | | | | | | | | | | | | | | Git uses a lock file to prevent concurrent modifications to the global config, so if unpack tasks for different recipes try to run "git lfs install" simultaneously the operation can fail: error: could not lock config file /home/build/.gitconfig: File exists exit status 255 Run `git lfs install --force` to reset Git configuration. Adding "--local" sets the smudge and clean filters in the local repository's config instead of modifying the user's global config. (Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7) Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix misleading "no output" msgViswanath Kraleti2024-03-221-1/+4
| | | | | | | | | | | When a command is run with a non-null log, errors are only output to the log and are not returned in the exception. In that case direct users to that logfile instead of telling the command had no output. (Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a) Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle URIs with single-valued query parametersRoss Burton2024-03-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst typically the URI query is a list of key-value pairs, that's not actually required by the URI specification. For example: http://example.com/foo?bar is a valid query, but this will result in the fetcher raising an exception: File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__ self.query = self._param_str_split(urlp.query, "&") File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]: ValueError: not enough values to unpack (expected 2, got 1) In this case the query is just "bar", but the fetcher is trying to split it into a key-value pair. The URI object exposes the parsed query explicitly as a dictionary of key-value pairs, so we have to be a little creative here: if a value is None then it isn't a key-value pair, but a bare key. Fix this by handling elements without the deliminator in _param_str_split() (by assigning the value to None), and handle a None value when formatting the query in _param_str_join(). (Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2/git: Escape parentheses in git src namePhilippe Rivest2024-03-221-1/+1
| | | | | | | | | | | | | | | | | | FIXES [YOCTO #15404] When using git fetcher on a repo with parentheses in its URL, the invocation of the git clone command will fail. The clone directory is not quoted thus the shell will return an error and won't execute the command. (Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5) Cc: Philippe Rivest <privest@genetec.com> (Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6) Signed-off-by: Philippe Rivest <technophil98@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Avoid clean upon failureRichard Purdie2024-03-032-2/+12
| | | | | | | | | | | | | | Currently when git fetches fail, it destroys all the existing local clone data. For large repositories this can introduce long build delays when for example, you just typo'd the git revision hash. The git fetcher should be able to recover most directories so when the fetch is for a git repo, avoid removing things unless clean is explicitly called (e.g. a -c cleanall task). (Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure that git LFS objects are availablePhilip Lorenz2024-02-231-2/+43
| | | | | | | | | | | | | | | | | | | | | The current implementation only performs a git lfs fetch alongside of a regular git fetch. This causes issues when the downloaded revision is already part of the fetched repository (e.g. because of moving back in history or the updated revision already being part of the repository at the time of the initial clone). Fix this by explicitly checking whether the required LFS objects are available in the downloade directory before confirming that a downloaded repository is up-to-date. This issue previously went unnoticed as git lfs would silently fetch the missing objects during the `unpack` task. With network isolation turned on, this no longer works, and unpacking fails. (Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae) Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: wget.py: always use the custom user agent"Alexander Kanavin2024-02-201-4/+3
| | | | | | | | | | | | | | | | | | This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc. There's been a report that this breaks downloads from Jfrog Artifactory as self.user_agent is set to 'Mozilla Firefox', and when Artifactory sees that, it sends a response tailored for showing in an interactive browser (which in my opinion it has every right to). If we're using wget, we should say so via wget's default; handling uncooperative servers should be done on per-recipe basis, and ideally with tickets to admins of those servers. (Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Make latest_versionstring extract tags with slashes ↵Peter Kjellerstedt2024-02-191-2/+7
| | | | | | | | | | | | | | correctly Before, everything up to the last slash was removed when extracting the names of the tags. This would lead to that a tag such as "agent/11.0.0" would be incorrectly identified as "11.0.0", which would then be treated as a correct version matching "^(?P<pver>\d+(\.\d+)+)". (Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: A bit of clean-up of latest_versionstring()Peter Kjellerstedt2024-02-191-11/+10
| | | | | | | | | This is mostly preparations for the next commit. (Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: add comment in try_premirrorsChen Qi2024-02-191-0/+5
| | | | | | | | | | The purpose of ensuring 'incremental fetch' is not easy to see from the codes. So add comments to explain this. (Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: fix a corner case in try_premirrorChen Qi2024-02-191-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For gitsm recipes, it's possible that some URL is used more than once. e.g., A -> B:rev1 (B is a submodule of A) A -> C (C is a submodule of A) C -> B:rev2 (B is a submodule of C) A anc C are both using B as submodules, but on different revs. Now if we have: B:rev1 -> D B:rev2 -> E Then, the mirror will not be fully used. Say we have all repo mirrors for A, B, C, D, E, then in theory it's not necessary to reach out to any network for downloading. But it's not the case. After downloading B(rev1) and its submodule D from mirrors, the fetch process continues to download C, thus B(rev2) and E. Now it finds that B needs an update because its submodule E needs an update. Of course this is true because E is not downloaded yet. Now the problem comes to whether to use mirror or not. The git.py defines try_premirror to return 'False' when the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and try_mirror returns False, resulting in not using mirror and going to upstream directly. We can see that the mirrors are not fully used. This is usually not problem, as the cost is only some network download. But in case the following two settings are there, we get errors. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.allowed.domain" In such case, the gitsm recipe A will fail to fetch. Note that all contents that A needs are in mirrors and now it's failing to fetch. This is unexpected. Note that the different revs of the same repo in gitsm recipe is not the only way to reveal this problem. For example, there might be a recipe call B that uses B:rev3. Check the protobuf and grpc recipes as an example. For now, we can use the following steps to reproduce this issue. To be clear, the grpc recipe in meta-oe is now 1.60.0. 1. Add in local.conf: DL_DIR = "${TOPDIR}/downloads-premirror" bitbake grpc -c fetch 2. Comment out the DL_DIR setting in local.conf and add the following lines: PREMIRRORS:append = " \ git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ " 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'. This command succeeds and this shows that the premirror holds everything we need. 4. Add the following lines and then 'bitbake grpc -c fetch'. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.domain" After step 4, the error message is as below: ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used This patch fixes this problem by handling this corner case, that is, if the URL is not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should try premirrors because trying to reach upstream is destined to fail. (Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git.py: Fetch mirror into HEADPavel Zhukov2024-02-191-1/+5
| | | | | | | | | | | | | | | Fix the issue with using of (pre)mirror in case if clonedir exists but outdated. Previous version of the code fetched new mirror content into FETCH_HEAD instead of branch which caused refetch from the upstream. Add new remote add fetch from it instead so the ref can be found by "_contains_ref" Fixes [Yocto #15369] (Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git2: support git's safe.bareRepositoryAndré Draszik2024-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When git is configured with safe.bareRepository=explicit [1], the bitbake git fetcher fails miserably. LWN has an article about the problem that this configuration option addresses and why it is useful in [2]. It also seems that it is being rolled out in some environments as a default for users. In order to allow having this configuration turned on for a user's environment in general, the fetcher has to be tought to use --git-dir= for all relevent git operations. The alternative, implemented here, is to forcibly turn off that option for all git operations. In the future, we could look into converting these to using the --git-dir= command line argument instead. While at it, fix one open-coded invocation of git that wasn't using ud.basecmd Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1] Link: https://lwn.net/Articles/892755/ [2] (Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97) Signed-off-by: André Draszik <andre.draszik@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget.py: always use the custom user agentThomas Perrot2024-02-101-3/+4
| | | | | | | | | | | | | | | | | | | Add the "--user-agent" paramater in the wget base command to perform all wget commands with this parameter, because a few HTTP servers block requests with the default wget user agent. For example, "hg.openjdk.org" never send a response to requests have been sent with wget: wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2 Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73 Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected. HTTP request sent, awaiting response... (Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c) Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: support temporary AWS credentialsToni Lammi2024-02-091-0/+1
| | | | | | | | | | | | | Support AWS_SESSION_TOKEN which is used in temporary AWS credentials. Fixes [YOCTO #15384]. (Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb) Signed-off-by: Toni Lammi <toni.lammi@kone.com> Reported-by: Toni Lammi <toni.lammi@tl-software.fi> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget/checkstatus(): include the URL in debugging output about ↵Alexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | status check failure Previously the output wasn't useful for finding out what was the actual URL that failed, particularly in heavily multi-threaded invocations: DEBUG: checkstatus() urlopen failed: HTTP Error 404: Not Found With this change, the problem is described specifically: DEBUG: checkstatus() urlopen failed for http://cdn.jsdelivr.net/yocto/sstate/all/universal/4f/91/sstate:gettext-minimal-native:x86_64-linux:0.22.4:r0:x86_64:11:4f91b650ebd7be601cbd0e3a37a8cc6385a3f4ee616f931969b50709ed8bf044_create_spdx.tar.zst: HTTP Error 404: Not Found This will help with CDN cache tests in particular. When some object isn't available, we need to know why: 4xx error, 5xx error, timeout error or any other issue. (Bitbake rev: ecd9b92815563509f55264ed6e7498aee797cedd) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/checkstatus(): do not print the URI twice in FetchError exceptionAlexander Kanavin2024-01-101-1/+1
| | | | | | | | | | | | Previously, there was duplicate clutter in the output, particularly if the URI points to sstate cache items: bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work (Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure GCP fetcher checks if file exists before download.Charlie Johnston2023-12-021-2/+5
| | | | | | | | | | | | | | | | | The GCP fetcher was calling bb.fetch2.check_network_access with "gsutil stat" as the command, but then never actually ran that command to check if the file exists. In cases where the file did not exist in a gs:// premirror, this would lead to an unhandled exception from do_fetch when the GCP python API tried to perform the download. This change resolves that issue by adding a runfetchcmd to call gsutil. (Bitbake rev: 1ab1d36c0af6fc58a974106b61ff4d37da6cb229) Signed-off-by: Charlie Johnston <charlie.johnston@loftorbital.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/aws: forward env-vars used in gitlab-ci K8sFelix Moessbauer2023-11-231-0/+2
| | | | | | | | | | | | | | | | | | | This patch adds the following variables to the allow-list, which are used in the "IAM roles for AWS when using the GitLab chart": - AWS_ROLE_ARN - AWS_WEB_IDENTITY_TOKEN_FILE These variables are set in the CI job environment and are needed to access the sstate cache artifacts in a connected S3 bucket. [1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com> (Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-23 14:27:53 +0000