summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: fetch: Add README on fetcher design constraintsRichard Purdie2021-11-111-0/+57
| | | | | | | | | There have been requests to better document the contraints of fetcher design and operation. This README attempts to start that. (Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie2021-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 15s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Handle mirror user/password replacements correctlyRichard Purdie2021-11-031-0/+4
| | | | | | | | | | | Username or password replacements in URIs were being appended rather than replaced in mirror url remapping. Fix this and add a test case. [YOCTO #13823] (Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-8/+8
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-032-3/+3
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Show warning for invalid github urlsRichard Purdie2021-11-031-0/+1
| | | | | | | | On master, tell the users they need to update their urls for github. (Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Handle github dropping git:// supportRichard Purdie2021-11-031-0/+4
| | | | | | | | | github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Add a warning asking users to set a branch in git urlsRichard Purdie2021-11-031-1/+4
| | | | | | | | | | | | There is much uncertainty around what tools and hosting providers will do about default git branch naming in the future. To help ensure we can handle the various scenarios, we will make branch names required in SRC_URI. To start that process, show users a warning if it isn't set. This may also allow us to change the default at some point in the future. (Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Allow git fetcher to support subdir paramRichard Purdie2021-11-031-8/+19
| | | | | | | | | | | | | | | | | | The git fetcher is odd in that it supports destsuffix as a parameter but not the default documented subdir parameter. destsuffix is more limited as it can't take absolute paths. Rework the code to correctly support subdir. Also cleanup to use the None default .get() values and be a bit more pythonic and use subpath as the variable name for subpath for code clarity. We could consider dropping destsuffix as a parameter as some future point. Also fix the tests not to pass in a subdir parameter which was never used but now causes errors. (Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for github prefix in npm shrinkwrap versionStefan Herbrechtsmeier2021-11-011-0/+2
| | | | | | | (Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for duplicate dependencies without urlStefan Herbrechtsmeier2021-11-011-1/+5
| | | | | | | (Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix downloadfilename issue with premirrorChen Qi2021-11-011-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The following commit to fix [Yocto #13039] causes regression of the behavior of PREMIRRORS. "bitbake: fetch2: fix premirror URI when downloadfilename defined" Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb as an example. SRC_URI = "\ http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \ ... " With the above commit, it now tries to fetch 1.4.0.tar.gz instead of freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz. The commit above tries to avoid fetching from invalid url such as: https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz. And its solution is to make basename to be 1.4.0.tar.gz, thus causing the regression. This patch fixes the above regression. For Yocto #13039, it now tries to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz. (Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/perforce: Fix typoRichard Purdie2021-10-261-1/+1
| | | | | | (Bitbake rev: cf421235bd6f5eb12e9193634c0e870ab035b191) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Do not attempt to verify donestamp if download is not completeCaner Altinbasak2021-10-141-1/+3
| | | | | | | | | | | | | | | | | npmsw(and maybe other fetchers) may try to download same files simulatenously. if a premirror is set, download method tries premirror location first. If file is not found, done is set to false. In the meantime, other fetchers may download the files required and verify_donestamp might return true. In this scenerio, fetcher doesn't try to download again and fails. Do not attempt to verify_donestamp if done is not set. By this way, we ensure that fetcher attempts to do another download after premirror failure. (Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8) Signed-off-by: Caner Altinbasak <cal@brightsign.biz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw: Avoid race condition with multiple npm fetchersCaner Altinbasak2021-10-141-0/+4
| | | | | | | | | | | | If multiple npmsw fetchers are trying to download the same npm file, one of them can try to download the file while other is calling verify. npmsw methods gets called without holding the lock, which causes race conditions in fetching and verification etc. Lock the lockfile before calling proxy fetcher methods. (Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a) Signed-off-by: Caner Altinbasak <cal@brightsign.biz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: document checkstatus APIRoss Burton2021-10-141-1/+5
| | | | | | | | | The checkstatus semantics are unconventional, so document them. (Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for local tarball and link sourcesStefan Herbrechtsmeier2021-10-081-3/+20
| | | | | | | (Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Create config npmrc in environment instantiationStefan Herbrechtsmeier2021-10-081-15/+27
| | | | | | | | | | | | | | | Create a configuration npmrc per npm environment to avoid repeated creation of the same configuration file. Create the file via python to avoid multiple npm config calls and add the ability to pass a file path instead of a temporary file. Deprecate the npm configs argument of the run function. The configs should be passed to npm environment or as command specific arguments. (Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Support npm archives with missing search directory modeStefan Herbrechtsmeier2021-10-081-0/+2
| | | | | | | | | | Delay directory restore and set execute/search directory mode bits in unpack to support npm archives with a missing search directory mode. (Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: remove the 'nugget' SRCREV cachingRoss Burton2021-10-081-18/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: clarify the command-no-found error messageRoss Burton2021-10-081-1/+1
| | | | | | | | | | | | | If runfetchcmd() fails with bb.process.NotFoundError, the message output is simply "Fetch command" which doesn't really explain what the problem is. Add "not found" to clarify what happened. (Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Allow peg-revision functionality to be disabledRichard Purdie2021-09-211-2/+8
| | | | | | | | | | | | Sometimes the peg revision functionality we use in the svn fetcher is not the correct option. Add a parameter nopegrevision which can be added to disable this behaviour. [YOCTO #6258] (Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use os.rename instead of mvRichard Purdie2021-09-211-4/+4
| | | | | | | | | os.rename will overwrite the destination file if present so we can use this instead of the process call overhead. (Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/s3: allow to switch profile from environment variableValentin Danaila2021-09-171-0/+1
| | | | | | | | | | Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles and switch between profiles as export of the AWS_PROFILE environment variable (Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58) Signed-off-by: Valentin Danaila <drlv85@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Avoid races over mirror tarball creationRichard Purdie2021-09-171-2/+19
| | | | | | | | | | | | | | There is a potential race over the mirror tarballs where a partial git repo could be extracted causing fetcher failures if the tarball is being rewritten whilst another build accesses it. Create the mirror tarball atomically to avoid this. [YOCTO #14441] (Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Enable ftpsDaniel Ammann2021-09-161-1/+1
| | | | | | | | | | The fetcher would fail with: Could not find a fetcher which supports the URL: ftps://... (Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add recursion guardRichard Purdie2021-09-111-0/+7
| | | | | | | | | | | Users sometimes put ${S} references in ${SRC_URI} without realising this can be problematic. Improve the error messages if they accidentally do. [YOCTO #11593] (Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: fix premirror URI when downloadfilename definedScott Weaver2021-09-071-1/+1
| | | | | | | | | | | | | When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also defined using the same URI, the downloadfilename is appended to the mirror URI and it should not be. [YOCTO #13039] (Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fix 'no_proxy' handlingEnrico Scholz2021-08-231-43/+44
| | | | | | | | | | | | | | | | | | | The urllib.request.ProxyHandler constructor only reads the $http_proxy + $https_proxy environment variables. $no_proxy is evaluated later when the url is opened. It is therefore not sufficient to just construct the proxy handler in the | with bb.utils.environment(**newenv): context, but the 'opener.open(r)' call must also be made there. (Bitbake rev: 076baf4fbd328d247508fd399866a397eb34f67e) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fetch securely by defaultRoss Burton2021-08-121-3/+16
| | | | | | | | | | | | | The days of broken certificates are behind us now, so instead of always passing --no-check-certificate to wget, don't pass it by default and instead only pass it BB_CHECK_SSL_CERTS = "0". [ YOCTO #14108 ] (Bitbake rev: 4104850dd36096a9ff01836c5fca9ac0e452bcf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: ensure all variables are set when calling urllibRoss Burton2021-08-121-13/+34
| | | | | | | | | | | | | | Instead of just exporting the proxy variables when calling into urllib, use bb.utils.environment() to export all of the known variables that are needed for proper connectivity. Specifically, this ensures that SSL_CERT_FILE is set, so that libssl can find the certificates in buildtools environments (Bitbake rev: 116637b0e9aabae7f680b102dbf3577b8a58f049) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: expose environment variable names that need to be exportedRoss Burton2021-08-121-22/+24
| | | | | | | | | | | | | | | There is a list of environment variable names that need to be exported when executing external commands, such as 'http_proxy'. To avoid duplication, make this a top-level variable. Also add SSL_CERT_FILE, which is used by OpenSSL to locate the certificate bundle. This is needed in buildtools environments where the default path isn't valid. (Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/tests/toaster: Override conversion fixupsRichard Purdie2021-08-061-1/+1
| | | | | | | | | Fix some references that missed during the overrides syntax migration or were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch. (Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc/lib: Update to use new override syntax containing colonsRichard Purdie2021-08-021-1/+1
| | | | | | | | | | This runs the overrides conversion script in OE-Core over the bitbake code base including the docs. A handful of things were excluded in toaster and for the Changelog file. (Bitbake rev: 47f8d3b24fd52381bf3b41e2f55a53e57841344c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/s3: allow to use credentials from environment variablesAdam Romanek2021-07-111-1/+4
| | | | | | | | | | | | | | | | | Previously access to AWS S3 was expected to be preconfigured and credentials to be stored in ~/.aws/credentials. With this change one can use Bitbake s3 fetcher without AWS credentials stored permanently as above, just with them exported as the following environment variables: - AWS_ACCESS_KEY_ID, - AWS_SECRET_ACCESS_KEY. - AWS_DEFAULT_REGION. (Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48) Signed-off-by: Adam Romanek <romanek.adam@gmail.com> Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: run gc in foreground to avoid race with tarAdam Romanek2021-07-101-1/+1
| | | | | | | | | | | | | | | | | | | | It looks like git gc can interrupts the package creation when BB_GENERATE_MIRRORS_TARBALL is in use. Log excerpts: tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output: tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. (Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877) Signed-off-by: Adam Romanek <romanek.adam@gmail.com> Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add check for empty SRC_URI hash stringScott Weaver2021-06-061-1/+4
| | | | | | | | | | | | | | | | | | | | | | No error was being reported when the hash string was set to empty. For example: SRC_URI[md5sum] = "" On a related note (not a bug): Because whitespace in the string will result in a checksum mismatch, the error message was updated to make it a little clearer why the error was thrown. For example: SRC_URI[md5sum] = " " or SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1" Now creates a message like this: File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5 checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected [YOCTO #14232] (Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: when checking latest versions, consider all numerical ↵Alexander Kanavin2021-05-141-1/+1
| | | | | | | | | | | | | | directories Previously the regex was maching x.y, but wasn't matching x, which is a problem e.g. here: https://download.gnome.org/sources/epiphany/ (the new gnome version scheme adds 40-series at the end). (Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/svn: Fix parsing revision of SVN repos with redirectsHarald Brinkmann2021-05-061-1/+1
| | | | | | | | | | svn was printing a message when encountering HTTP redirects. This confused the revision parser. (Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146) Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/s3: Add progress handler for S3 cp commandPrzemyslaw Gorszkowski2021-05-011-1/+40
| | | | | | | | Adds progress support for fetchers from S3. (Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: Fix crash when using git LFS and submodulesNiels Avonds2021-04-201-1/+1
| | | | | | | | | | | | | | | | Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] (Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Niels Avonds <niels@codebits.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: add support for disabling shared clones on unpackHenning Schild2021-04-181-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | By default the unpacker will create a "shared" clone when cloning from the DL_DIR to the WORKDIR. This patch introduces an option to control that behaviour. Imagine some recipe steps are executed in a namespace that is different from the one your downloader and unpacker ran in. (chroot) Because a "shared" clone has an absolute reference to its "alternate" you now have to make that "alternate" visible in that new namespace (chroot) at the exact place. With this patch you can unpack "noshared" and get a stand-alone copy. This copy will also work if the "alternate" is not visible or existant. The switch is a global bitbake switch and will affect all git urls. Build systems that need "noshared" most likely need it for everything they do with git. (Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102) Signed-off-by: Henning Schild <henning.schild@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add Azure Storage fetcher implementationAlejandro Hernandez Samaniego2021-03-112-1/+96
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allows bitbake to fetch from an Azure Storage account. The fetcher submodule is compatible with the az:// URI protocol, its functionality is based on bitbakes wget fetcher, superior in performance to using a propietary tool like azcopy which can handle cloud storage account operations with more functionality (that we dont need in a fetcher) but less compatibility. A sample URI uses can be defined in the following way: SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<container>/foo.tar.xz" This fetcher can easily be used with PREMIRRORS and SSTATE_MIRRORS, e.g.: SSTATE_MIRRORS = "file://.* az://<azure-storage-account>.blob.core.windows.net/sstate-cache/PATH;downloadfilename=PATH \n" PREMIRRORS_prepend = "\ git://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ ftp://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ http://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ https://.*/.* az://<azure-storage-account>.blob.core.windows.net/downloads/ \n \ " Can also be used with non-public access Azure Storage accounts/containers via a Shared Access Signature by declaring the AZ_SAS variable which will be automatically used by the fetcher: AZ_SAS="?sv=2000-01-01&ss=...&sig=somesignature" (Bitbake rev: b103b02f2ce2f8f5079f17ec1a854f904c2110a4) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Avoid crashing when connection drops mid checkstatusAlejandro Hernandez Samaniego2021-02-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | If an exception is raised when running host python code, the fetcher immediately crashes, this might be temporary depending on the servers reliability. Catch the exception when the connection was reset and try once again to fetch the data. File: '/usr/lib/python3.8/socket.py', lineno: 669, function: readinto 0665: if self._timeout_occurred: 0666: raise OSError("cannot read from timed out object") 0667: while True: 0668: try: *** 0669: return self._sock.recv_into(b) 0670: except timeout: 0671: self._timeout_occurred = True 0672: raise 0673: except error as e: Exception: ConnectionResetError: [Errno 104] Connection reset by peer (Bitbake rev: d0f5c5905bc664e415a05e3130dfe0ae541d8b3e) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-1012-53/+53
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: download LFS content too during do_fetchMatt Hoosier2021-01-231-3/+41
| | | | | | | | | | | | | | Insert an explicit pass to fetch all blobs needed by Git LFS, during the fetch() function. This avoids the default behavior of Git LFS to wait until 'git checkout' to begin downloading the blobs pointed to by LFS records. Network access is not allowed at that point in the recipe's lifecycle. [YOCTO #14191] (Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c) Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: handle empty elements in _param_str_splitRoss Burton2021-01-231-1/+1
| | | | | | | | | | | | | _param_str_split is used to split ?foo=1;bar=2 into a dictionary, but throws an exception if a lone semicolon is used as the value doesn't split into two items. Fix by checking that the result of the first split has content. (Bitbake rev: 7662f8c8676d87cb318f811423cc02fe8cb146f6) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/perforce: Fix localfile to include ud.moduleRichard Purdie2021-01-231-1/+2
| | | | | | | | | As reported by Katu Txakur, the output depends on the module path set so this needs to be accounted for in localfile. (Bitbake rev: 9861ed37bb1c5d09c3b4852d2a252e3f3c86ab14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Update user-agentChristophe Priouzeau2021-01-231-3/+8
| | | | | | | | | | | | With the usage of enterprise proxy, the user-agent defined are too old and refused by proxy configuration. Updating to something more modern is desirable. (Bitbake rev: 7001fdd7c4dca372cbebd8fd2c0b03c5d43f9400) Signed-off-by: Christophe Priouzeau <christophe.priouzeau@st.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Fix usehead for non-default namesJoey Degges2021-01-081-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The usehead url parameter for git repositories causes bitbake to use whatever commit the repository HEAD is pointing to if the repository happens to have the name 'default'. This is the default name so in many cases it works just fine, but if a different name is specified with the url parameter 'name=newName' then it will fail to parse the recipe with an error along the lines of: ERROR: ExpansionError during parsing /path/to/my/recipe.bb Traceback (most recent call last): File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init: > ud.setup_revisions(d) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions: for name in self.names: > self.revisions[name] = srcrev_internal_helper(self, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'): if srcrev == "AUTOINC": > srcrev = ud.method.latest_revision(ud, d, name) File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'): except KeyError: > revs[key] = rev = self._latest_revision(ud, d, name) return rev File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'): raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \ > (ud.unresolvedrev[name], ud.host+ud.path)) bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo Let's fix this by setting the unresolved rev of _all_ repository names to 'HEAD' when the usehead url parameter is specified. Update the currently failing test, test_local_gitfetch_usehead_withname, to now expect success. This change preserves existing behavior that allows usehead to be overridden by a valid looking revision if one happens to be specified instead of AUTOREV. (Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c) Signed-off-by: Joey Degges <jdegges@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-04 16:56:53 +0000