summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/fetch2
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: fetch2: Add GIT_SSH_COMMAND to the list of exportsPavel Zhukov2022-04-141-0/+1
| | | | | | | | | | GIT_SSH_COMMAND is more convinient to use if arguments have to be passed and the user doesn't want to create a wrapper script around ssh. (Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: canonicalize ids in generated tarballsOlaf Mandel2022-04-141-1/+1
| | | | | | | | | | | | | | Change the owner information in the mirror tarballs generated using BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit 0178ab83, which used the original pokybuild:user information, but failed to clean up the numerical user and group ids. Now set the more canonical values of oe:oe and 0:0. (Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> CC: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: stop generated tarballs from leaking infoOlaf Mandel2022-03-261-1/+4
| | | | | | | | | | | | | | | | | | | | When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs of git repositories, they leaked local information: username, group and time of the last fetch. Remove all these by setting fixed information: * uname = pokybuild * gname = users * mtime = committer time of newest commit in repo The username and group value were taken from the archives available on the downloads.yoctoproject.org mirror. The modification time is chosen so it still retains some relationship to the contents of the archive. (Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: support checkstatusDaniel Wagenknecht2022-03-041-0/+40
| | | | | | | | | | This implements support for sstate mirrors using ssh as transport protocol. (Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: fix path handlingDaniel Wagenknecht2022-03-041-0/+5
| | | | | | | | | | Fix absolute paths and paths containing the ":" character. Both is necessary for supporting sstate mirrors via ssh (not implemented yet). (Bitbake rev: df5505a1ba15524c3a185360d687854300aef342) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: ssh: username and password are optionalDaniel Wagenknecht2022-03-041-1/+1
| | | | | | | | | | Support URLs like ssh://HOST/PATH. They were previously not recognized due to a missing @ sign. (Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc) Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: Replace remaining "blacklist"/"whitelist" usageScott Murray2022-02-211-1/+1
| | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the remaining non-backwards-compatibility related usage of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few places as appropriate. (Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Abstract fetcher environment to a functionRichard Purdie2022-02-202-8/+12
| | | | | | | | | | | The changing of the environment inside the wget fetcher can race if threading is used, such as with sstate in OE-Core. Abstract the function so the environment can be correct before the function is called, removing the race since the enviroment is then no longer changed. (Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: wget: Fix grammar "can happen"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: 52630eefb5174e4ca357ac57085093a7f5767bd8) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Remove duplicate "the"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: 02e29e90eac56e46635e9ae44e5c5da684b70ed2) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix typo "conform" and "processed"Zygmunt Krynicki2022-02-201-2/+2
| | | | | | | (Bitbake rev: e1c13a820f9e3154699ec32d84cce502464e4f6b) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: move loop-invariant load of BB_ORIGENVZygmunt Krynicki2022-02-201-4/+3
| | | | | | | | | | | BB_ORIGENV is used as a fallback environment block. It is repeatedly accessed inside the loop. Since it is a loop invariant, move it out of the loop. (Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/cooker: Fix source revision handling with floating upstreamsRichard Purdie2022-02-172-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | Where a git url uses a tag instead of a full source revision, breakage can currently occur in builds. Issues include: * the revision being looked up in multiple tasks (fetch and unpack) * the risk a different revision may be obtained in those tasks * that some tasks may not be allowed to access the network * that a revision may not be consistent throughout a given build * rerunning a specific task may given inconsistent results To fix this, stop the workers from cleaning out the source revision store. This should only be done in the cooker itself (based on current policy). Also, where the code "sees" an upstream access, mark the recipe as not to be cached. The reparse re-triggers the upstream lookup by the server. Add a test to ensure that if get_srcrev isn't called, the user is told they're using a configuration that is known to break. (Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Put all downloaded files in the npm2 directoryPeter Kjellerstedt2022-02-052-4/+8
| | | | | | | | | | | | | Previously npm files that specify downloadfilename= in the SRC_URI would be downloaded to the root of ${DL_DIR} rather than in the ${DL_DIR}/npm2 directory where all other npm files are downloaded. This should make it simpler when setting up and configuring a premirror with the downloaded npm packages. (Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Correct handling of replacing the basename in URIsPeter Kjellerstedt2022-02-051-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The solution implementated in commit 96c30007 (fetch2: fix downloadfilename issue with premirror) missed two corner cases. The first is if the basename of the original URI also appears somewhere else in the replacement URI, in which case it would also be replaced. The second is if the basename of the original URI partially matches the basename of the replacement URI, in which case the inital part of the basename from the replacement URI would be left behind. The second case caused test_npm_premirrors_with_specified_filename to fail. The solution is to prefix the basename with a slash when matching to avoid partial matches, and only replace the basename at the end of the URI. This also adds two test cases that test for these problems. Before this they would give the following errors: - ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ ++++++++++ - ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ (Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add crate fetcherRichard Purdie2022-01-192-0/+139
| | | | | | | | | This imports the crate fetcher from OE-Core to resolve various module issues and adds some very very basic tests of that new fetcher. (Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: npm: Use temporary file for empty user configStefan Herbrechtsmeier2021-12-141-9/+5
| | | | | | | | | | | | Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG' because npm otherwise failed if configs and npmrc aren't set: double-loading config "/dev/null" as "global", previously loaded as "user" (Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: npm: Quote destdir in run chmod commandStefan Herbrechtsmeier2021-12-141-1/+1
| | | | | | | | | | | Quote destdir in run chmod command to support special characters in package name and to avoid syntax error for packages like '@(._.)/execute'. (Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: do not hardcode tarball compressors in version checkAlexander Kanavin2021-12-081-1/+1
| | | | | | | (Bitbake rev: ba3aa8591327d43935f000c6884637997438ecb2) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: add redirectauth parameterJustin Bronder2021-12-081-1/+11
| | | | | | | | | | | | | | | | | | | | Add a parameter that limits sending Basic authentication in the Authorization header to only the first host and not any that we're redirected to. Ignoring potential security concerns, temporary AWS URLs will reject any request that includes authentication details in both the query parameters (from the redirect) and in the Authorization header. Temporary AWS URLs are now being used for release assets from private Github repositories. According to the previous discussion linked below, they're also in use by bitbucket. See also: https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/ (Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0) Signed-off-by: Justin Bronder <jsbronder@cold-front.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add striplevel support to unpackStefan Herbrechtsmeier2021-11-261-0/+2
| | | | | | | | | | | | | | | | | | | | Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading components (levels) from file names on extraction. For example, if the archive `archive.tar.gz` contains `some/file`, the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1` will extract `some/file` to `other/file`. This is useful to extract archives to a specified directory instead of the original root component of the archive. The feature is required for the npm support. The npm package contents should reside in a subfolder inside a npm archive (usually it is called package/). npm strips one directory layer when installing the package. (Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Unify tar command in unpackStefan Herbrechtsmeier2021-11-261-8/+9
| | | | | | | | | | | The tar command and its arguments are repeated for many archive types in the unpack function. Unify the common parts in a variable to prepare further extension. (Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix url remap issue and add testcaseRichard Purdie2021-11-211-1/+1
| | | | | | | | | | Using "" as a target for .replace() is a really bad idea as it duplicates the replacement for every character in the string. Add a testcase which triggered this and correct the code to return the correct result. (Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Add README on fetcher design constraintsRichard Purdie2021-11-111-0/+57
| | | | | | | | | There have been requests to better document the contraints of fetcher design and operation. This README attempts to start that. (Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/wget: Add timeout for checkstatus calls (30s)Richard Purdie2021-11-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | We had an issue where a webserver serving sstate had filesystem issues so would accept connections but effectively not do anything with them. This causes bitbake to hang whilst processing things like sstate objects inside the checkstatus() calls. It can be replicated by setting up a server like: socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null and pointing SSTATE_MIRRORS in OE at that address. Adding a timeout to the checkstatus calls of 15s means that whilst the system will pause, it will then continue and not hang entirely. Since there isn't a large transfer here, 30s should be a reasonable response time after which we should fall back to building things ourselves. [YOCTO #13716] (Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: Handle mirror user/password replacements correctlyRichard Purdie2021-11-031-0/+4
| | | | | | | | | | | Username or password replacements in URIs were being appended rather than replaced in mirror url remapping. Fix this and add a test case. [YOCTO #13823] (Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-8/+8
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-032-3/+3
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Show warning for invalid github urlsRichard Purdie2021-11-031-0/+1
| | | | | | | | On master, tell the users they need to update their urls for github. (Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/git: Handle github dropping git:// supportRichard Purdie2021-11-031-0/+4
| | | | | | | | | github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Add a warning asking users to set a branch in git urlsRichard Purdie2021-11-031-1/+4
| | | | | | | | | | | | There is much uncertainty around what tools and hosting providers will do about default git branch naming in the future. To help ensure we can handle the various scenarios, we will make branch names required in SRC_URI. To start that process, show users a warning if it isn't set. This may also allow us to change the default at some point in the future. (Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Allow git fetcher to support subdir paramRichard Purdie2021-11-031-8/+19
| | | | | | | | | | | | | | | | | | The git fetcher is odd in that it supports destsuffix as a parameter but not the default documented subdir parameter. destsuffix is more limited as it can't take absolute paths. Rework the code to correctly support subdir. Also cleanup to use the None default .get() values and be a bit more pythonic and use subpath as the variable name for subpath for code clarity. We could consider dropping destsuffix as a parameter as some future point. Also fix the tests not to pass in a subdir parameter which was never used but now causes errors. (Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for github prefix in npm shrinkwrap versionStefan Herbrechtsmeier2021-11-011-0/+2
| | | | | | | (Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for duplicate dependencies without urlStefan Herbrechtsmeier2021-11-011-1/+5
| | | | | | | (Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: fix downloadfilename issue with premirrorChen Qi2021-11-011-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The following commit to fix [Yocto #13039] causes regression of the behavior of PREMIRRORS. "bitbake: fetch2: fix premirror URI when downloadfilename defined" Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb as an example. SRC_URI = "\ http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \ ... " With the above commit, it now tries to fetch 1.4.0.tar.gz instead of freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz. The commit above tries to avoid fetching from invalid url such as: https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz. And its solution is to make basename to be 1.4.0.tar.gz, thus causing the regression. This patch fixes the above regression. For Yocto #13039, it now tries to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz. (Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/perforce: Fix typoRichard Purdie2021-10-261-1/+1
| | | | | | (Bitbake rev: cf421235bd6f5eb12e9193634c0e870ab035b191) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Do not attempt to verify donestamp if download is not completeCaner Altinbasak2021-10-141-1/+3
| | | | | | | | | | | | | | | | | npmsw(and maybe other fetchers) may try to download same files simulatenously. if a premirror is set, download method tries premirror location first. If file is not found, done is set to false. In the meantime, other fetchers may download the files required and verify_donestamp might return true. In this scenerio, fetcher doesn't try to download again and fails. Do not attempt to verify_donestamp if done is not set. By this way, we ensure that fetcher attempts to do another download after premirror failure. (Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8) Signed-off-by: Caner Altinbasak <cal@brightsign.biz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw: Avoid race condition with multiple npm fetchersCaner Altinbasak2021-10-141-0/+4
| | | | | | | | | | | | If multiple npmsw fetchers are trying to download the same npm file, one of them can try to download the file while other is calling verify. npmsw methods gets called without holding the lock, which causes race conditions in fetching and verification etc. Lock the lockfile before calling proxy fetcher methods. (Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a) Signed-off-by: Caner Altinbasak <cal@brightsign.biz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: document checkstatus APIRoss Burton2021-10-141-1/+5
| | | | | | | | | The checkstatus semantics are unconventional, so document them. (Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npmsw: Add support for local tarball and link sourcesStefan Herbrechtsmeier2021-10-081-3/+20
| | | | | | | (Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Create config npmrc in environment instantiationStefan Herbrechtsmeier2021-10-081-15/+27
| | | | | | | | | | | | | | | Create a configuration npmrc per npm environment to avoid repeated creation of the same configuration file. Create the file via python to avoid multiple npm config calls and add the ability to pass a file path instead of a temporary file. Deprecate the npm configs argument of the run function. The configs should be passed to npm environment or as command specific arguments. (Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Support npm archives with missing search directory modeStefan Herbrechtsmeier2021-10-081-0/+2
| | | | | | | | | | Delay directory restore and set execute/search directory mode bits in unpack to support npm archives with a missing search directory mode. (Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/gitsm: remove the 'nugget' SRCREV cachingRoss Burton2021-10-081-18/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The cached revisions which are used to decide if a repository doesn't need to be updated are misleading when used in conjunction with mirror tarballs and can cause partial fetches to happen, resulting in unpack errors as repositories were not fetched. A concrete example: edk2-firmware in meta-arm is at version 202102 (ef91b0). This is built on the autobuilder so the source mirror contains the repository as a mirror tarball. If I build edk2-firmware 202102 the gitsm fetcher will initially download the top-level repository and then iterate into the submodules to also fetch those repositories, including cmocka from cryptomilk.org. edk2-firmware will then unpack and build successfully. I then update edk2-firmware to 202105 (e1999b) and build it. Gitsm.needs_update() starts by calling Git.needs_update() which returns False, as the mirror tarball contains this revision. It then looks at the "nuggets" which are SRCREVs it has fetched before. The mirror tarball itself contains the nugget for e1999b as this has been built on the autobuilder, so needs_update return False, no more fetching is done, and the build proceeds to unpack. However, as part of the 202105 upgrade the URL of the cmocka submodule changed, and this new repository was never fetched. This means that unpack fails as one of the required git repositories isn't available. The nugget codepaths appear to be an attempt at optimising the fetch process, but have demonstratable failure cases. Just removing them entirely solves the edk2-firmware example, and all of the fetcher test cases still pass. (Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: clarify the command-no-found error messageRoss Burton2021-10-081-1/+1
| | | | | | | | | | | | | If runfetchcmd() fails with bb.process.NotFoundError, the message output is simply "Fetch command" which doesn't really explain what the problem is. Add "not found" to clarify what happened. (Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Allow peg-revision functionality to be disabledRichard Purdie2021-09-211-2/+8
| | | | | | | | | | | | Sometimes the peg revision functionality we use in the svn fetcher is not the correct option. Add a parameter nopegrevision which can be added to disable this behaviour. [YOCTO #6258] (Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use os.rename instead of mvRichard Purdie2021-09-211-4/+4
| | | | | | | | | os.rename will overwrite the destination file if present so we can use this instead of the process call overhead. (Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/s3: allow to switch profile from environment variableValentin Danaila2021-09-171-0/+1
| | | | | | | | | | Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles and switch between profiles as export of the AWS_PROFILE environment variable (Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58) Signed-off-by: Valentin Danaila <drlv85@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Avoid races over mirror tarball creationRichard Purdie2021-09-171-2/+19
| | | | | | | | | | | | | | There is a potential race over the mirror tarballs where a partial git repo could be extracted causing fetcher failures if the tarball is being rewritten whilst another build accesses it. Create the mirror tarball atomically to avoid this. [YOCTO #14441] (Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Enable ftpsDaniel Ammann2021-09-161-1/+1
| | | | | | | | | | The fetcher would fail with: Could not find a fetcher which supports the URL: ftps://... (Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add recursion guardRichard Purdie2021-09-111-0/+7
| | | | | | | | | | | Users sometimes put ${S} references in ${SRC_URI} without realising this can be problematic. Improve the error messages if they accidentally do. [YOCTO #11593] (Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-23 17:18:44 +0000