summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/tests
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: fetch/npmsw: add more short forms for git operationsMark Asselstine2023-02-261-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | >From the npm-install documentation [1] the CLI provides a set of short forms when the install fetches from git. These include "github:" example: npm install github:mygithubuser/myproject "gist:" example: npm install gist:101a11beef "gitlab:" example: npm install gitlab:mygitlabuser/myproject "bitbucket:" example: npm install bitbucket:mybitbucketuser/myproject Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] by Stefan Herbrechtsmeier added support for the "github:" but the others would marked as 'Unsupported dependency'. The other prefixes are added in this commit, along with extending the tests to cover some of these. However, there is one more short form for github which npm-install allows which forgoes the prefix altogether. example: npm install mygithubuser/myproject Unfortunately this format is a bit problematic as it lacks any easily identifiable 'marker' to match against, and it could be either the github short form or install from folder format. Experimentation shows that the folder format requires a leading './' or '/', so we use this to rule out the ambiguity. If this approach to folder and github formats disambiguation is incorrect it won't matter anyways as the folder format is unrecognized by the code as-is and thus with this change or without, things would fail. Since we have to be less strict in the check for git operations we move it to be the last install format which we check, such that the less ambiguous formats can be sorted out first. [1] https://docs.npmjs.com/cli/v9/commands/npm-install [Yocto #14236] (Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add real git lfs tests and decoratorPaulo Neves2023-02-221-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | Added tests that verify that git-lfs works with an actual real git-lfs server. This was not previously the case because the repo in the test was a simulation of git-lfs but not a real git lfs repo. The 2 added tests are almost the same but test that the git lfs file checkout is successfult with or without the lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers 2 different code paths for git lfs. lfs=1, when used on git lfs repositories triggers the git lfs downloading at the fetch bare stage. lfs query parameter unset triggers the git lfs downloading only on checkout as an implicit behavior of git. This leads to possible network access on the unpack stage and outside the DL_DIR. lfs=0 actually disables git-lfs functionality even if supported. (Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: git-lfs restore _find_git_lfsPaulo Neves2023-02-221-16/+24
| | | | | | | | | | Not restoring the mocked _find_git_lfs leads to other tests failing. (Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d) Signed-off-by: Paulo Neves <paulo@myneves.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Add support for new BB_HASH_CODEPARSER_VALS for cache ↵Richard Purdie2023-01-241-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | optimisation Currently the codeparser cache ends up being extended for every parse run since there are values in the functions such as the result of os.getpid() from LOGFIFO in OE-Core. Digging into that issue, there are also lots of similar but different functions being parsed where the change might just be a path to WORKDIR, a change in PN or PV or something like DATE/TIME. There is no reason we have to use these changing values when computing the dependenies of the functions. Even with a small tweak like: BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn" the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run to around 1.3MB and remaining static for oe-core and meta-oe. In my local build, admittedly heavily experimented with, the cache had grown to 120MB. The benefits of doing this are: * faster load time for bitbake since the cache is smaller to read from disk and load into memory * being able to skip saving the cache upon shutdown * lower memory footprint for bitbake * faster codeparser data lookups (since there is less data to search) We only use these special values when passing code fragments to the codeparser to parse so the real variable values should otherwise be used in the hash data. The overall effect of this change, combined with others to avoid saving unchanged cache files can be ~2s on a ~16s parse on my local system and results in a more responsive feeling bitbake. It also allows parsing performance to be investigated more consistently. (Bitbake rev: f24bbaaddb36f479a59a958e7fc90ef454c19473) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add parameter to recipe_uriPavel Zhukov2023-01-201-1/+1
| | | | | | | | | | | | While the parameter is not required it allows testing of possible regression in fetcher code when parameter specified and mirrors are used. (Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/tests: Add test for MercurialPavel Zhukov2023-01-181-0/+44
| | | | | | | | | | | Basic test for mercurial functionality including PREMIRRORONLY with disabled network (Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Always use threadlockRichard Purdie2022-12-311-16/+1
| | | | | | | | | | | | | | | | With the move to a server idle thread, we always need threading. The existing accessor functions could end up turning this off! I was going to hold the lock whilst changing it, check if the value was already set, cache the result and also fix the event code to always release the lock with a try/finally. Instead, disable the existing functions and use a with: block to handle the lock, keeping things much simpler. (Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: builtins fix for 'd' deletionRichard Purdie2022-12-292-11/+12
| | | | | | | | | | | | | | | | | | | I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/data/codeparser: Add dependencies from python module functionsRichard Purdie2022-12-171-7/+7
| | | | | | | | | | | | | | | | | | | | Moving code into python modules is a very effective way to reduce parsing time and overhead in recipes. The downside has always been that any dependency information on which variables those functions access is lost and the hashes can therefore become less reliable. This patch adds parsing of the imported module functions and that dependency information is them injected back into the hash dependency information. Intermodule function references are resolved to the full function call names in our module namespace to ensure interfunction dependencies are correctly handled too. (Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c) (Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop non-multiconfig aware siggen supportRichard Purdie2022-12-111-1/+1
| | | | | | | | | All siggens in common use should now support multiconfig, drop the compatibility code. (Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: correctly match versioned directoriesAlexander Kanavin2022-12-114-0/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). (Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: allow python snippets to include a dictionaryMark Asselstine2022-11-101-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #14917] Attempting to use a dictionary in a python code snippet for variable assignment results in an error. For example attempting something such as IDX = "green" VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}" produces the error expansion of VAL threw ExpansionError: Failure expanding variable VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]} which triggered exception SyntaxError: '{' was never closed (Var <VAL>, line 1) The existing __expand_python_regexp__, "\${@.+?}", will match the first close curly bracket encountered, resulting in incomplete and un-parsable code, and thus produce the error. We can correct this by allowing a single depth of nested curly brackets in __expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will match up to and including the matching close curly bracket to the open, '${@', curly bracket, even if there are one or more singly nested curly brackets present. This change allows the usecase described above to function. This change can't be made on its own though. The old regex would, in an obscure way, handle the case where a python snippet contained an unexpandable variable. Since the unexpandable variable is in curly brackets it would cause incomplete/un-parsable python code and thus remain unparsed. So something like VAL = "${@d.getVar('foo') + ${unsetvar}}" would remain unparsed as the close curly bracket in "${unsetvar}" would match and terminate the snippet prematurely. This quirk resulted in the proper handling of python snippets with unexpanded variables. With the change to __expand_python_regexp__ the full snippet will match and be parsed, but to match the old/correct behavior we would not want to parse it until ${unsetvar} can be expanded. To ensure the old/correct behavior for python snippets with unexpanded variables remains in place we add a check for unexpanded variables in the python snippets before running them. This handling of unparsed variables brings two benefits. The first we now have an explicit check visible to all for unexpanded variables instead of a somewhat hidden behavior. The second is that if there are multiple python snippets the old behavior would run the code for each but a single snippet with unexpanded variables would mean all snippets would remain unparsed, meaning more and repeated processing at a later time. For example: "${@2*2},${@d.getVar('foo') ${unsetvar}}" old behavior would give: "${@2*2},${@d.getVar('foo') ${unsetvar}}" new behavior will give: "4,${@d.getVar('foo') ${unsetvar}}" The old behavior would calculate '2*2' but toss the result when the second snippet would fail to parse resulting in future recalculations (or fetching from cache), while the new behavior avoids this. (Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow handling of a file:// url within a submoduleRichard Purdie2022-10-261-2/+2
| | | | | | | | | CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add a parameter to the commands in the tests to allow this to continue to work. (Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: bb.tests.fetch.URLHandle: add 2 new testsMark Asselstine2022-10-261-0/+2
| | | | | | | | | | | | | | | | | Add a test for special characters in user and password to qualify decodeurl() inspired by a bug report describing that '=' signs in a password was problematic. Add a second test to qualify decodeurl() as related to the change in commit 628c4bf6c89b [fetch2/__init__: handle @ in package names]. Relates to [YOCTO #14476] (Bitbake rev: ee04cf09c7022168c035affa654773652a49793e) Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: Add test for possible gitsm deadlockPavel Zhukov2022-09-011-0/+9
| | | | | | | (Bitbake rev: b0506480baa9bcf3ef645b0aed5a07ad9950245c) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: Add Timeout classPavel Zhukov2022-09-011-0/+19
| | | | | | | | | | | | | The class and exception aim to test rare cases there deadlocks are possible. Can be used in context managers: with Timeout(<value>): do_deadlock() (Bitbake rev: c5fcdd804d422f959a189b270d72123a50e74da6) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler/cooker: Implement recipe and global classesRichard Purdie2022-08-121-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-08-122-0/+4
| | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Drop DL_DIR fallback for local file fetcherRichard Purdie2022-07-151-0/+2
| | | | | | | | | | | | | | | | | | | | | | A long time ago, we made DL_DIR a final fallback for the local fetcher. Since then we added checksum support and task hashes and the world has changed. There were warnings added some time ago if this fallback triggers and it is now time to drop it entirely. The original use case was for sstate however the sstate code now sets FILESPATH correctly so DL_DIR is no longer needed. There have been a few small bugs exposed by this change, missing mkdir calls and some minor test issues that needed tweaks. In general this simplifies and improves the fetcher code flow though. This completes a cleanup that ensures local files are correctly covered at parse time which ensures rebuilds and reparses happen at the right times. (Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: bb.fatal when trying to checksum non-existing filesPaulo Neves2022-07-151-0/+5
| | | | | | | | | | | | If the local fetcher was not able to find the file anywhere but it was included in the SRC_URI for checksumming just make it a fatal error. Ensure a list of searched locations is included too to match the runtime error that would have resulted. (Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101) Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ConfHandler/BBHandler: Improve comment error messages and add testsRichard Purdie2022-07-041-0/+23
| | | | | | | | | | | | | | | | | | | Currently if you trigger one of the comment errors, the newline characters are stripped and the line numbers are incorrect. In one case it prints the empty line which is also unhelpful. Rework the code around these errors so the line numbers are correct and the lines in question are more clearly displayed complete with newlines so the user can more clearly see the error. I also added a couple of simplistic test cases to ensure that errors are raised by the two known comment format errors. [YOCTO #11904] (Bitbake rev: 712da71b24445c814d79a206ce26188def8fce0a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add test for broken mirror tarballPavel Zhukov2022-06-211-0/+25
| | | | | | | | | | | | | | | With PREMIRRORS set and BB_NO_NETWORK = "1" bitbake should not try to fetch into non-initialized git directory if tarball is broken (or not in gzip format) [Yocto 14822] (Bitbake rev: c9aaca3dd2dfdf4a291d6e1f6263037e0f54b4b6) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add tests for premirror using real projectPavel Zhukov2022-06-111-0/+39
| | | | | | | | | | | Existing test uses fake local repo for better performance. Adding test which uses real (but still small) project hosted on yoctoproject.org. (Bitbake rev: 247f3536a691cdaa3f96768d1b42653b1da9ae84) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Drop unnecessary duplicated functionPavel Zhukov2022-06-111-9/+0
| | | | | | | | | Function "git" is defined in super class FetcherTest already. (Bitbake rev: bfed4ad2d50bcd635bb2ef99977e03e879fc79a2) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add tests to cover BB_FETCH_PREMIRRORONLY functionalityPavel Zhukov2022-06-081-0/+63
| | | | | | | | | | | | Basic test to cover functionality of BB_FETCH_PREMIRRORONLY using local git repository. Local repository has been chosen to allow easy manipulation with the repo to simulate behaviour reported in [Yocto 13233] (Bitbake rev: 773e0815ba0c2183afcb169cda525b7625e60e42) Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/parse: Fix one test overwriting anotherRichard Purdie2022-04-211-1/+1
| | | | | | | | | Fix an issue where two tests have the same name with one overwriting the other. (Bitbake rev: da812d938fd79e2cc7bdf355ccf5b0f9ead684c4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: canonicalize ids in generated tarballsOlaf Mandel2022-04-141-2/+4
| | | | | | | | | | | | | | Change the owner information in the mirror tarballs generated using BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit 0178ab83, which used the original pokybuild:user information, but failed to clean up the numerical user and group ids. Now set the more canonical values of oe:oe and 0:0. (Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> CC: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data: Ensure vardepsexclude or BB_BASEHASH_IGNORE_VARS covers ↵Richard Purdie2022-04-031-5/+23
| | | | | | | | | | | | | | contains items Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency on WARN_QA. Fix it and add some tests. (Bitbake rev: 6aecc2fe51a52020f6f13be08449e18d42e7a6b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: stop generated tarballs from leaking infoOlaf Mandel2022-03-261-0/+30
| | | | | | | | | | | | | | | | | | | | When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs of git repositories, they leaked local information: username, group and time of the last fetch. Remove all these by setting fixed information: * uname = pokybuild * gname = users * mtime = committer time of newest commit in repo The username and group value were taken from the archives available on the downloads.yoctoproject.org mirror. The modification time is chosen so it still retains some relationship to the contents of the archive. (Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896) Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Update for master -> main change upstreamRichard Purdie2022-02-231-1/+1
| | | | | | (Bitbake rev: f0fc0fe94161d4dd4f34df8426222ac590ef6736) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename setscene enforce filtering variableScott Murray2022-02-211-2/+2
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS (Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c) (Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename environment filtering variablesScott Murray2022-02-211-2/+2
| | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS (Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a) (Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename basehas and taskhash filtering variablesScott Murray2022-02-211-1/+1
| | | | | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS the derived code variables basewhitelist and taskwhitelist have been renamed to basehash_ignore_vars and taskhash_ignore_tasks, respectively. [RP: Added compatibility mapping code and tweaked naming] (Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/cooker: Fix source revision handling with floating upstreamsRichard Purdie2022-02-171-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | Where a git url uses a tag instead of a full source revision, breakage can currently occur in builds. Issues include: * the revision being looked up in multiple tasks (fetch and unpack) * the risk a different revision may be obtained in those tasks * that some tasks may not be allowed to access the network * that a revision may not be consistent throughout a given build * rerunning a specific task may given inconsistent results To fix this, stop the workers from cleaning out the source revision store. This should only be done in the cooker itself (based on current policy). Also, where the code "sees" an upstream access, mark the recipe as not to be cached. The reparse re-triggers the upstream lookup by the server. Add a test to ensure that if get_srcrev isn't called, the user is told they're using a configuration that is known to break. (Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Add missing branch param for git urlsRichard Purdie2022-02-121-44/+44
| | | | | | | | | We started to require the branch parameter for git urls. This updates the tests to match and quieten some noise. (Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: npm: Put all downloaded files in the npm2 directoryPeter Kjellerstedt2022-02-051-1/+1
| | | | | | | | | | | | | Previously npm files that specify downloadfilename= in the SRC_URI would be downloaded to the root of ${DL_DIR} rather than in the ${DL_DIR}/npm2 directory where all other npm files are downloaded. This should make it simpler when setting up and configuring a premirror with the downloaded npm packages. (Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Correct handling of replacing the basename in URIsPeter Kjellerstedt2022-02-051-18/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The solution implementated in commit 96c30007 (fetch2: fix downloadfilename issue with premirror) missed two corner cases. The first is if the basename of the original URI also appears somewhere else in the replacement URI, in which case it would also be replaced. The second is if the basename of the original URI partially matches the basename of the replacement URI, in which case the inital part of the basename from the replacement URI would be left behind. The second case caused test_npm_premirrors_with_specified_filename to fail. The solution is to prefix the basename with a slash when matching to avoid partial matches, and only replace the basename at the end of the URI. This also adds two test cases that test for these problems. Before this they would give the following errors: - ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ ++++++++++ - ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] + ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz'] ? +++++ (Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Make test_npm_premirrors work with the current fetcherPeter Kjellerstedt2022-02-051-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two totally opposite use cases for how a premirror is expected to behave in combination with specifying a downloadfilename= parameter in the SRC_URI. On the one hand there is the expectation that it works like any other mirror, which means the premirror is expected to contain a file with the original name specified in the SRC_URI. On the other hand there is the expectation that one can use the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in which case it is expected to contain a file with the name from the downloadfilename= parameter. The latter case has been how downloaded files have been handled until commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename defined), where the fetcher was changed to store files as per the first case. This is also when the test_npm_premirrors test case was modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to expect the first case. The above change was later reverted in commit 96c30007 (fetch2: fix downloadfilename issue with premirror). However the test_npm_premirrors test case was not updated to match, and has been failing ever since. This has probably gone unnoticed because the npm related test cases require that npm is installed on the host. This commit updates test_npm_premirrors to expect that premirrors use the filenames specified by downloadfilename= as this matches the current fetcher implementation and also is the most likely use case for premirrors. It also tries to mimic how one typically might setup the premirror directory by simply copying the download directory. (Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Only set the Git user name/email if they are not ↵Peter Kjellerstedt2022-02-051-18/+13
| | | | | | | | | | | already set If they are set, assume they are correct. (Bitbake rev: a3a3a40b96c29051d97d0c96eb01784cbd1f5420) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Unify how git commands are runPeter Kjellerstedt2022-02-051-46/+27
| | | | | | | | | | | | This unifies the three separate implementations of git() from GitMakeShallowTest, GitShallowTest and GitLfsTest into one implementation in FetcherTest. It also makes use of this in FetcherLocalTest and FetcherNetworkTest. (Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Improve the verbose messages for skipped testsPeter Kjellerstedt2022-02-051-2/+2
| | | | | | | | | | | | | | | | | | | This changes: test_foo .. skipped 'Network tests being skipped' test_foo .. skipped 'npm not installed, tests being skipped' to: test_foo .. skipped 'network test' test_foo .. skipped 'npm not installed' to avoid double "skipped" in the output. (Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Remove unnecessary \n from mirror variablesPeter Kjellerstedt2022-02-051-0/+2
| | | | | | | | | | | Since commit 044fb04d (fetch2: Allow whitespace only mirror entries) there is no need to separate the entries in mirror variables with "\n". (Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Skip the crate tests if network tests are disabledPeter Kjellerstedt2022-02-051-23/+23
| | | | | | | | | Both test_crate_url and test_crate_url_multi require Internet access. (Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add crate fetcherRichard Purdie2022-01-191-0/+35
| | | | | | | | | This imports the crate fetcher from OE-Core to resolve various module issues and adds some very very basic tests of that new fetcher. (Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Drop gnu urls from wget connectivity testRichard Purdie2021-12-201-3/+0
| | | | | | | | | These urls are no longer adding much to the test coverage but the intermittent network issues connecting to them are painful. Drop the urls. (Bitbake rev: bdf5739c5d831dc97a7d81568f94a0953c71017f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Improve lockfile handling raceRichard Purdie2021-12-141-2/+38
| | | | | | | | | | | | | | Internal bitbake shutdown timings must have changed recently as we're seeing race issues on the autobuilder around the removal of the bitbake.lock file. Improve the lockfile race code to cover bitbake's lockfile too and use it in all the tests. [YOCTO #14658] [YOCTO #14652] (Bitbake rev: bd1912bed64424f9fb28396b71bb49b6090ed087) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: add a test for version check where compression changesAlexander Kanavin2021-12-092-0/+65
| | | | | | | (Bitbake rev: b6f0c29346ad6463c0e521248633e71886bfb5dc) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: test/fetch: Add striplevel unpack parameter testStefan Herbrechtsmeier2021-11-261-0/+15
| | | | | | | (Bitbake rev: 7e1ca7ab50e3c6b642c3c11504c7c8f52cfa4528) Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix url remap issue and add testcaseRichard Purdie2021-11-211-0/+1
| | | | | | | | | | Using "" as a target for .replace() is a really bad idea as it duplicates the replacement for every character in the string. Add a testcase which triggered this and correct the code to return the correct result. (Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Set TOPDIR in testRichard Purdie2021-11-151-1/+2
| | | | | | | | | | It was clear with testing that we're asuming bitbake sets TOPDIR correctly when running these tests. Remove that implict assumption and make it explicit. (Bitbake rev: c8f1eb377ceb1fc78cbfaed976107720ad78c075) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-04 17:56:55 +0000