summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/codeparser.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: codeparser: Add function decorators for vardepsJoshua Watt2025-04-101-3/+7
| | | | | | | | | | | | Adds bb.parse.vardeps bb.parse.excludevardeps function decorators that can be used to explicitly add or exclude variables from a python function parsed by bitbake (Bitbake rev: 030fb3dee067640a3a50f24a53d200bdb5048376) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Skipping typing when inspecting Python modulesPedro Ferreira2025-03-091-2/+10
| | | | | | | | | | | | If a custom python module is added thru BBIMPORTS and it uses typing(Any,Tuple,Union...), codeparser will fail because inspect.py raises TypeError exception if the object is a built-in module, class, or function. (Bitbake rev: 0ecfd0b8540220633e71d24cd73cc5306863ae3c) Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Allow code visitor expressions to be declared in metadataRichard Purdie2024-08-291-4/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allow the metadata to define code visitor expressions which mean that custom dependencies can be handled in function libraries. An example is the qa.handle_error function in OE which can set something like: """ def handle_error_visitorcode(name, args): execs = set() contains = {} warn = None if isinstance(args[0], ast.Constant) and isinstance(args[0].value, str): for i in ["ERROR_QA", "WARN_QA"]: if i not in contains: contains[i] = set() contains[i].add(args[0].value) else: warn = args[0] execs.add(name) return contains, execs, warn handle_error.visitorcode = handle_error_visitorcode """ Meaning that it can have contains optimisations on ERROR and WARN_QA instead of hard dependencies. One drawback to this solution is the parsing order. Functions with visitorcode need to be defined before anything else references them or the visitor code will not function for the earlier references. (Bitbake rev: 5bd0c65c217394cde4c8e382eba6cf7f4b909c97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Skip non-local functions for module dependenciesRichard Purdie2024-07-011-0/+5
| | | | | | | | | | | | | | If modules do something like "from glob import glob" then we end up checksumming the glob code. That leads to bugs as the code can change between different python versions for example, leading to checksum instability. We should ignore functions not from the current file as implemented by this change. (Bitbake rev: 1e6f862864539d6f6a0bea3e4479e0dd40ff3091) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data: Ensure module function contents changing is ↵Richard Purdie2024-07-011-11/+17
| | | | | | | | | | | | | | | | | | | | accounted for Currently, if a pylib function changes contents, the taskhash remains unchanged since we assume the functions have stable output. This is probably a poor assumption so take the code of the function into account in the taskhashes. This avoids certain frustrating build failures we've been seeing in automated testing. To make this work we have to add an extra entry to the python code parsing cache so that we can store the hashed function contents for efficiency as in the python module case, that isn't used as the key to the cache. The cache version changes since we're adding data to the cache. (Bitbake rev: b2c3438ebe62793ebabe2c282534893908d520b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: remove redundant list conversionAntonin Godard2024-05-211-1/+1
| | | | | | | (Bitbake rev: 89712949de9476e4674864a8dcd6862fefe92eae) Signed-off-by: Antonin Godard <antoningodard@pm.me> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: support shell substitutions in quotesAntonin Godard2024-05-211-7/+22
| | | | | | | | | | | | | | | | | | | | | | | | | The current shell substitution mechanism only works without quotes. For example: var1=$(cmd1 ...) Will work and add `cmd1` to the correspondind `run.do_*` file. However, although quite common, this syntax is not supported: var1="$(cmd1 ...)" This commit adds this feature by adding a step to process_words() to check whether we are dealing with quotes first, and by iterating on what's between them to detect new shell substitution candidates. These candidates are tested and parsed like before in the next step. The original `part` being part of the candidates means the syntax var1=$(cmd1 ...) is still valid. (Bitbake rev: f56e1a37b2ba1773ed308043d7eb073cc2e6c06e) Signed-off-by: Antonin Godard <antoningodard@pm.me> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake/codeparser.py: address ast module deprecations in py 3.12Alexander Kanavin2024-01-021-10/+10
| | | | | | | | | | | | | Specifically: /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead if isinstance(node.args[0], ast.Str): /srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead value = node.args[0].s (Bitbake rev: de8ba2770d9a1a94af3d084f9540da7e2fae6022) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: add missing 'import os'Chris Laplante2023-10-231-0/+1
| | | | | | | | (Bitbake rev: 302969885d37a76edec3aa79181e98f8d7e28021) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: replace deprecated ast.Str and 's'Chris Laplante2023-10-171-2/+2
| | | | | | | | | These have been deprecated since 3.8 (Bitbake rev: 5419a8473d6d4cd1d01537de68ad8d72cf5be0b2) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Update debug variable referenceRichard Purdie2023-09-221-1/+1
| | | | | | | | | The code has changed and the debug message didn't work. Fix it. The output is still incredibly useful. (Bitbake rev: f1fa4bb3066e2bbaff0b69088ba5c6c6c597b93d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache ↵Richard Purdie2023-01-261-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | location Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data: Add vardepsexclude support to module dependency codeRichard Purdie2022-12-171-0/+6
| | | | | | | | | | | We need to be able to exclude dependencies from the python module dependency code. Add support for the vardepexclude flag for these. It only works from the configuration namespace rather than per recipe for efficiency. (Bitbake rev: 1aa672b01037fda4ca82f2c7e394783287c09ecd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/data/codeparser: Add dependencies from python module functionsRichard Purdie2022-12-171-5/+38
| | | | | | | | | | | | | | | | | | | | Moving code into python modules is a very effective way to reduce parsing time and overhead in recipes. The downside has always been that any dependency information on which variables those functions access is lost and the hashes can therefore become less reliable. This patch adds parsing of the imported module functions and that dependency information is them injected back into the hash dependency information. Intermodule function references are resolved to the full function call names in our module namespace to ensure interfunction dependencies are correctly handled too. (Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c) (Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add copyright headers where missingRichard Purdie2022-08-121-0/+2
| | | | | | | | | Where copyright headers were not present, add them to make things clear. (Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Avoid log bufer overhead in cache caseRichard Purdie2022-03-181-2/+18
| | | | | | | | | | | | | | | | | Creating the new log instances triggers a lot of python logging overhead in a commonly called function (about 600,000 for parsing OE-Core). We only need the log functionality if we're parsing, not if we just hit from the cache. Therefore defer the log setup overhead until we know it is a cache miss. Whilst this complicates the code slightly, the performance gain is worth it as for parsing OE-Core we drop 60 million funciton calls (from 225 overall). (Bitbake rev: ac868167ad854f9bb32dcb2e63528870547805a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Fix TypeError in bitbake debug modeTomasz Dziendzielski2021-03-151-3/+3
| | | | | | | | | | | | | | | | | Commit 75f87db413 fixed the confusion between bitbake and python logger but in codeparser still old method of setting debug level was used causing TypeError, because debug level value was incorrectly returned and assigned to event.msg. | File "./bitbake/lib/bb/ui/knotty.py", line 660, in main | event.msg = event.fn + ': ' + event.msg | TypeError: can only concatenate str (not "int") to str [YOCTO #14298] (Bitbake rev: bec9ea7ab28a8dfad1a6010ca5c6ec691754748d) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-24/+2
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+4
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Switch to sha256 from md5Richard Purdie2019-01-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've reports of hash collision with codeparser. Looking at the way collision problems occur with md5 and the way our function templating works, I can believe we may run into issues. This patch therefore switches to sha256. Performance wise, parse time could appear to rise by 4s in 374s Before: 384329 in 2.966s (md5) After: 349743 in 2.340s (sha256) 34723 in 1.245s (md5) since we still have md5 used elsewhere in the code, something we should look at next (using sha256 everywhere is around 5.3s in total) Unfortunately this does nearly double the size of the codeparser cache file due to the hash size change. (Bitbake rev: 4bed7a97198176503fe8c72d8dd7c36b28fc9dd1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: pysh: Improve error handling for shell codeRobert Yang2018-12-091-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The p_error() is used for printing errors when parse shell code, but it can't the EOF error correctly - Add the following lines to quilt.inc do_configure_prepend () { find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \; if [ hello ]; then } - Before the patch: $ rm -fr cache/ tmp/cache/; bitbake -p [snip] WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb [snip] bb.pysh.sherrors.ShellSyntaxError: None followed by: We can see that this isn't easy to debug, let p_error() check wheter it is EOF and print appropriate errors can improve the error message. And don't let codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error() only raise ShellSyntaxError), but make it print the last 5 lines which might be useful for debuging. - After the patch $ rm -fr cache/ tmp/cache/; bitbake -p [snip] ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are: find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \; if [ hello ]; then autotools_do_configure sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile [snip] File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None): w('Unexpected EOF') > raise sherrors.ShellSyntaxError(''.join(msg)) bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF (Bitbake rev: 44790597951638e32eb1672de2e40bd5a603326b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Track oe.utils.conditional usageMartin Jansa2018-07-301-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | * base_conditional was already removed from oe-core: http://git.openembedded.org/openembedded-core/commit/?id=0391fcad9103abca0796a068f957d0df63ab4776 after the usage was migrated to oe.utils.conditional: http://git.openembedded.org/openembedded-core/commit/?id=c97acbd034532895ce57c6717ed1b3ccc7900b0d so we can handle just ".conditional" version * add 1st parameter to variable dependencies, that way when you use FOO = "${@oe.utils.conditional('VAR', 'VALUE', 'true', 'false')" FOO variable will have dependency on VAR variable and you don't need to add FOO[vardeps] += "VAR" manually every time you use oe.utils.conditional * this is similar to contains tracking from: https://bugzilla.yoctoproject.org/show_bug.cgi?id=3890 http://git.openembedded.org/bitbake/commit/?id=ed2d0a22a80299de0cfd377999950cf4b26c512e http://git.openembedded.org/bitbake/commit/?id=0b9d117631ce909312d53b93289e61defc6be01c but conditional is simpler, we don't need to handle the first parameter as a set (Bitbake rev: 5156b4bb6876dac636be9726df22c8ee792714dd) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: add some commentsPaul Eggleton2017-04-051-0/+23
| | | | | | | | | | | Add a few comments at the top of the file explaining what it's for, and a comment pointing out that you need to increment the cache version when changing any code that changes the output. (Bitbake rev: c1780bc1872bb35bc28c76eff9110717f02d9db2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: improve handling of contains_any() and filter()Paul Eggleton2017-04-051-3/+8
| | | | | | | | | | | | | | | | Ensure we handle bb.utils.contains_any() as separate items, rather than how we handle contains() where every item must be in the list. Additionally, enable handling bb.utils.filter() which for the purposes of looking at dependencies is the same as contains_any(). Additionally bump the codeparser cache and recipe cache versions to invalidate the user's existing caches (ensuring that the changes take effect and avoiding "taskhash mismatch" errors respectively). (Bitbake rev: 496e3c84820a2a889d99d3604659e47a550941d5) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/codeparser: ensure BufferedLogger respects target logging levelPaul Eggleton2017-03-291-1/+2
| | | | | | | | | | | | | | BufferedLogger was sending log records to the target logger without checking if the logger is enabled for the level of the record - and handle() doesn't check this either (it's normally checked earlier when the relevant log function is called e.g. logger.debug()), leading for example to debug messages from codeparser getting printed when the log level for the main BitBake logger was set to logging.WARNING. (Bitbake rev: 968a77388dd1a24c1dadec6ce49bf53ebb5b643f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser.py: support deeply nested tokensPatrick Ohly2016-11-301-12/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | For shell constructs like echo hello & wait $! the process_tokens() method ended up with a situation where "token" in the "name, value = token" assignment was a list of tuples and not the expected tuple, causing the assignment to fail. There were already two for loops (one in _parse_shell(), one in process_tokens()) which iterated over token lists. Apparently the actual nesting can also be deeper. Now there is just one such loop in process_token_list() which calls itself recursively when it detects that a list entry is another list. As a side effect (improvement?!) of the loop removal in _parse_shell(), the local function definitions in process_tokens() get executed less often. Fixes: [YOCTO #10668] (Bitbake rev: d18a74de9ac75ba32f84c40620ca9d47c1ef96a3) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Use hashlib for hashing, not hash()Richard Purdie2016-06-031-2/+6
| | | | | | | | | | | | | | | "hash() is randomised by default each time you start a new instance of recent versions (Python3.3+) to prevent dictionary insertion DOS attacks" which means we need to use hashlib.md5 to get consistent values for the codeparser cache under python 3. Prior to this, the codeparser cache was effectively useless under python3 as shown by performance regressions. (Bitbake rev: 12d43cf45ba48e3587392f15315d92a1a53482ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Small optimisation to stop repeated hash() callsRichard Purdie2016-06-031-3/+4
| | | | | | | | | No functionality change, just avoids function call overhead in a function which loops heavily. (Bitbake rev: 633c0c19f87a92497a7e9771811cdc953e1b7047) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-10/+5
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Increase cache version after code parser dependency changesRichard Purdie2016-05-191-1/+1
| | | | | | (Bitbake rev: 4d880f1d5794f569b6eb5f6e7e3001dd671633c0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.codeparser: track variable flag referencesChristopher Larson2016-05-191-3/+9
| | | | | | | | | | Previously we only tracked the flags (minus excluded) of variables we depend on, but not the flags we use explicitly. (Bitbake rev: bdeb3dcd7c92e62a7c079e7b27048c4114f24a3a) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/cache: drop some unused argumentsMarkus Lehtonen2016-02-181-4/+4
| | | | | | | | | | Drop unused 'd' argument from the cache save methods, simplifying the API. (Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Improve handling of data.expand() dependenciesRichard Purdie2016-02-041-0/+12
| | | | | | | | | | | | | | | | | Currently bitbake doesn't parse into data.expand() expressions, relying on high level expansion of python code to handle this. One of the tests does however test this works. We don't really want to be doing string expansion on python code, so specifically parse into expand() function calls so that when the high level behaviour is tweaked, the self tests continue to pass and that we do continue to handle expand() function calls as best we can. (Bitbake rev: b12c17be5e4a74c9680876605c87f46501f78d28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Add support for correct linenumbersRichard Purdie2016-01-061-2/+8
| | | | | | | | | | | | | | | Currently, if there is something like a python indentation error in a python function, the linenumbers and even file aren't reported correctly. This allows lineno and filename parameters to be passed in to correct this. The lack of a lineno parameter to python's compile() function is worked around by using empty linefeeds. Ugly, but effective and with minimal performance overhead. (Bitbake rev: 5796ed550d127853808f38257f8dcc8c1cf59342) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Only load the codeparser cache onceRichard Purdie2015-11-161-0/+4
| | | | | | | | | | The server state gets reset multiple times during startup and currently we reload the codeparser cache each time. This is pointless and causes unnecessary interaction time with bitbake. (Bitbake rev: 4f700316933e8e7b2d27366e5ce6176895b913e7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Add repr() methodsRichard Purdie2015-06-271-0/+5
| | | | | | | | | These aid when debugging parts of the codeparser cache since the object contents becomes identifiable. (Bitbake rev: 344b098c7eafc2bcc5c6b44ea47985bc0cb446b5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Allow empty functionsRichard Purdie2015-06-271-0/+3
| | | | | | | | | | | | | | | | | The main parser and other code copes with empty python functions but the python codeparser does not. Fix this to avoid errors with code like: python do_build () { } which currently results in the obtuse: "Failure expanding variable do_build: IndexError: string index out of range" [YOCTO #7829] (Bitbake rev: e4f594c670189e04d58ce7d160fc1d86123620af) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.codeparser.py: Remove reference for oe.utils.containsOtavio Salvador2014-11-201-1/+1
| | | | | | | | | | | | The oe.utils.contains function has been removed from OE-Core metadata as the references for it has been replaced to use the bb.utils.contains. (Bitbake rev: 5cfdebe7a67dccc7552ff80c1ccc970e36d562df) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser cache improvementsRichard Purdie2014-07-261-34/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out the codeparser cache is the bottleneck I've been observing when running bitbake commands, particularly as it grows. There are some things we can do about this: * We were processing the cache with "intern()" at save time. Its actually much more memory efficient to do this at creation time. * Use hashable objects such as frozenset rather than set so that we can compare objects * De-duplicate the cache objects, link duplicates to the same object saving memory and disk usage and improving speed * Using custom setstate/getstate to avoid the overhead of object attribute names in the cache file To make this work, a global cache was needed for the list of set objects as this was the only way I could find to get the data in at setstate object creation time :(. Parsing shows a modest improvement with these changes, cache load time is significantly better, cache save time is reduced since there is now no need to reprocess the data and cache is much smaller. We can drop the compress_keys() code and internSet code from the shared cache core since its no longer used and replaced by codeparser specific pieces. (Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Fix to better catch all getVar referencesRichard Purdie2014-05-031-3/+3
| | | | | | | | | | | | Currently if you do localdata.getVar, the code parser simply ignores the references. Change the code to use endswith() to catch more of the references. These names are probably unique enough to get away with this. Bump the cache version to ensure things get updated. (Bitbake rev: cf763cddc3faa2361b4c4dbd08419e4ebabf208f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils, bb.codeparser: Add bb.utils.contains_anyOtavio Salvador2014-04-291-1/+1
| | | | | | | | | | | | | | | | | This includes contains_any in the special handling code for sstate. It does not take into account the equivalence of the values. In current code, considering 'bb.utils.contains_any("A", "foo bar", ...)': A = "foo" A = "bar" A = "foo bar" All those will get different signatures. (Bitbake rev: d1e3345d715e488ec3f5515fb0e1fb39366346bc) Signed-off-by: Otavio Salvador <otavio@ossystems.com.br> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: don't interact with the cache for subshellsChristopher Larson2014-04-291-9/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Doing so was causing leakage between the execs of the main value and that of the subshell value, and was causing the cached subshell value to be used for the overall variable. At the least this could cause execs contamination between two variables that while differing, run the same subshell. Beyond that, it's possible we could have been using an incomplete cached value of a subshell for that of the main value. Before this, bb_codeparser.dat would change between parses with differing bbfile parse order. After, it does not change. The codeparser cache version is bumped, to ensure we don't use potentially incorrect cached values from previous runs. This should hopefully resolve the difficult-to-reproduce issues we've seen at Mentor Graphics where bitbake emits a script to run a task and misses dependent functions, resulting in 'command not found' failures. This issue has also been mentioned on the oe devel list, where someone hit a case where oe_runmake was missing from a do_install task (IIRC). Adding debug information showed that bitbake's information about the variable dependencies for this task is inaccurate in the failure cases. (Bitbake rev: 97537e4786a1e3a329249497498b59b8f5174fc3) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Fix var_execs to append to execs, not referencesRichard Purdie2014-03-021-1/+1
| | | | | | | | | | | When using the "execs" information in new code, it became clear that the returned data was incorrect and there were missing exec'd functions. This corrects the error and changes one of the test results to match the correct behaviour. (Bitbake rev: 8a24f2d3b735bbc59ca4a09670cabbadb1868c1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data_smart: Optimise parsing speedRichard Purdie2013-12-031-3/+6
| | | | | | | | | | | | | | | | | | | | | The previous "contains" changes caused a ~3% parsing speed impact. Looking at the cause of those changes was interesting: * Use of defaultdict was slower than just checking for missing entries and setting them when needed. * Even the "import collections" adversely affects parsing speed * There was a missing intern function for the contains cache data * Setting up a log object for each variable has noticeable overhead due to the changes in the code paths uses, we can avoid this. * We can call getVarFlag on "_content" directly within VariableParse for a noticeable speed gain since its a seriously hot code path. This patch therefore tweaks the code based on the above observations to get some of the speed back. (Bitbake rev: fca802187a2a30686a8a07d2b6b16a3e5716e293) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data/codeparser: Improve handling of contains functionsRichard Purdie2013-11-261-3/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | One of the current frustrations with the sstate checksums is that code like base_contains('X', 'y',...) adds a full dependency on X and varies depend even on whitespace changes in X. This patch adds special handling of the contains functions to expand the first parameter and check for the flag specified by the second parameter (assuming its a string). The result is then appended to the value of the variable with a "Set" or "Unset" status. If the flag is added/removed, the stored variable value changes and hence the checksum changes. No dependency on X is added so it is free to change with regard to other flags or whitespace. This code is far from ideal, ideally we'd have properly typed variables however it fixes a major annoyance of the current checksums and is of enough value its worth adding in a stopgap solution. It shouldn't significantly restrict any propely typed variable implementation in future. (Bitbake rev: ed2d0a22a80299de0cfd377999950cf4b26c512e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Drop unneeded variable separationRichard Purdie2013-11-261-3/+1
| | | | | | | | | There is no good reason to separately track var_references and references so merge them and remove the unneeded variable. (Bitbake rev: 64d4cbd6360c96574cece70205ea3aecc3f8bae6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Track bb.utils.contains usageRichard Purdie2013-02-151-2/+3
| | | | | | | | | | | | | | The bb.utils.contains function usage is getting increasingly used in the metadata but isn't handled automatically by the python dependency tracking code. This patch changes that and also adds the "OE" names for the functions. Whilst there are reasons this is a bad idea, its likely outweighed by the shear number of these references and the current holes in dependency information which we're now relying heavily upon. (Bitbake rev: 0b9d117631ce909312d53b93289e61defc6be01c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Track appendVar and prependVar calls as we do for getVarRichard Purdie2013-02-151-1/+1
| | | | | | | | | We need to track appendVar and prependVar calls just as we do for getVar in order to ensure we're not missing variable dependencies. (Bitbake rev: 767b4751232f4ee3979deb4d3f733fcf9ee2bd44) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: refactor out codeparser cache into a separate classPaul Eggleton2012-05-231-134/+57
| | | | | | | | | | We want to be able to reuse most this functionality for the file checksum cache. (Bitbake rev: 0fe3cb1438d297f90dd0fc6b26362ecbff75c76d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* codeparser: Call intern over the set contents for better cache performanceRichard Purdie2012-03-121-0/+21
| | | | | | | | See the comment in the code in the commit for more information. (Bitbake rev: 2d56dc7b1f0d186e14c4c8a949b280b6b3fc31de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-23 17:34:04 +0000