summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: cooker: Explictly shut down the sync threadRichard Purdie2020-08-241-1/+7
| | | | | | | | | | | | | | | | Hongxu Jia reported a problem where the bb_cache files were not always being written out correctly. This was due to the sync thread being terminated prematurely. Whilst the preceeding changes mean the exit handler for this thread is now correctly called since we switch to using sys.exit() instead of os._exit(), this write can happen after we drop the bitbake lock, leading to potential races. Avoid that headache by adding in explicit thread join() calls before we drop the lock (which atexit or Finalize can't do). (Bitbake rev: afd1900939f7b042297558f4cb01f50f3a299267) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure parse_quit thread is closed downRichard Purdie2020-08-241-6/+8
| | | | | | | | | | | Each run through the parser would leak a thread from the queue created to shut the parser down. Close this down correctly and clean up the code flow slightly whilst in the area, making sure this thread does shut down correctly (we don't care if it loses data). (Bitbake rev: 51ba35e9bbd4da8b5f3b3b5b8213cb634a6b0f06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Simplfy idle callback handler functionRichard Purdie2020-08-131-4/+6
| | | | | | | | | | Having the idle callbacks abstracted via the configuration object makes no sense. Its like this for historical reasons from the multiple server backends but we don't need this now so simplfy. (Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle multiconfig name mappings correctlyRichard Purdie2020-07-231-1/+1
| | | | | | | | | | | | | | "bitbake mc:arm:bash mc:arm:busybox" works but "bitbake multiconfig:arm:bash multiconfig:arm:busybox" does not. The reason is the list is modified whilst iterating. Don't do that. [YOCTO #13607] (Bitbake rev: cd041a78d96e656438d93fb1e288080b8a6fe8bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve multiconfig configuration error reportingRichard Purdie2020-07-231-1/+6
| | | | | | | | | This avoids a traceback if an invalid multiconfig is referenced in the bitbake commandline and tweaks the message to make it more understanable. (Bitbake rev: f31d7d0ad57b0ecc2ae06ed4b547c98df2aaa1a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix unmatched files handling leading to misleading warningsRichard Purdie2020-07-211-30/+50
| | | | | | | | | | | | | | | | | | | | | Currently if all recipes in a layer are skipped, there are warnings that the BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this for skipped recipes. The current code is hard to understand, not least as it passes variables which functions modify by reference rather than giving a return value. Update calc_bbfile_priority() to return values rather than modifying them. Refactor the code to try and make it clearer what its doing and fix the skipped recipe issue by passing in the list of parsed files. The function is complicated by the need to not rerun regex matching more than we ever have to which complicates the flow, it would be easier if we just reran operations multiple times. (Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cache: Cache size optimizationJoshua Watt2020-06-101-7/+1
| | | | | | | | | | | Now that there is a cache object per multiconfig, it is not necessary for each cache object to parse all other multiconfigs. Instead, each cache now only parses the files for it's multiconfig. (Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cache: Use multiconfig aware cachesJoshua Watt2020-06-101-17/+25
| | | | | | | | | | | | | | | Splits the parsing cache to maintain one cache per multiconfig instead of one global cache. This is necessary now that the files and appends can vary for each multiconfig. A bb.cache.MulticonfigCache dictionary-like proxy object is created instead of a single bb.cache.Cache object. This object will create and properly initialize bb.cache.Cache object for each multiconfig, and each of these caches has a dedicated cache file with a name based on the multiconfig. (Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Split file collections per multiconfigJoshua Watt2020-06-101-57/+82
| | | | | | | | | | | | Splits the cooker to track a collection per multiconfig instead of a single collection for all multiconfigs. Practically speaking, this allows each multiconfigs to each have different BBMASKs that apply to it instead of each one using the mask specified in the base configuration. (Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Respect multiconfig parameterJoshua Watt2020-03-071-3/+3
| | | | | | | | | | The cooker had a multiconfig parameter for the findProviders() and findBestProviders() API, but it was being ignored. (Bitbake rev: ea0b68ac2b77676ed1c63f0ee1ae5d300f2b4696) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Reset parse status unpon clientCompleteRichard Purdie2020-03-021-0/+2
| | | | | | | | | | | | | | | | If for example a tinfoil connection edits the datastore, a subsequent connection can be "corrupted" by those changes. By setting the parse status of the caches as False at exit, the behaviour becomes the same as a newly setup server as a new data store is setup. This avoids problems in tests when BB_SERVER_TIMEOUT is set as the server is properly reset between connections. [YOCTO #13812] (Bitbake rev: e66759106e21da2b34a6cdec7aa681ad2204da54) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Reset loghandlerRichard Purdie2020-02-191-0/+3
| | | | | | | | | When parsing, reset the loghandler when finished, else the messages can be misleading. (Bitbake rev: 7af80cd1dd577b05d39a3cc5d5c547a2549e39df) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Empty siggen cache during parsingRichard Purdie2020-02-171-1/+2
| | | | | | | | | | | | | | | | | | | When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. (Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/toaster: replaced deprecated method warn() with warning()Frazer Clews2020-02-061-1/+1
| | | | | | | | | | Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything (Bitbake rev: a194f275235f22411cb2368f06a44f61ceb6a0f3) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: amend code to use proper singleton comparisons where possibleFrazer Clews2020-01-191-4/+4
| | | | | | | | | | | amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. (Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-4/+0
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt2020-01-101-8/+8
| | | | | | | | | | When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. (Bitbake rev: 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove a left-over comment about expanded_dataPeter Kjellerstedt2019-11-141-4/+0
| | | | | | | | | | This should have been removed together with expanded_data in commit e3694e73 (cooker/command: Drop expanded_data). (Bitbake rev: 33197db8abf82be240d7c1c5c9d2484a08a90849) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Don't daemonize server processJoshua Watt2019-09-271-1/+0
| | | | | | | | | | | | | | | The hash server process is terminated and waited on with join(), so it should not be a daemon. Daemonizing it cause races with the server cleanup, especially in the selftest because the process may not have terminated and cleanup up its socket before the test cleanup runs and tries to do it. [YOCTO #13542] (Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-181-8/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"Martin Jansa2019-09-161-1/+0
| | | | | | | | | | | | | | This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82. This ignores the layer priority, making the issue much worse. E.g. I'm seeing a lot of failures caused by missing users, because base-passwd bbappends applied in unexpected order caused different passwd.master to be found in re-ordered FILESPATH. (Bitbake rev: 2dc862237dba82da37c8ac9289e0a21409b1305c) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Ensure bbappends are found in stable orderWes Lindauer2019-09-071-0/+1
| | | | | | | | | | | | | | | Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency. (Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82) Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: remove generation of recipe-depends.dotChen Qi2019-08-281-21/+4
| | | | | | | | | | | | | | | | | | | | | | The information of recipe-depends.dot is misleading. e.g. $ grep xz recipe-depends.dot | grep bzip2 "bzip2" -> "xz" "xz" -> "bzip2" Users would wonder why they get some circular dependency. The information is derived from removing the task names of task-depends.dot. It's not giving people any additonal information, and it's misleading. So we remove the generation of this file. (Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve hash server startup code to avoid exit tracebacksRichard Purdie2019-08-141-12/+17
| | | | | | | | | | | | | At exit the hashserv code was causing tracebacks as join() wasn't being called from the thread that started the process. Ensure that the hashserver is started from the pre_serve hook which is the final thread the cooker runs in. This avoids the traceback at the expense of some horrific poking into data stores which will ultimately need improving through a proper API. (Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Cleanup the queue before call process.join()Robert Yang2019-08-081-0/+8
| | | | | | | | | | | | | | | | | | | | | | Fixed: $ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p Press *one* Ctrl-C when the parsing process is at about 50%, then the processes are not exited: Keyboard Interrupt, closing down... Timeout while waiting for a reply from the bitbake server It hangs at process.join(), according to: https://docs.python.org/3.7/library/multiprocessing.html Cleanup the queue before call process.join() can fix the problem. (Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Switch from threads to multiprocessingRichard Purdie2019-08-061-9/+11
| | | | | | | | | | There were hard to debug lockups when trying to use threading to start hashserv as a thread. Switch to multiprocessing which doesn't show the same locking problems. (Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/hashserv: Allow autostarting of a local hash server using ↵Richard Purdie2019-08-061-1/+14
| | | | | | | | | | | | | | | | | BB_HASHSERVE Its useful, particularly in the local developer model of usage, for bitbake to start and stop a hash equivalence server on local port, rather than relying on one being started by the user before the build. The new BB_HASHSERVE variable supports this. The database handling is moved internally into the hashserv code so that different threads/processes can be used for the server without errors. (Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: lib: Cleanup /usr/bin/env pythonRobert Yang2019-06-281-1/+0
| | | | | | | (Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure mcdeps are processed even if only one multiconfigRichard Purdie2019-06-111-1/+5
| | | | | | | | | If you have no BBMULTICONFIG set but set mcdepends, they're currently ignored. We can handle them correctly with this small tweak. (Bitbake rev: 578f0c02f6a13f4315e7c2ce8b5e876dd2025055) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add compability handling for multiconfig: prefix migrationRichard Purdie2019-06-101-0/+3
| | | | | | | | | | This allows "multiconfig:" targets to continue to work by internally mapping them to the new "mc:" naming, allowing older builds to work as before. (Bitbake rev: c4d90890547af642e99cc541af3415df3559563e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'Richard Purdie2019-06-101-9/+9
| | | | | | | | | | | | | | After real world use its clear the "multiconfig:" prefix to multiconfig tasks, whilst clear, is also clumbersome. Switch to use the short version instead. mcdepends will continue to work with "multiconfig:" for now as well. The commandline will only accept mc: going forward. [YOCTO #11168] (Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Show base multiconfig environmentJoshua Watt2019-05-301-10/+21
| | | | | | | | | | | | | | | | Adds support to the 'bitbake -e' command so that it can display the base environment for a multiconfig. It was previously possible to get the base environment for the main environment by running "bitbake -e", but there was no support for getting the base environment for a multiconfig without specifying a recipe. A user can now print the base environment for the multiconfig "foo" by running: $ bitbake -e multiconfig:foo (Bitbake rev: 3d657af8a6120193d45d01968605b30075a56198) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Rename __depends in all multiconfigsJoshua Watt2019-05-301-2/+3
| | | | | | | | | | | | | | | | | | | The renaming of the __depends variable to __base_depends and file watches needs to occurs for all multiconfigs, not just the base config. Failing to do this for all multiconfigs will result in a huge increase in the size of the parsing cache (about 5x for a single mulitconfig) because all multiconfig caches will still depend on the base config files. This will also seen a similar jump in the amount of time required to load the parsing cache from memory, both because the cache is larger and because of explosion of additional existence checks that must be done for the base files. [YOCTO #13359] (Bitbake rev: 34137a00f60280e9e806070c6507a0fc6284b834) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-13/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is emptyRobert Yang2019-03-241-1/+1
| | | | | | | | | | | | | | | | | | The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty, this caused "bitbake-layers show-layers" can't show these layers, this was incorrect since these layer did exist. Add these layer to bbfile_config_priorities can fix the problem. Fixed: Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf $ bitbake show-layers | grep oe-core There was nothing, now the layer is shown (Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Tweak multiconfig dependency resolutionRichard Purdie2019-02-251-29/+24
| | | | | | | | | | | | | | | | | | | There were a couple of problems with the multiconfig dependency resolution: - the "if mc" condition triggering this code wasn't correct, it needs to be "if more than one multiconfig" configured - after adding providers we need to call add_unresolved again and rebuild mcdeps within the "while new" loop By fixing these issues we allow various other combinations of multiconfig builds to work which previously didn't. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Split recipes to parse amongst threads ahead of timeRichard Purdie2019-01-081-11/+6
| | | | | | | | | | | | | | | | We have two choices, split the recipes amongst the parsing threads in blocks ahead of time, or have a queue which parsers pull from when idle. The optimum approach depends on how similar the pieces are. For the single recipe reparse case, there is currently a significant wait for the feeder thread to start (around 0.25s in a 2s command). Its possible splitting into blocks in advance may be unluckly for some other workloads but experimentally it seems to work better overall for me at least. (Bitbake rev: ae79868861568d673a70472e85a4bde9e2d84a8f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove Feeder() since its no longer needed with moderm ↵Richard Purdie2019-01-081-37/+5
| | | | | | | | | | | | | multiprocessing There used to be many bugs in multiprocessing and we implemented our own feeder process to avoid them. Now that we have python 3.x, these are fixed and just using the standard Queue mechanism appears to work fine. We can therefore drop the unneeded code and simplify. (Bitbake rev: b2d39fc37fcf3c81a562ec1ef4f8b4c1a493fc57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix indirect multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2019-01-031-20/+31
| | | | | | | | | | | | | | | | | | When an indirect multiconfig dependency exists, such as: A depends on B, B has a multiconfig dependency to C,and our build target is A, the multiconfig dependency to C is not processed on time, hence no providers are added for it, causing an exception in the runqueue because the dependency does exist in it. Call add_unresolved() for all available multiconfigs before processing providers for multiconfig dependencies, detecting mcdepends on time so providers for them can be added correctly. (Bitbake rev: 8a6bc7584ad61b4de98af92a86066602006262f9) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add some timing debug messages to the server startupRichard Purdie2018-12-081-0/+19
| | | | | | | | | We're seeing slow startup in bitbake, add some timeing debug messages so the logs are more useful for debugging when its slow. (Bitbake rev: 8d1fc115b8a176009f1f3a8ce840b422e7e0b45e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2018-11-221-8/+1
| | | | | | | | | | | | | | | | | | | | | | | | | When multiconfig is enabled the cooker adds providers for all the targets to be built on all the multiconfig variables that were set, regardless if there is a dependency to it or not. This causes an issue when a certain target is incompatible with one or more of the multiconfigs, e.g. the target is not in COMPATIBLE_MACHINE for one of the MACHINEs being built, causing the cooker to error out since no providers can be found for that certain target on that multiconfig. This patch modifies the behavior to only look for PROVIDERS for a target on the multiconfig that was selected to be built, PROVIDERS are then looked for in other multiconfigs only when there is a defined dependency to them. [YOCTO #12985] (Bitbake rev: f2106a3a767542359fdde238abcf5fe35ab3a144) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/runqueue: Turn universe warnings into verbnotesRichard Purdie2018-09-101-1/+1
| | | | | | | | | If the user puts universe on the commandline, they don't really want warnings so use the new verbnote level instead. (Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker.py: Fix incorrect bb files matched warningMark Hatle2018-09-051-1/+4
| | | | | | | | | | | | | | | | | | | | In the case of a sublayer of an existing layer, where the sublayer and main layer share a path, the system may not match the paths properly resulting in: No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer' because it has already matched the main layer. Fix this issue by sorting the collection items based on the pattern, using longest to shortest. Obviously regex wildcards could still be an issue but these are typically not used, so this simply fix should work in the existing cases. (Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add support for multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2018-08-011-3/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the capability for tasks from different multiconfigs to depend on one another. These dependencies can be enabled using the following format: task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on" For the sake of simplicity consider the following example: Assuming we have set up multiconfig builds, one for qemux86 and one for qemuarm, named x86 and arm respectively. Adding the following line to an image recipe (core-image-sato): do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" Would state that core-image-sato:do_image from x86 will depend on core-image-minimal:do_rootfs from arm so it can be executed. This patch makes modifications to: - cooker: To glue both multiconfigs in one place and make sure the dependencies can be provided. - taskdata: To parse and add a new kind of dependency (mcdepends) to the taskdata object. - runqueue: To differentiate tasks from different multiconfigs, add the specified dependencies to the corresponding tasks, and create a working runqueue that contains tasks from both multiconfigs. - siggen: To avoid looking for tasks from different multiconfigs on objects where they dont belong. The taskdata objects are still not aware of the concept of multiconfig, so each object doesnt know which multiconfig its building, hence why the mcdepends are added to all taskdata objects equally (we really dont expect many of these), but the actual dependencies are added only to the required tasks by the runqueue. (Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix environment double key expansion issueRichard Purdie2018-03-041-3/+2
| | | | | | | | | | | The base configuration needs key expansion and anon python execution, the parsed configurations do not. Fix this consistently, its been broken and causing double key expansion for a while, only relised when we started double anonymous python exeution too. (Bitbake rev: 6138897de5ac6becf3bff56ce7a78f3ec208fcdf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Run registered anonymous python before displaying environmentRichard Purdie2018-03-041-1/+3
| | | | | | | | | | | The output of bitbake -e can differ from what actually is used due to anonymous python making changes to the data store. Execute any anonymous python added in the base configuration to make things more consistent. (Bitbake rev: bcdc2f73e3b4a10b1e479c2891f251d9507a9e30) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: don't stop file notifier when cooker is shutdownRobert Yang2018-02-141-2/+0
| | | | | | | | | | | | | | | | | | | | | It should be live/exited with server rather than cooker, fixed: $ bitbake --server-only -T -1 Set MACHINE = "qemux86" in conf/local.conf $ bitbake quilt Set MACHINE = "qemuppc" in conf/local.conf $ bitbake quilt [snip] ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed. [snip] This is because the server doesn't know local.conf is changed since the notifiers are stopped, so it doesn't reparse, and then we would get the errors, let the notifiers live/exited with server can fix the problem. (Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: fix for BBFILE_PATTERN matches bbappendRobert Yang2018-02-141-3/+7
| | | | | | | | | | | | | | | | | | | | | The old code couldn't handle nestled layers correctly, e.g.: parent_layer/sub_layer/foo.bb Note there are two layers, parent_layer and sub_layer. And in parent_layer/conf/layer.conf: BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/" This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb, so it warns that no files matched sub_layer, this is the expected behavior, but it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it should warn and let the user fix the problem. Check the bbappend in already "matched set" before return it as matched by "unmatched set" can fix the problem. (Bitbake rev: ec90245d28e52ea718d2ce084eb304cdc4355c9c) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-20 09:19:43 +0000