summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: cooker: Ensure commands clean up any parser processesRichard Purdie2022-12-311-0/+3
| | | | | | | | | When finishing a command, we need to ensure any parsing processes that may have been started are cleaned up before we reset the cooker state. (Bitbake rev: 6569ab64bea35de21acc89053ba76e2828163f3f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/cookerdata: Rework the way the datastores are resetRichard Purdie2022-12-311-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | As far as I could tell, the current code could result in some strange situations where some data was set back to the original data store copy but the multiconfig data was not restored. There are also some changes made to the datastore which did not persist. The data store was also being reset at every client reset, which seems a little excessive if we can reset it to the original condition properly. Move the __depends -> __base_depends rename into databuilder along with the __bbclasstype change so these are saved in the original data. Tweak the databuilder code to be clearer and easier to follow on which copies are where, then save copies of all the mc datastores. Finally, drop the cache invalidation upon reset for the base config as we shouldn't need that now (oe-selftest -r tinfoil works with memory resident bitbake which was the original reproducer requiring that change). (Bitbake rev: 5f8b9b9c35b4ec0c8c539bf54ca85f068f4a5094) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Add enable/disable heartbeat codeRichard Purdie2022-12-311-0/+4
| | | | | | | | | | | Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. (Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure cache is cleared for partial resetsRichard Purdie2022-12-291-0/+3
| | | | | | | | | | We're still seeing occasional SiggenRecipeInfo coherency issues, add some further reset points into the parsing code to ensure the cache is cleared before reparsing. (Bitbake rev: 26ed783caf11dc9ebf53d3790681eb44c0c360f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/siggen: Fix cache issues with signature handlingRichard Purdie2022-12-291-0/+1
| | | | | | | | | | | | | There is a bug in the current cache code where the restored data structures were "inverted" leading to some very weird issues, not sure how anything worked like that, this patch fixes it. Also fix some issues with multiconfig cache ordering problems by resetting the stream counters when appropriate. (Bitbake rev: cd06beb948eff5eaf2d474f5b127d51a61b0b2ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Allow compression of the data in SiggenRecipeInfoRichard Purdie2022-12-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | The data in SiggenRecipeInfo is large and has a lot of duplication. The size causes a few problems, impacting: - bitbake's overall memory usage - the amount of data sent over IPC between parsing processes and the server - the size of the cache files on disk - the size of "sigdata" hash information files on disk The data consists of strings (some large) or frozenset lists of variables. To reduce the impact we can: a) deplicate the data b) pass references to the object on the second usage (e.g. over IPC or saving into pickle). This patch does this for SiggenRecipeInfo mostly behind the scenes but we do need a couple of reset points so that streamed data is written correctly on the second usage. (Bitbake rev: 9a2b13af483c20763d6559a823310954884f6ab1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen: Rework stamps functionsRichard Purdie2022-12-171-1/+1
| | | | | | | | | | | | | | | The current method of passing either a task's datastore, or dataCaches and a filename into the stamp functions is rather horrible. Due to the different contexts, fixing this is hard but we do control the bitbake side of the API usage so we can migrate those to use other functions and then only support a datastore in the public bb.build API which is only called from task context in OE-Core. (Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add dummy dataCaches from task context/datastoreRichard Purdie2022-12-171-0/+2
| | | | | | | | | | | | | | | One of the challenges in maintaining the code is that it sometimes uses a datacaches structure and sometimes a datastore. Rather than continue the current dual API madness, have the worker contexts create a dummy datacaches structure with the entries we need. Whilst this does need to be kept in sync with the real structure, that doesn't change and this allows the code to be simplified. With this new approach, we can unify the stamps dependency code again. (Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Start sync thread a little earlierJoshua Watt2022-12-081-6/+7
| | | | | | | | | | | Starts the sync thread slightly earlier to give it some extra time to dump out the caches while the main process tears down the parsing processes (Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Use event to terminate parser threadsJoshua Watt2022-12-081-12/+3
| | | | | | | | | | Uses an event to terminate the parser threads instead of a queue. This is not only simpler, but saves about 500ms on shutdown time (Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/cache: Optionally allow adding siggen hash data to the ↵Richard Purdie2022-12-081-5/+5
| | | | | | | | | | | | | | | bitbake cache Being able to track siggen hash construction data can be useful for cache debugging. For now, add an extra cache class which contains this information. It can be enabled in the same way as the hob data cache through a feature flag to cooker. This allows us to experiment with the data without carrying larger patches around and ultimately may allow use to have a hash mismatch debugging mode that is more easily enabled. (Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop broken/unused codeRichard Purdie2022-11-201-2/+2
| | | | | | | | | | | | | | Some parts of functions in Cache() were broken and unused, there was also a totally unused function. This was historical as a result of the cooker parsing process needing to handle cached entries in the main thread but parsing actions in seperate processes. Document the way it works, update the function name to be clear about what it now does and drop the old code which was unused. (Bitbake rev: af83ee32df85c8e4144f022a1f58493eb72cb18e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/cookerdata: Move recipe parsing functions from cache to ↵Richard Purdie2022-11-201-2/+1
| | | | | | | | | | | | | | | databuilder When 'NoCache' was written, databuilder/cookerdata didn't exist. It does now and the recipe parsing functionality contained in NoCache clearly belongs there, it isn't a cache function. Move those functions, renaming to match the style in databuilder but otherwise not changing functionality for now. Fix up the callers to match (which make it clear this is the right move). (Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix a typoOliver Lang2022-10-251-1/+1
| | | | | | | (Bitbake rev: cf23612f4e8946b9ed4c9f87b451f32b8c471df2) Signed-off-by: Oliver Lang <quantenkeks@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Fix a few more logger debug() callsJoshua Watt2022-09-211-12/+12
| | | | | | | | | | | f68682a7 ("logging: Make bitbake logger compatible with python logger") replaced several .debug() calls to make them comply with the standard python logging API, but a few were missed. (Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: BBHandler/cooker: Implement recipe and global classesRichard Purdie2022-08-121-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Drop sre_constants usageRichard Purdie2022-06-111-3/+2
| | | | | | | | | | | | | | | | | As reported by Martin Jansa <Martin.Jansa@gmail.com>: bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. (Bitbake rev: 3c0cd401472ffee06d5a93bdba566cb033851fcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Drop unused loopRichard Purdie2022-04-211-13/+12
| | | | | | (Bitbake rev: 1c811ad6f10560e7a7fb6830cf83707551ba04bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Reset and rebuild inotify watchesRichard Purdie2022-04-031-25/+29
| | | | | | | | | | | | | The recent inotify changes can cause entire build trees to be monitored which is suboptimal for performance. Rather than trying increasingly convoluted tricks to try and handle add/removed directories, rebuild the inotify watch when we reparse the configuration or metadata. (Bitbake rev: 3df322a200c28b45af1f2c92478c85eb7d20c38b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Avoid error if siggen wasn't setupRichard Purdie2022-04-011-1/+2
| | | | | | | | In some cases we'd never have setup a siggen so don't error in that case. (Bitbake rev: bbaaf2cf7b5a9339d3790610e622020c19d52f5a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure any existing hashserv connection is closedRichard Purdie2022-04-011-0/+2
| | | | | | | | | | Ensure any exiting hash server connection is terminated before we start a new bitbake session. This avoids errors seen with memory resident bitbake when the asyncio event loop isn't closed correctly. (Bitbake rev: 42ff9de77f24e2a0bec48a14b64c4b538e00b4af) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Restore sys.path and sys.modules between parsesRichard Purdie2022-04-011-0/+8
| | | | | | | | | | | When memory resident bitbake is active and we re-parse, the old module configuration is present which can lead to strange errors. Reset this when reparsing so the state is consistent. This fixes memory resident bitbake errors. (Bitbake rev: 951942c3c284ec2c62e730e145688033190af9b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Further fixes to inotify to fix memres bitbake issuesRichard Purdie2022-04-011-1/+11
| | | | | | | | | | | | The previous fix for inotify wasn't quite correct as we need to modify bbseen before calling add_filewatch(). We also need to ensure the parse mtime cache is cleared when directories are added/removed. There was also a typo in the original fix and the wrong watcher was being changed. Fix the various issues which improves memory resident bitbake testing results. (Bitbake rev: 66cadd6be58bce5f7a56556cf92efd8159fb0b0e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Rework force parser shutdownRichard Purdie2022-03-301-12/+25
| | | | | | | | | | | | | | | | | | The "force" option to parser shutdown was often the cause of lockups and there is no good reason we should have two different behaviours. Change and unify the codepaths to always: * Wait for longer for a controlled shutdown of a process (0.5s). Usually it will be much faster if it has finished so the delay doesn't really matter. * Send processes a SIGINT * Failing that, send a SIGTERM * Call .close() if available to release zombies This means we no longer need the "force" parameter to the function so it is removed. (Bitbake rev: de88c62ef873e9fce78ba162f5311d846de96f2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Fix signal handling lockupsRichard Purdie2022-03-301-2/+26
| | | | | | | | | | | | | | | | | | | | If a parser process is terminated while holding a write lock, then it will lead to a deadlock (see https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate). With SIGTERM, we don't want to terminate holding the lock. We also don't want a SIGINT to cause a partial write to the event stream. I tried using signal masks to avoid this but it doesn't work, see https://bugs.python.org/issue47139 Instead, add a signal handler and catch the calls around the critical section. We also need a thread lock to ensure other threads in the same process don't handle the signal until all the threads are not in the lock. (Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Simplify parser init function handlingRichard Purdie2022-03-301-12/+8
| | | | | | | | Not sure why this is so convoluted but we should simplify it! (Bitbake rev: 6195343c46ba9d2685fc2d42366922f88ff3f369) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve exception handling in parsing processRichard Purdie2022-03-301-24/+24
| | | | | | | | | If an exception occurs in the parsing process, ensure the cleanup is called via all codepaths using a try/finally. (Bitbake rev: e1ba2a69f1fc02f01a851bce20b1badf0b991f03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix main loop starvation when parsingRichard Purdie2022-03-301-1/+5
| | | | | | | | | | When parsing, the parser isn't servicing the main loop so a Ctrl+C in the UI won't be seen on the cooker/server side. Fix this by returning when queue timeouts occur. This helps where there is a hung or slow parsing thread. (Bitbake rev: a2cde38311a51112dca5e7bb4e7feaf4e6a281b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix exception handling in parsersRichard Purdie2022-03-301-7/+6
| | | | | | | | | | We shouldn't be generating exception inside a generator. Rearrange the code to improve the handling of this. Also fix the misconverted code from when multiconfig was added and pass the exception as "result". (Bitbake rev: ae89e23696de2f27c00ae00922933395171de5d5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix inotify watches causing memory resident bitbake corruptionRichard Purdie2022-03-261-0/+10
| | | | | | | | | | | | | | | | | | | | Thanks to great debugging from pavel@zhukoff.net we had a simpler reproducer for the corruption see in oe-selftest when using BB_SERVER_TIMEOUT=60, i.e. with bitbake in memory resident mode. This was effectively: oe-selftest -r devtool.DevtoolUpgradeTests.test_devtool_upgrade devtool.DevtoolUpgradeTests.test_devtool_upgrade_git -j 1 -K The issue is that if directories are removed (such as workspace), if they are added again, we don't have the watches in place any more. This patch adds some slightly paranoid checks to ensure we do the correct things for directory additions and removals (we track directories, not files specifically to avoid running out of watches). [YOCTO #14023] (Bitbake rev: 2c414f659d793d732041614caedd773959eb4f27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/siggen: Support exit calls and use for hashserv clientRichard Purdie2022-03-261-0/+1
| | | | | | | | | | | | | We have shutdown functions within the async client code but the siggen doesn't currently call them. We notice on python 3.10 (such as on fedora35) that at exit, there is a stray asyncio process left behind. Usually this doesn't cause problems but it could potentially be a cause of a hang. For general cleanliness and completness, add in hooks to call the exit handler. (Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix environment history printingRichard Purdie2022-03-111-0/+2
| | | | | | | | | | | | | | | | | | | | bitbake -e is not showing variable history correctly for all variables. The issue was triggered by the change to knotty to default to enabling variable tracking in the base datastore but that exposed another issue. The real problem is that calling reset() inside cooker reverts to the data tracking enabled by the UI for the base datastore, then turns off tracking. In the case of the environment printing code, it needs it to be left on. Tweak the code to ensure data store tracking really is enabled. The code here is clearly a bit of a mess but this at least fixes a clear regression until more invasive improvements can be made. (Bitbake rev: 9596bffc9903f023d1cc97ce531ddb1cf010c4d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: more verbose error messageMarta Rybczynska2022-03-081-1/+1
| | | | | | | | | | | The "Parsing halted due to errors" is shown when user has old-style variables requiring renaming. Make it clear that it isn't an additional error, but a result of earlier errors. (Bitbake rev: e81b6ae61dd8b5d0f77146263d79dfe66c66b060) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Replace "abort" usage in task handlingScott Murray2022-02-211-12/+12
| | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the use of "abort" with "halt" in code related to handling task failure. (Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix typo "isn't" and "tuples"Zygmunt Krynicki2022-02-201-2/+2
| | | | | | | (Bitbake rev: ac688b6e93faa51d48768b677685499862d3e7c4) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Improve parsing failure from handled exception usabilityRichard Purdie2022-02-171-1/+3
| | | | | | | | | | | | | When a recipe raises a BBHandledException, it means the error was already shown to the user. Adding an additional one here isn't helpful. What is helpful is to mention that parsing was halted. Tweak the code to do this with improves the messages the user sees and helps understand what happened. (Bitbake rev: fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Fix task-depends.dot for multiconfig targetsRichard Purdie2021-11-131-1/+3
| | | | | | | | | | | | The right hand side of dependencies in the task dependency file generated by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix this. [YOCTO #14621] (Bitbake rev: 1d5ca721040c5e39aefa11219f62710de6587701) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle parsing results queue raceRichard Purdie2021-11-101-5/+9
| | | | | | | | | | The previous fix introduced a race where the queue might not be empty but all the parser processes have exited. Handle this correctly to avoid occasional errors. (Bitbake rev: 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove debug code, oops :(Richard Purdie2021-11-101-1/+0
| | | | | | (Bitbake rev: 19291665fa8b6cc331290f2542af3e8e653203f1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Handle parse threads disappearing to avoid hangsRichard Purdie2021-11-081-1/+10
| | | | | | | | | | If one of the parse threads disappears during parsing for some reason, bitbake currently hangs. Avoid this (and zombie threads hanging around) by joining() threads which have exited. (Bitbake rev: dc86a533d951d13643ce446533370da804782afc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: check if upstream hash equivalence server is availableJose Quaresma2021-11-051-1/+11
| | | | | | | | | | When the user specify an invalid upstream hash equivalence server in BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server. (Bitbake rev: be45aeb9a84f30c28711e87e2d2a4a86320a8d94) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-031-3/+3
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/command: Add a dummy event for tinfoil testingRichard Purdie2021-09-261-0/+5
| | | | | | | | | | We need a command genetating an event to test through the tinfoil API. The current test has IO load issues so add a dummy version which won't have the IO constraints. (Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Allow upstream for local hash equivalence serverJoshua Watt2021-09-111-1/+6
| | | | | | | | | | | | | | The hash equivalence server has had the option to support a read-only upstream server for some time now when launched as a standalone program, but there was no way to set the upstream when using a locally started server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used to specify an upstream server when a local hash equivalence server is used (e.g. BB_HASHSERVE is "auto") (Bitbake rev: 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Fix typos in exiting messageMartin Jansa2021-09-011-1/+1
| | | | | | | (Bitbake rev: 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"Martin Jansa2021-09-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock (Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: asyncrpc: Catch early SIGTERMJoshua Watt2021-07-291-2/+1
| | | | | | | | | | | | | | | | | If the SIGTERM signal is sent to an asyncrpc server before it has installed the SIGTERM handler in the main loop, it may miss the signal which will can cause the calling process to wait forever on the join(). To resolve this, the calling process should mask of SIGTERM before forking the server process and the server should unmask the signal only after the handler is installed. To simplify the usage of the server, an new helper function called serve_as_process() is added to do this automatically and correctly. Thanks: Scott Murray <scott.murray@konsulko.com> for helping debug (Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Add REQUIRED_VERSION checksCharlie Davies2021-03-151-2/+8
| | | | | | | | | | | | | Add logic to check that if REQUIRED_VERSION has been set that the cooker class method findBestProviders properly handles the case where the REQUIRED_VERSION has not been found. Fixes [YOCTO #10096] (Bitbake rev: 5df201d746f26154213e6d15d2721cd35b38b59e) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: lib: add Required Version to Bitbake --show-versions commandCharlie Davies2021-03-151-8/+16
| | | | | | | | | | | | | This commit adds a new column to the Bitbake --show-versions command called Required Version. This column will display any packages which have a REQUIRED_VERSION successfully set. Fixes [YOCTO #10096] (Bitbake rev: 90c7d1815e41243323d32b9dbb865757a922578a) Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Force parser shutdown after catching an exceptionJan Brzezanski2021-02-211-5/+5
| | | | | | | | | | | | | | | | Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing parser shutdown as default in case of build abort. In this case bitbake sometimes hangs after facing error during parsing, waiting for child processes to finish. Killing it then will spawn zombie processes. Thus we force the shutdown after catching an exception. (Bitbake rev: 915330e1dbae1ee8fd9a0358decf2c294f771961) Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-08-04 11:55:35 +0000