summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/runqueue.py
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: runqueue: Ensure deferred tasks are sorted by multiconfigRichard Purdie2022-08-081-5/+8
| | | | | | | | | | | | | We have to prefer one multiconfig over another when deferring tasks, else we'll have cross-linked build trees and nothing will be able to build. In the original population code, we sort like this but we don't after rehashing. Ensure we have the same sorting after rehashing toa void deadlocks. (Bitbake rev: 27228c7f026acb8ae9e1211d0486ffb7338123a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: add cpu/io pressure regulationAryaman Gupta2022-07-281-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prevent the scheduler from starting new tasks if the current cpu or io pressure is above a certain threshold and there is at least one active task. This threshold can be specified through the "BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf. The threshold represents the difference in "total" pressure from the previous second. The pressure data is discussed in this oe-core commit: 061931520b buildstats.py: enable collection of /proc/pressure data where one can see that the average and "total" values are available. From tests, it was seen that while using the averaged data was somewhat useful, the latency in regulating builds was too high. By taking the difference between the current pressure and the pressure seen in the previous second, better regulation occurs. Using a shorter time period is appealing but due to fluctations in pressure, comparing the current pressure to 1 second ago achieves a reasonable compromise. One can look at the buildstats logs, that usually sample once per second, to decide a sensible threshold. If the thresholds aren't specified, pressure is not monitored and hence there is no impact on build times. Arbitary lower limit of 1.0 results in a fatal error to avoid extremely long builds. If the limits are higher than 1,000,000, then warnings are issued to inform users that the specified limit is very high and unlikely to result in any regulation. The current bitbake scheduling algorithm requires that at least one task be active. This means that if high pressure is seen, then new tasks will not be started and pressure will be checked only for as long as at least one task is active. When there are no active tasks, an additional task will be started and pressure checking resumed. This behaviour means that if an external source is causing the pressure to exceed the threshold, bitbake will continue to make some progress towards the requested target. This violates the intent of limiting pressure but, given the current scheduling algorithm as described above, there seems to be no other option. In the case where only one bitbake build is running, the implications of the scheduler requirement will likely result in pressure being higher than the threshold. More work would be required to ensure that the pressure threshold is never exceeded, for example by adding pressure monitoring to make and ninja. (Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy Macleod <randy.macleod@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix unihash cache mismatch issuesRichard Purdie2022-06-021-0/+3
| | | | | | | | | | | | | | | Very occasionally we see errors in eSDK testing on the autobuilder where the task hashes in the eSDK don't match what was just built. I was able to inspect one of these build directories and noticed that the bb_unihashes.dat file in the eSDK was zero sized. Whilst inspecting the code to understand the cause, I noticed that updated hashes are not saved out in subsequent updates of the values in the rehash process. Add a missing sync call to ensure this happens. (Bitbake rev: 7912dabbcf444a3c3d971cca4a944a8b931e301b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sig file location when using multiconfigRichard Purdie2022-04-301-1/+1
| | | | | | | | | | | | We're using the wrong data store when trying to locate siginfo files, fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting. [YOCTO #14774] (Bitbake rev: 0ed800e19a3197f8e622c8d3b630aae384e60aba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop pointless variable assignmentRichard Purdie2022-04-211-1/+0
| | | | | | | | | This is set at the start of the loop anyway so it does nothing. Drop the pointless code. (Bitbake rev: e6a3173c9cdf349ccbd4cf612868f92cce8717c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Replace "abort" usage in task handlingScott Murray2022-02-211-6/+6
| | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language replace the use of "abort" with "halt" in code related to handling task failure. (Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename allowed multiple provider variableScott Murray2022-02-211-2/+2
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED (Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd) (Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename setscene enforce filtering variableScott Murray2022-02-211-17/+17
| | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS (Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c) (Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Drop BB_STAMP_POLICY/BB_STAMP_WHITELISTRichard Purdie2022-01-121-22/+1
| | | | | | | | | | | | The different stamp policies were poor versions of the siggen code and task hashes, predating it and being used by packaged staging. They had many limitations, hence their replacement. I'm not aware of any users of that code any more so I believe it and the assoicated stamp whitelist variable can simply be removed. (Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option handlingRichard Purdie2021-11-081-0/+1
| | | | | | | | | | The previous fix for runall option handling had a small bug in it, it didn't clear the originally processed task list which meant it was running too many tasks. Fix this so the list is reset and rebuild correctly. (Bitbake rev: 87c9e120897ed04dfc64d4752fc602f9bfcb8645) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix runall option task deletion ordering issueRichard Purdie2021-11-051-19/+16
| | | | | | | | | | | | | | | | The runbuild option handling in runqueue was flawed as items deleted from the main task list may be dependencies and hence cause index errors. Rather than modify runtaskentries straight away, compute a new shorted list and use that as an input to the second phase. This avoids the need to add tasks back to the list meaning delcount can be simplifed to a simple counter. The second use case in runonly doen't re-add items so doesn't have this issue. (Bitbake rev: 3428e3c54eb5cc03ff96f9cee6dc839afee7a419) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Fix string concatination potential performance issuesRichard Purdie2021-11-031-18/+19
| | | | | | | | | | | | | | | | Python scales badly when concatinating strings in loops. Most of these references aren't problematic but at least one (in data.py) is probably a performance issue as the issue is compounded as strings become large. The way to handle this in python is to create lists which don't reconstruct all the objects when appending to them. We may as well fix all the references since it stops them being copy/pasted into something problematic in the future. This patch was based on issues highligthted by a report from AWS Codeguru. (Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Clean up use of len()Richard Purdie2021-11-031-33/+33
| | | | | | (Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/knotty: Improve UI handling of setscene task countingRichard Purdie2021-09-171-3/+18
| | | | | | | | | | | | | | | | The recent fixes to merge setscene and normal task accounting in runqueue fixed some display issues but broke the task numbering of setscene tasks. Add new accounting methods to the stats structure specifically designed for setscene. This accounts for the fact that setscene tasks can rerun multiple times in the build. Then use the new data in the UI to correctly display the numbers the user wants to see to understand progress. (Bitbake rev: ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up task stats handlingRichard Purdie2021-09-111-15/+7
| | | | | | | | | | | | | | | | | | | | | When we parallelised normal and setscene tasks, the task stats handling was left separate pending further thought. We had to remove handling of the setscene tasks from the UI in order to maintain consistent task numbering. Currently, "0 of 0" tasks can be shown as setscene tasks execute until the first normal task runs. The only use left for sq_stats is in the active task numbers which we can use the length of sq_ive for instead. We can therefore drop it and return stats in all cases. This removes the "0 of 0" task problem since the stats in all normal and setscene tasks matches. [YOCTO #14479] (Bitbake rev: eae6e947e37e18cded053814bd2a268b44fb25cd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix issues with multiconfig deferred task deadlock messagesRichard Purdie2021-09-081-0/+6
| | | | | | | | | | | | | | | | | | | | | In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] (Bitbake rev: ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid deadlock avoidance task graph corruptionRichard Purdie2021-09-081-1/+2
| | | | | | | | | | If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. (Bitbake rev: 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve multiconfig deferred task issuesRichard Purdie2021-08-051-26/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. (Bitbake rev: bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Handle deferred task rehashing in multiconfig buildsRichard Purdie2021-05-011-4/+10
| | | | | | | | | | | | If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issueRichard Purdie2021-05-011-4/+4
| | | | | | | | | | | | | | | | | | | | | We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix deferred task issuesRichard Purdie2021-04-181-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Further fixes for confused setscene tasksRichard Purdie2021-04-061-13/+26
| | | | | | | | | | | | | | | There is further evidence of tasks ending up being "covered" and "notcovered" which shouldn't happen and is bad. The code that caused this problem last time appears to have issues where stamps for tasks already exist. Split out the setscene stamp checking code to a separate function and use this when checking "hard dependencies" (like pseudo-native) so that if the stamps exist and it will be "covered", it is not put on the notcovered list. (Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/event: Add an event for notifying of stale setscene tasksRichard Purdie2021-03-231-0/+14
| | | | | | | | | | | Use the new functionality in build.py to identify stale setscene tasks and send an event to the metadata listing them. The metadata then has the option of performing cleanup operations if it thinks that appropriate. (Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Print pseudo.log if fakeroot task failedTomasz Dziendzielski2021-03-111-9/+37
| | | | | | | | | | | | | | | | | | | | | | | | Currently if pseudo fails we can only see the path to pseudo.log. If we have no access to server and can only rely on bitbake log then debugging becomes impossible. This printing needs to be added in runqueue level, not inside task execution, because in some cases task fails with pseudo abort really early and we don't even see any log. In this change I'm adding pseudo log printing in every fakeroot task failure that logged `mismatch`, `error` or `fatal` to logfile, because we have no other way to communicate with pseudo if it failed or not. Only lines from last pseudo server execution will be printed. (Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com> Signed-off-by: Adrian Walag Signed-off-by: Paulo Neves <ptsneves@gmail.com> Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com> Signed-off-by: Wiktor Baura <wbaura@gmail.com> Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add setscene task overlap sanity checkRichard Purdie2021-03-091-0/+4
| | | | | | | | | | We've seen hard to debug issues where a task ends up in both the covered and notcovered list. Add a sanity check to ensure if this happens in future, we see it in the logs. (Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task execution corruption issueRichard Purdie2021-03-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | We've seen occasional issues where linux-yocto:do_compile_kernelmodules would run without do_shared_workdir running before it. do_shared_workdir is an setscene task but never has an sstate object generated so it will always rerun. This should not happen since compile_kernemodules should only execute if a setscene that depends on it didn't run and that should trigger do_shared_workdir not to be marked as covered. The issue is that build-appliance-image:do_package is one of the tasks which covers linux-yocto:do_compile_kernelmodules but it is also a noexec task and has a dependecy on pseudo-native:do_populate_sysroot. In the problem case, pseudo-native:do_populate_sysroot is unavailable but marked as covered since it is noexec. The "harddeps" code then also marks it as notcovered. No task should ever be both covered and notcovered and this is where the problems come from. The solution is for the harddeps code only to to fail tasks if they've not already been handled in some way. The code is assuming code couldn't have handled revdeps at this point but we now have clear evidence they can. (Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker/runqueue: Add support for BB_DEFAULT_UMASKRichard Purdie2021-02-161-0/+1
| | | | | | | | | | | | | Currently each task has to have a umask specified individually. This is leading to determinism issues since it is easy to miss specifying this for an extra task. Add support for specifing the default task umask globally which simplifies the problem. (Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Prevent bitbake from executing event handler for wrong ↵Tomasz Dziendzielski2021-02-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | multiconfig target When multiconfig is used bitbake might try to run events that don't exist for specific mc target. In cooker.py we pass `self.databuilder.mcdata[mc]` data that contains names of events' handlers per mc target, but fire_class_handlers uses global _handlers variable that is created during parsing of all the targets. This leads to a problem where bitbake runs event handler that don't exist for a target or even overrides them - if multiple targets use event handler with the same name but different code then only one version will be executed for all targets. See [YOCTO #13071] for detailed bug information. Add mc target name as a prefix to event handler name so there won't be two different handlers with the same name. Add internal __BBHANDLERS_MC variable to have the handlers lists per machine. (Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-32/+32
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig ↵Tomasz Dziendzielski2021-02-061-3/+3
| | | | | | | | | | | | | | | | target When we run `devtool build mc` recipe's task dependencies are expanded to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig and "do_package_sysroot" as multiconfigname. | ERROR: Multiconfig dependency mc:do_populate_sysroot depends on | nonexistent multiconfig configuration named do_populate_sysroot (Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Don't use sys.argvRichard Purdie2020-08-251-5/+4
| | | | | | | | | | | | We should not be using sys.argv inside the server to decide which targets the user added on the commandline. There might not even be a commandline. Thankfully the targets variable is easily accessible in this context and contains this exact data we want. (Bitbake rev: 5b12bf30bccdd00262e74964223220c649040be4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/msg: Cleanup verbose option handlingRichard Purdie2020-08-251-2/+2
| | | | | | | | | | | | The levels of indirection to set these verbose logging options is rather crazy. This attempts to turn things into two specific options with much more specific meanings. For now its all still controlled by the commandline verbose option and should funciton as previously, with the addition that the BB_VERBOSE_LOGS option can now be task specific. (Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid unpickle errors in rare casesRichard Purdie2020-06-281-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | In rare cases the pickled data from a task contains "</event>" which causes backtrace. This can be reproduced with something like: do_unpack_prepend () { bb.warn("</event>") } There are several solutions but the easiest is to catch this exception and look for the next marker instead as this should be the only way such an unpickle error could occur. This fixes rare exceptions seen on the autobuilder. Also add in other potential exceptions listed in the pickle manual page so that better debug is obtained should there be an error in this code path in future. exitcode doesn't need the same handling since we control what is in that data field and it could never contain </exitcode> (Bitbake rev: 5ada512d6f9cbbdf1172ff7818117c38b12225ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen: Pass all data caches to hash functionsJoshua Watt2020-06-121-3/+5
| | | | | | | | | | | | | | | | | | | Passing all the data caches to the task hashing functions allows them to correctly account for mcdepends in task signatures. This allows tasks to be correctly re-run when a mcdepends changes. By default, the legacy behavior is maintained for derived signature generators by passing a special proxy object that can either be used to access all multiconfigs or the legacy behavior. If a derived signature generator is updated, it can set the supports_multiconfig_datacaces property to instruct bitbake it deals with multiconfigs properly. [YOCTO #13724] (Bitbake rev: 8ff9203de4fce9c104c2987d86980c9f34036b97) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: command: Move split_mc_pn to runqueueJoshua Watt2020-06-101-0/+6
| | | | | | | | | | | | | All of the other multiconfig splitting functions are located in runqueue so move the function to split a pn/fn there also so that its easier to see them all together. Fixes a case where the findBestProvider() command wasn't working for multiconfig because it was looking for a prefix of "multiconfig:" instead of the newer "mc:" (Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Split file collections per multiconfigJoshua Watt2020-06-101-5/+6
| | | | | | | | | | | | Splits the cooker to track a collection per multiconfig instead of a single collection for all multiconfigs. Practically speaking, this allows each multiconfigs to each have different BBMASKs that apply to it instead of each one using the mask specified in the base configuration. (Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Lower setscene complete logging levelJoshua Watt2020-03-131-1/+1
| | | | | | | | | | | | | | | Lowers the level of the log message when setscene tasks have completed. This message can occur multiple times when hash equivalence is enabled, since the runqueue switches between executing setscene tasks and normal tasks. Since this is primarily of use when debugging hash equivalence, use the hash equivalence logger at VERBOSE level. [YOCTO #13813] (Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Lower hash equivalence loggingJoshua Watt2020-03-131-3/+3
| | | | | | | | | | | | | | Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] (Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Log hash equivalence with a different loggerJoshua Watt2020-03-131-5/+6
| | | | | | | | | | Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging. (Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/msg: Use log level instead of debug countJoshua Watt2020-03-131-1/+1
| | | | | | | | | | | | | Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used (Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: teach runonly/runall to accept "do_task" as well as "task"Chris Laplante2020-03-071-2/+6
| | | | | | | | | | | | | | | Previously --runonly=do_task would give a misleading error like: ERROR: Could not find any tasks with the tasknames ['do_task'] to run within the recipes of the taskgraphs of the targets... The problem is that BitBake tried to find "do_do_task". So teach it to only add the do_ prefix if it's not already there. (Bitbake rev: 694904bde980606dc67c201da61f4fb685679b17) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-3/+1
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Use a set for the setscene tasks listRichard Purdie2020-01-131-2/+2
| | | | | | | | | | This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). (Bitbake rev: f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task dependency corner case in sanity testRichard Purdie2020-01-021-1/+4
| | | | | | | | | | | A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. (Bitbake rev: d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure task dependencies are run correctlyRichard Purdie2020-01-021-0/+6
| | | | | | | | | | | | | | | | | | | | | We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. (Bitbake rev: 7a92b7f58ab187eddfe550bd6fb687240c7b11bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix equiv hash handling build failuresRichard Purdie2020-01-021-1/+2
| | | | | | | | | | | | | Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. (Bitbake rev: 40928f6991436cf687821015324483b205abfcb1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only call into the migrations function if migrations activeRichard Purdie2019-12-161-1/+2
| | | | | | | | | This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. (Bitbake rev: d446fa89d206fbc6d098215163c968ea5a8cf4a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task filteringRichard Purdie2019-12-161-3/+2
| | | | | | | | | We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. (Bitbake rev: 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise out pointless loop iterationRichard Purdie2019-12-161-9/+3
| | | | | | (Bitbake rev: 105d1f0748edde7753a4063e6fdc758ffc8a8a9e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task migration code slightlyRichard Purdie2019-12-161-8/+9
| | | | | | | | | | | | | Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. (Bitbake rev: e28ec69356f1797de3e4e3fca0fef710bc4564de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-19 12:06:59 +0000