diff options
| -rw-r--r-- | documentation/poky-ref-manual/technical-details.xml | 132 |
1 files changed, 64 insertions, 68 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml index af62ee4f35..19ce9ab224 100644 --- a/documentation/poky-ref-manual/technical-details.xml +++ b/documentation/poky-ref-manual/technical-details.xml | |||
| @@ -117,7 +117,7 @@ | |||
| 117 | The term "package" can also be used to describe recipes. | 117 | The term "package" can also be used to describe recipes. |
| 118 | However, since the same word is used for the packaged output from the Yocto | 118 | However, since the same word is used for the packaged output from the Yocto |
| 119 | Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), | 119 | Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), |
| 120 | this document avoids using the term "package" when refering to recipes. | 120 | this document avoids using the term "package" when referring to recipes. |
| 121 | </para> | 121 | </para> |
| 122 | </section> | 122 | </section> |
| 123 | 123 | ||
| @@ -165,7 +165,7 @@ | |||
| 165 | As mentioned in the previous paragraph, building from scratch ensures that | 165 | As mentioned in the previous paragraph, building from scratch ensures that |
| 166 | everything is current and starts from a known state. | 166 | everything is current and starts from a known state. |
| 167 | However, building from scratch also takes much longer as it generally means | 167 | However, building from scratch also takes much longer as it generally means |
| 168 | rebuiding things that don't necessarily need rebuilt. | 168 | rebuilding things that don't necessarily need rebuilt. |
| 169 | </para> | 169 | </para> |
| 170 | 170 | ||
| 171 | <para> | 171 | <para> |
| @@ -241,7 +241,7 @@ | |||
| 241 | affect the output for target packages. | 241 | affect the output for target packages. |
| 242 | Also, the build process has the objective of making native/cross packages relocatable. | 242 | Also, the build process has the objective of making native/cross packages relocatable. |
| 243 | The checksum therefore needs to exclude <filename>WORKDIR</filename>. | 243 | The checksum therefore needs to exclude <filename>WORKDIR</filename>. |
| 244 | The simplistic approach for excluding the worknig directory is to set | 244 | The simplistic approach for excluding the working directory is to set |
| 245 | <filename>WORKDIR</filename> to some fixed value and create the checksum | 245 | <filename>WORKDIR</filename> to some fixed value and create the checksum |
| 246 | for the "run" script. | 246 | for the "run" script. |
| 247 | </para> | 247 | </para> |
| @@ -301,77 +301,73 @@ | |||
| 301 | 301 | ||
| 302 | <para> | 302 | <para> |
| 303 | Thus far, this section has limited discussion to the direct inputs into a task. | 303 | Thus far, this section has limited discussion to the direct inputs into a task. |
| 304 | Information based on direct inputs is referred to as the "basehash" in the code. | 304 | Information based on direct inputs is referred to as the "basehash" in the |
| 305 | However, there is still the question of a task's indirect inputs, the things that | 305 | code. |
| 306 | were already built and present in the build directory. | 306 | However, there is still the question of a task's indirect inputs - the |
| 307 | The checksum (or signature) for a particular task needs to add the hashes of all the | 307 | things that were already built and present in the build directory. |
| 308 | tasks on which the particular task depends. | 308 | The checksum (or signature) for a particular task needs to add the hashes |
| 309 | Choosing which dependencies to add is a policy decision. | 309 | of all the tasks on which the particular task depends. |
| 310 | However, the effect is to generate a master checksum that combines the | 310 | Choosing which dependencies to add is a policy decision. |
| 311 | basehash and the hashes of the task's dependencies. | 311 | However, the effect is to generate a master checksum that combines the basehash |
| 312 | </para> | 312 | and the hashes of the task's dependencies. |
| 313 | 313 | </para> | |
| 314 | <para> | 314 | |
| 315 | While figuring out the dependencies and creating these checksums is good, | 315 | <para> |
| 316 | what does the Yocto Project build system do with the checksum information? | 316 | At the code level, there are a variety of ways both the basehash and the |
| 317 | The build system uses a signature handler that is responsible for | 317 | dependent task hashes can be influenced. |
| 318 | processing the checksum information. | 318 | Within the BitBake configuration file, we can give BitBake some extra information |
| 319 | By default, there is a dummy "noop" signature handler enabled in BitBake. | 319 | to help it construct the basehash. |
| 320 | This means that behaviour is unchanged from previous versions. | 320 | The following statements effectively result in a list of global variable |
| 321 | OECore uses the "basic" signature handler through this setting in the | 321 | dependency excludes - variables never included in any checksum: |
| 322 | <filename>bitbake.conf</filename> file: | ||
| 323 | <literallayout class='monospaced'> | 322 | <literallayout class='monospaced'> |
| 324 | BB_SIGNATURE_HANDLER ?= "basic" | 323 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" |
| 324 | BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS" | ||
| 325 | BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER" | ||
| 326 | BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET" | ||
| 325 | </literallayout> | 327 | </literallayout> |
| 326 | Also within the BitBake configuration file, we can give BitBake | 328 | The previous example actually excludes |
| 327 | some extra information to help it handle this information. | 329 | <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link> |
| 328 | The following statements effectively result in a list of global | 330 | since it is actually constructed as a path within |
| 329 | variable dependency excludes - variables never included in | 331 | <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on |
| 330 | any checksum: | 332 | the whitelist. |
| 333 | </para> | ||
| 334 | |||
| 335 | <para> | ||
| 336 | The rules for deciding which hashes of dependent tasks to include through | ||
| 337 | dependency chains are more complex and are generally accomplished with a | ||
| 338 | python function. | ||
| 339 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples | ||
| 340 | of this and also illustrates how you can insert your own policy into the system | ||
| 341 | if so desired. | ||
| 342 | This file defines the two basic signature generators <filename>OE-Core</filename> | ||
| 343 | uses: "OEBasic" and "OEBasicHash". | ||
| 344 | By default, there is a dummy "noop" signature handler enabled in BitBake. | ||
| 345 | This means that behavior is unchanged from previous versions. | ||
| 346 | <filename>OE-Core</filename> uses the "OEBasic" signature handler by default | ||
| 347 | through this setting in the <filename>bitbake.conf</filename> file: | ||
| 331 | <literallayout class='monospaced'> | 348 | <literallayout class='monospaced'> |
| 332 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" | 349 | BB_SIGNATURE_HANDLER ?= "OEBasic" |
| 333 | BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS" | ||
| 334 | BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER" | ||
| 335 | BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET" | ||
| 336 | BB_HASHTASK_WHITELIST += "(.*-cross$|.*-native$|.*-cross-initial$| \ | ||
| 337 | .*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)" | ||
| 338 | </literallayout> | 350 | </literallayout> |
| 339 | This example is actually where <filename>WORKDIR</filename> | 351 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the |
| 340 | is excluded since <filename>WORKDIR</filename> is constructed as a | 352 | "OEBasic" version but adds the task hash to the stamp files. |
| 341 | path within <filename>TMPDIR</filename>, which is on the whitelist. | 353 | This results in any metadata change that changes the task hash, automatically |
| 354 | causing the task to be run again. | ||
| 355 | This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link> | ||
| 356 | values and changes to metadata automatically ripple across the build. | ||
| 357 | Currently, this behavior is not the default behavior for <filename>OE-Core</filename> | ||
| 358 | but is the default in <filename>poky</filename>. | ||
| 342 | </para> | 359 | </para> |
| 343 | 360 | ||
| 344 | <para> | 361 | <para> |
| 345 | The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and | 362 | It is also worth noting that the end result of these signature generators is to |
| 346 | excludes certain kinds of tasks from the dependency chains. | 363 | make some dependency and hash information available to the build. |
| 347 | The effect of the previous example is to isolate the native, target, | 364 | This information includes: |
| 348 | and cross-components. | ||
| 349 | So, for example, toolchain changes do not force a rebuild of the whole system. | ||
| 350 | </para> | ||
| 351 | |||
| 352 | <para> | ||
| 353 | The end result of the "basic" handler is to make some dependency and | ||
| 354 | hash information available to the build. | ||
| 355 | This includes: | ||
| 356 | <literallayout class='monospaced'> | 365 | <literallayout class='monospaced'> |
| 357 | BB_BASEHASH_task-<taskname> - the base hashes for each task in the recipe | 366 | BB_BASEHASH_task-<taskname> - the base hashes for each task in the recipe |
| 358 | BB_BASEHASH_<filename:taskname> - the base hashes for each dependent task | 367 | BB_BASEHASH_<filename:taskname> - the base hashes for each dependent task |
| 359 | BBHASHDEPS_<filename:taskname> - The task dependencies for each task | 368 | BBHASHDEPS_<filename:taskname> - The task dependencies for each task |
| 360 | BB_TASKHASH - the hash of the currently running task | 369 | BB_TASKHASH - the hash of the currently running task |
| 361 | </literallayout> | 370 | </literallayout> |
| 362 | There is also a "basichash" <filename>BB_SIGNATURE_HANDLER</filename>, | ||
| 363 | which is the same as the basic version but adds the task hash to the stamp files. | ||
| 364 | This results in any metadata change that changes the task hash, | ||
| 365 | automatically causing the task to be run again. | ||
| 366 | This removes the need to bump <filename>PR</filename> | ||
| 367 | values and changes to metadata automatically ripple across the build. | ||
| 368 | Currently, this behavior is not the default behavior. | ||
| 369 | However, it is likely that the Yocto Project team will go forward with this | ||
| 370 | behavior in the future since all the functionality exists. | ||
| 371 | The reason for the delay is the potential impact to the distribution feed | ||
| 372 | creation as they need increasing <filename>PR</filename> fields | ||
| 373 | and the Yocto Project currently lacks a mechanism to automate incrementing | ||
| 374 | this field. | ||
| 375 | </para> | 371 | </para> |
| 376 | </section> | 372 | </section> |
| 377 | 373 | ||
| @@ -433,7 +429,7 @@ | |||
| 433 | <literallayout class='monospaced'> | 429 | <literallayout class='monospaced'> |
| 434 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | 430 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" |
| 435 | </literallayout> | 431 | </literallayout> |
| 436 | This method, as well as the following example, also works for mutliple directories. | 432 | This method, as well as the following example, also works for multiple directories. |
| 437 | <literallayout class='monospaced'> | 433 | <literallayout class='monospaced'> |
| 438 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | 434 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" |
| 439 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | 435 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" |
| @@ -555,7 +551,7 @@ | |||
| 555 | Once you are aware of such a change, you can take steps to invalidate the cache | 551 | Once you are aware of such a change, you can take steps to invalidate the cache |
| 556 | and force the task to run. | 552 | and force the task to run. |
| 557 | The step to take is as simple as changing a function's comments in the source code. | 553 | The step to take is as simple as changing a function's comments in the source code. |
| 558 | For example, to invalidate package shared state files, change the comment statments | 554 | For example, to invalidate package shared state files, change the comment statements |
| 559 | of <filename>do_package</filename> or the comments of one of the functions it calls. | 555 | of <filename>do_package</filename> or the comments of one of the functions it calls. |
| 560 | The change is purely cosmetic, but it causes the checksum to be recalculated and | 556 | The change is purely cosmetic, but it causes the checksum to be recalculated and |
| 561 | forces the task to be run again. | 557 | forces the task to be run again. |
| @@ -576,7 +572,7 @@ | |||
| 576 | <para> | 572 | <para> |
| 577 | This section describes the mechanism by which the Yocto Project build system | 573 | This section describes the mechanism by which the Yocto Project build system |
| 578 | tracks changes to licensing text. | 574 | tracks changes to licensing text. |
| 579 | The section also describes how to enable commercially licensed receipes, | 575 | The section also describes how to enable commercially licensed recipes, |
| 580 | which by default are disabled. | 576 | which by default are disabled. |
| 581 | </para> | 577 | </para> |
| 582 | 578 | ||
