summaryrefslogtreecommitdiffstats
path: root/documentation/ref-manual/ref-bitbake.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/ref-manual/ref-bitbake.xml')
-rw-r--r--documentation/ref-manual/ref-bitbake.xml472
1 files changed, 472 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
new file mode 100644
index 0000000..28496de
--- /dev/null
+++ b/documentation/ref-manual/ref-bitbake.xml
@@ -0,0 +1,472 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='ref-bitbake'>
6
7 <title>BitBake</title>
8
9 <para>
10 BitBake is a program written in Python that interprets the
11 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> used by
12 the OpenEmbedded build system.
13 At some point, developers wonder what actually happens when you enter:
14 <literallayout class='monospaced'>
15 $ bitbake core-image-sato
16 </literallayout>
17 </para>
18
19 <para>
20 This chapter provides an overview of what happens behind the scenes from BitBake's perspective.
21 </para>
22
23 <note>
24 BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships.
25 As such, it has no real knowledge of what the tasks being executed actually do.
26 BitBake just considers a list of tasks with dependencies and handles
27 <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>
28 consisting of variables in a certain format that get passed to the tasks.
29 </note>
30
31 <section id='ref-bitbake-parsing'>
32 <title>Parsing</title>
33
34 <para>
35 BitBake parses configuration files, classes, and <filename>.bb</filename> files.
36 </para>
37
38 <para>
39 The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
40 This file resides in the
41 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>
42 within the <filename>meta/conf/</filename> directory.
43 BitBake finds it by examining its
44 <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment
45 variable and looking for the <filename>meta/conf/</filename>
46 directory.
47 </para>
48
49 <para>
50 The <filename>bitbake.conf</filename> file lists other configuration
51 files to include from a <filename>conf/</filename>
52 directory below the directories listed in <filename>BBPATH</filename>.
53 In general, the most important configuration file from a user's perspective
54 is <filename>local.conf</filename>, which contains a user's customized
55 settings for the OpenEmbedded build environment.
56 Other notable configuration files are the distribution
57 configuration file (set by the
58 <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable)
59 and the machine configuration file
60 (set by the
61 <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable).
62 The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment
63 variables are both usually set in
64 the <filename>local.conf</filename> file.
65 Valid distribution
66 configuration files are available in the <filename>meta/conf/distro/</filename> directory
67 and valid machine configuration
68 files in the <filename>meta/conf/machine/</filename> directory.
69 Within the <filename>meta/conf/machine/include/</filename>
70 directory are various <filename>tune-*.inc</filename> configuration files that provide common
71 "tuning" settings specific to and shared between particular architectures and machines.
72 </para>
73
74 <para>
75 After the parsing of the configuration files, some standard classes are included.
76 The <filename>base.bbclass</filename> file is always included.
77 Other classes that are specified in the configuration using the
78 <filename><link linkend='var-INHERIT'>INHERIT</link></filename>
79 variable are also included.
80 Class files are searched for in a <filename>classes</filename> subdirectory
81 under the paths in <filename>BBPATH</filename> in the same way as
82 configuration files.
83 </para>
84
85 <para>
86 After classes are included, the variable
87 <filename><link linkend='var-BBFILES'>BBFILES</link></filename>
88 is set, usually in
89 <filename>local.conf</filename>, and defines the list of places to search for
90 <filename>.bb</filename> files.
91 By default, the <filename>BBFILES</filename> variable specifies the
92 <filename>meta/recipes-*/</filename> directory within Poky.
93 Adding extra content to <filename>BBFILES</filename> is best achieved through the use of
94 BitBake layers as described in the
95 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
96 Creating Layers</ulink>" section of the Yocto Project Development Manual.
97 </para>
98
99 <para>
100 BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and
101 stores the values of various variables.
102 In summary, for each <filename>.bb</filename>
103 file the configuration plus the base class of variables are set, followed
104 by the data in the <filename>.bb</filename> file
105 itself, followed by any inherit commands that
106 <filename>.bb</filename> file might contain.
107 </para>
108
109 <para>
110 Because parsing <filename>.bb</filename> files is a time
111 consuming process, a cache is kept to speed up subsequent parsing.
112 This cache is invalid if the timestamp of the <filename>.bb</filename>
113 file itself changes, or if the timestamps of any of the include,
114 configuration files or class files on which the
115 <filename>.bb</filename> file depends change.
116 </para>
117
118 <note>
119 <para>
120 You need to be aware of how BitBake parses curly braces.
121 If a recipe uses a closing curly brace within the function and
122 the character has no leading spaces, BitBake produces a parsing
123 error.
124 If you use a pair of curly brace in a shell function, the
125 closing curly brace must not be located at the start of the line
126 without leading spaces.
127 </para>
128
129 <para>
130 Here is an example that causes BitBake to produce a parsing
131 error:
132 <literallayout class='monospaced'>
133 fakeroot create_shar() {
134 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
135 usage()
136 {
137 echo "test"
138 ###### The following "}" at the start of the line causes a parsing error ######
139 }
140 EOF
141 }
142 </literallayout>
143 Writing the recipe this way avoids the error:
144 <literallayout class='monospaced'>
145 fakeroot create_shar() {
146 cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
147 usage()
148 {
149 echo "test"
150 ######The following "}" with a leading space at the start of the line avoids the error ######
151 }
152 EOF
153 }
154 </literallayout>
155 </para>
156 </note>
157 </section>
158
159 <section id='ref-bitbake-providers'>
160 <title>Preferences and Providers</title>
161
162 <para>
163 Once all the <filename>.bb</filename> files have been
164 parsed, BitBake starts to build the target (<filename>core-image-sato</filename>
165 in the previous section's example) and looks for providers of that target.
166 Once a provider is selected, BitBake resolves all the dependencies for
167 the target.
168 In the case of <filename>core-image-sato</filename>, it would lead to
169 <filename>packagegroup-core-x11-sato</filename>,
170 which in turn leads to recipes like <filename>matchbox-terminal</filename>,
171 <filename>pcmanfm</filename> and <filename>gthumb</filename>.
172 These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
173 </para>
174
175 <para>
176 Sometimes a target might have multiple providers.
177 A common example is "virtual/kernel", which is provided by each kernel package.
178 Each machine often selects the best kernel provider by using a line similar to the
179 following in the machine configuration file:
180 </para>
181
182 <literallayout class='monospaced'>
183 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
184 </literallayout>
185
186 <para>
187 The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename>
188 is the provider with the same name as the target.
189 </para>
190
191 <para>
192 Understanding how providers are chosen is made complicated by the fact
193 that multiple versions might exist.
194 BitBake defaults to the highest version of a provider.
195 Version comparisons are made using the same method as Debian.
196 You can use the
197 <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
198 variable to specify a particular version (usually in the distro configuration).
199 You can influence the order by using the
200 <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename>
201 variable.
202 By default, files have a preference of "0".
203 Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
204 package unlikely to be used unless it is explicitly referenced.
205 Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
206 <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
207 <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
208 versions until they have undergone sufficient testing to be considered stable.
209 </para>
210
211 <para>
212 In summary, BitBake has created a list of providers, which is prioritized, for each target.
213 </para>
214 </section>
215
216 <section id='ref-bitbake-dependencies'>
217 <title>Dependencies</title>
218
219 <para>
220 Each target BitBake builds consists of multiple tasks such as
221 <filename>fetch</filename>, <filename>unpack</filename>,
222 <filename>patch</filename>, <filename>configure</filename>,
223 and <filename>compile</filename>.
224 For best performance on multi-core systems, BitBake considers each task as an independent
225 entity with its own set of dependencies.
226 </para>
227
228 <para>
229 Dependencies are defined through several variables.
230 You can find information about variables BitBake uses in the BitBake documentation,
231 which is found in the <filename>bitbake/doc/manual</filename> directory within the
232 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
233 At a basic level, it is sufficient to know that BitBake uses the
234 <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and
235 <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when
236 calculating dependencies.
237 </para>
238 </section>
239
240 <section id='ref-bitbake-tasklist'>
241 <title>The Task List</title>
242
243 <para>
244 Based on the generated list of providers and the dependency information,
245 BitBake can now calculate exactly what tasks it needs to run and in what
246 order it needs to run them.
247 The build now starts with BitBake forking off threads up to the limit set in the
248 <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable.
249 BitBake continues to fork threads as long as there are tasks ready to run,
250 those tasks have all their dependencies met, and the thread threshold has not been
251 exceeded.
252 </para>
253
254 <para>
255 It is worth noting that you can greatly speed up the build time by properly setting
256 the <filename>BB_NUMBER_THREADS</filename> variable.
257 See the
258 "<ulink url='&YOCTO_DOCS_QS_URL;#building-image'>Building an Image</ulink>"
259 section in the Yocto Project Quick Start for more information.
260 </para>
261
262 <para>
263 As each task completes, a timestamp is written to the directory specified by the
264 <filename><link linkend='var-STAMP'>STAMP</link></filename> variable.
265 On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename>
266 directory and does not rerun
267 tasks that are already completed unless a timestamp is found to be invalid.
268 Currently, invalid timestamps are only considered on a per
269 <filename>.bb</filename> file basis.
270 So, for example, if the configure stamp has a timestamp greater than the
271 compile timestamp for a given target, then the compile task would rerun.
272 Running the compile task again, however, has no effect on other providers
273 that depend on that target.
274 This behavior could change or become configurable in future versions of BitBake.
275 </para>
276
277 <note>
278 Some tasks are marked as "nostamp" tasks.
279 No timestamp file is created when these tasks are run.
280 Consequently, "nostamp" tasks are always rerun.
281 </note>
282 </section>
283
284 <section id='ref-bitbake-runtask'>
285 <title>Running a Task</title>
286
287 <para>
288 Tasks can either be a shell task or a Python task.
289 For shell tasks, BitBake writes a shell script to
290 <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script.
291 The generated shell script contains all the exported variables, and the shell functions
292 with all variables expanded.
293 Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
294 Looking at the expanded shell functions in the run file and the output in the log files
295 is a useful debugging technique.
296 </para>
297
298 <para>
299 For Python tasks, BitBake executes the task internally and logs information to the
300 controlling terminal.
301 Future versions of BitBake will write the functions to files similar to the way
302 shell tasks are handled.
303 Logging will be handled in a way similar to shell tasks as well.
304 </para>
305
306 <para>
307 Once all the tasks have been completed BitBake exits.
308 </para>
309
310 <para>
311 When running a task, BitBake tightly controls the execution environment
312 of the build tasks to make sure unwanted contamination from the build machine
313 cannot influence the build.
314 Consequently, if you do want something to get passed into the build
315 task's environment, you must take a few steps:
316 <orderedlist>
317 <listitem><para>Tell BitBake to load what you want from the environment
318 into the data store.
319 You can do so through the <filename>BB_ENV_EXTRAWHITE</filename>
320 variable.
321 For example, assume you want to prevent the build system from
322 accessing your <filename>$HOME/.ccache</filename> directory.
323 The following command tells BitBake to load
324 <filename>CCACHE_DIR</filename> from the environment into the data
325 store:
326 <literallayout class='monospaced'>
327 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR"
328 </literallayout></para></listitem>
329 <listitem><para>Tell BitBake to export what you have loaded into the
330 environment store to the task environment of every running task.
331 Loading something from the environment into the data store
332 (previous step) only makes it available in the datastore.
333 To export it to the task environment of every running task,
334 use a command similar to the following in your
335 <filename>local.conf</filename> or distro configuration file:
336 <literallayout class='monospaced'>
337 export CCACHE_DIR
338 </literallayout></para></listitem>
339 </orderedlist>
340 </para>
341
342 <note>
343 A side effect of the previous steps is that BitBake records the variable
344 as a dependency of the build process in things like the shared state
345 checksums.
346 If doing so results in unnecessary rebuilds of tasks, you can whitelist the
347 variable so that the shared state code ignores the dependency when it creates
348 checksums.
349 For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
350 example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
351 </note>
352 </section>
353
354 <section id='ref-bitbake-commandline'>
355 <title>BitBake Command Line</title>
356
357 <para>
358 Following is the BitBake help output:
359 </para>
360
361 <screen>
362$ bitbake --help
363Usage: bitbake [options] [recipename/target ...]
364
365 Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
366 It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
367 will provide the layer, BBFILES and other configuration information.
368
369Options:
370 --version show program's version number and exit
371 -h, --help show this help message and exit
372 -b BUILDFILE, --buildfile=BUILDFILE
373 Execute tasks from a specific .bb recipe directly.
374 WARNING: Does not handle any dependencies from other
375 recipes.
376 -k, --continue Continue as much as possible after an error. While the
377 target that failed and anything depending on it cannot
378 be built, as much as possible will be built before
379 stopping.
380 -a, --tryaltconfigs Continue with builds by trying to use alternative
381 providers where possible.
382 -f, --force Force the specified targets/task to run (invalidating
383 any existing stamp file).
384 -c CMD, --cmd=CMD Specify the task to execute. The exact options
385 available depend on the metadata. Some examples might
386 be 'compile' or 'populate_sysroot' or 'listtasks' may
387 give a list of the tasks available.
388 -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
389 Invalidate the stamp for the specified task such as
390 'compile' and then run the default task for the
391 specified target(s).
392 -r PREFILE, --read=PREFILE
393 Read the specified file before bitbake.conf.
394 -R POSTFILE, --postread=POSTFILE
395 Read the specified file after bitbake.conf.
396 -v, --verbose Output more log message data to the terminal.
397 -D, --debug Increase the debug level. You can specify this more
398 than once.
399 -n, --dry-run Don't execute, just go through the motions.
400 -S, --dump-signatures
401 Don't execute, just dump out the signature
402 construction information.
403 -p, --parse-only Quit after parsing the BB recipes.
404 -s, --show-versions Show current and preferred versions of all recipes.
405 -e, --environment Show the global or per-package environment complete
406 with information about where variables were
407 set/changed.
408 -g, --graphviz Save dependency tree information for the specified
409 targets in the dot syntax.
410 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
411 Assume these dependencies don't exist and are already
412 provided (equivalent to ASSUME_PROVIDED). Useful to
413 make dependency graphs more appealing
414 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
415 Show debug logging for the specified logging domains
416 -P, --profile Profile the command and save reports.
417 -u UI, --ui=UI The user interface to use (e.g. knotty, hob, depexp).
418 -t SERVERTYPE, --servertype=SERVERTYPE
419 Choose which server to use, process or xmlrpc.
420 --revisions-changed Set the exit code depending on whether upstream
421 floating revisions have changed or not.
422 --server-only Run bitbake without a UI, only starting a server
423 (cooker) process.
424 -B BIND, --bind=BIND The name/address for the bitbake server to bind to.
425 --no-setscene Do not run any setscene tasks. sstate will be ignored
426 and everything needed, built.
427 --remote-server=REMOTE_SERVER
428 Connect to the specified server.
429 -m, --kill-server Terminate the remote server.
430 --observe-only Connect to a server as an observing-only client.
431 </screen>
432 </section>
433
434 <section id='ref-bitbake-fetchers'>
435 <title>Fetchers</title>
436
437 <para>
438 BitBake also contains a set of "fetcher" modules that allow
439 retrieval of source code from various types of sources.
440 For example, BitBake can get source code from a disk with the metadata, from websites,
441 from remote shell accounts, or from Source Code Management (SCM) systems
442 like <filename>cvs/subversion/git</filename>.
443 </para>
444
445 <para>
446 Fetchers are usually triggered by entries in
447 <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>.
448 You can find information about the options and formats of entries for specific
449 fetchers in the BitBake manual located in the
450 <filename>bitbake/doc/manual</filename> directory of the
451 <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
452 </para>
453
454 <para>
455 One useful feature for certain Source Code Manager (SCM) fetchers is the ability to
456 "auto-update" when the upstream SCM changes version.
457 Since this ability requires certain functionality from the SCM, not all
458 systems support it.
459 Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update".
460 This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename>
461 variable.
462 See the
463 "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section
464 in the Yocto Project Development Manual for more information.
465 </para>
466
467 </section>
468
469</chapter>
470<!--
471vim: expandtab tw=80 ts=4 spell spelllang=en_gb
472-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-13 03:56:29 +0000