| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
|
|
| |
The previous fix introduced a race where the queue might not be empty
but all the parser processes have exited. Handle this correctly to avoid
occasional errors.
(Bitbake rev: 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 15s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
combined with +=/.=
Operations like XXX:append += "YYY" are almost always wrong and this
is a common mistake made in the metadata. Show warnings for these usages
with a view to making it a fatal error eventually.
(Bitbake rev: 8c31e75557dc6a8d8f407b5d24d6327889a3e3b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 19291665fa8b6cc331290f2542af3e8e653203f1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
The previous fix for runall option handling had a small bug in it, it
didn't clear the originally processed task list which meant it was running
too many tasks. Fix this so the list is reset and rebuild correctly.
(Bitbake rev: 87c9e120897ed04dfc64d4752fc602f9bfcb8645)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently if you have something like:
SRC_URI = "file://foobar;subdir=${S}"
and a file like:
foobar/1/somefile
and then move it to:
foobar/2/somefile
the task checksums don't reflect/notice this. The file-checksum fields
encode two pieces of data, the file path and whether or not the file
exists. Changing the code which uses these fields is problematic.
We can however add a "/./" path element which means "include the bit
after the marker in the checksum" which the path walking code can use
to mark which bits of the path are visible to the fetcher.
I'm not convinced this is great design but it does appear to work.
(Bitbake rev: b4975d2ecf615ac4c240808fbc5a3f879a93846b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
If one of the parse threads disappears during parsing for some reason, bitbake
currently hangs. Avoid this (and zombie threads hanging around) by joining()
threads which have exited.
(Bitbake rev: dc86a533d951d13643ce446533370da804782afc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.
(Bitbake rev: 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
When the user specify an invalid upstream hash equivalence server in
BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server.
(Bitbake rev: be45aeb9a84f30c28711e87e2d2a4a86320a8d94)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The runbuild option handling in runqueue was flawed as items deleted from the
main task list may be dependencies and hence cause index errors.
Rather than modify runtaskentries straight away, compute a new shorted list
and use that as an input to the second phase. This avoids the need to add tasks
back to the list meaning delcount can be simplifed to a simple counter.
The second use case in runonly doen't re-add items so doesn't have this
issue.
(Bitbake rev: 3428e3c54eb5cc03ff96f9cee6dc839afee7a419)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Add a test to ensure that a parameter like protocol=git can be rewritten
to a different url and protocol.
(Bitbake rev: 69b4f9a09ff74378788cc2ec1ad58cd66b27ca59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
In theory we can leak the so file descriptor so refactor the code
to avoid that.
(Bitbake rev: dfad69d4d8c894a5e1e2686023e41552de09bf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
On master, tell the users they need to update their urls for github.
(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 07fca7e3ab696ba985b3ef86ab9031d688bf2df2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.
(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.
Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.
We could consider dropping destsuffix as a parameter as some future point.
Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.
(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: 9dcd9a3c671b24982aff9a994e31cb9f491471f5)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.
"bitbake: fetch2: fix premirror URI when downloadfilename defined"
Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.
This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
(Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
for premirror
Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.
Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.
(Bitbake rev: 20aabc3d53f69949810ecf02295725db947ffef8)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.
(Bitbake rev: 3b4d2e3b5024324058360a2a28f33c34114218d0)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
Our minimum python version is 3.6 so these are now unneeded.
(Bitbake rev: 15f16bf973666807e94f3b356095f8265af0282a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 17d74fc64003770a94dfffa2ab102254fa52d585)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: cf421235bd6f5eb12e9193634c0e870ab035b191)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: 473e2a5486bd972ad0f808db089abcb8945d3a48)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The switch to using json has messed up the type handling as the code
does assume that set()s are present. Add a decoder to reconstruct
the set() objects. Also fix the change of tuples to lists for the
file checksums and fix an existing type bug where dicts insteads of
lists was used.
Drop some old siginfo format handling code which is now long since
obsolete.
(Bitbake rev: 2d704842c0928f8dbe78fd081042aa7280af96be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
npmsw(and maybe other fetchers) may try to download same files simulatenously.
if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.
In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.
(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.
(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since OE is about to change to zstd compression of sstate, it would make it
timely to convert the siginfo files from pickle which isn't reproducible
to json which is both reproducible and also human readable. At the same time
add zstd compression. This makes the siginfo files smaller, reprodubicle
and easier to debug.
Backwards compatibility mixing the two formats hasn't been supported since
in reality if sstate changes at the same time, files will be in one format
or the new one but comparing mixed formats won't make much sense.
Since json doesn't support sets, we translate them into lists in the files
themselves. We only use sets in bitbake since it makes things easier in
the internal code, sorted lists are fine for the file format.
[YOCTO #13973]
(Bitbake rev: 22c18494c9072788e6e26eb73de70378ae5c5bf5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The code works fine if we pass a file descriptor in and we need to
do this from the siggen code so add that as a valid input.
(Bitbake rev: fc8d74b7de576005ecf67920501dc7e694880fda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
The checkstatus semantics are unconventional, so document them.
(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(Bitbake rev: c78ebac71ec976fdf27ea24767057882870f5c60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: 4f983dc419a1a6f635a5d333f253d49244cec374)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
When writing functions into shell scripts, write then in a deterministic
order. This is unlikely to affect anything at runtime but it does change
the signatures of the generated useradd postinst scripts in OE-Core and is
a good thing to be consistent about in general.
(Bitbake rev: 8a181dc8f3c8c9c9885ea3011cb234321a296d92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.
(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|