summaryrefslogtreecommitdiffstats
path: root/scripts/lib/resulttool
Commit message (Collapse)AuthorAgeFilesLines
* resulttool/store: Fix permissions of logarchiveRichard Purdie2024-11-261-0/+1
| | | | | | | | | We want the results directory to be visable to other users, tweak the permissions of the created directory to ensure this is the case. (From OE-Core rev: ed9d887e8d71a800db19826264de552f7736dc6a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improve repo layout for oeselftest resultsRichard Purdie2024-11-231-1/+1
| | | | | | | | | | | | Having all oe-selftest results on top of each other results in a large 640MB json file which is hard to use. Split the results out per machine and test type. This also stops the toolchain raw logs from overwriting each other meaning more than one MACHINE is preserved. (From OE-Core rev: 4b890f04bc7d147b4a11b824a84f3d2abd75ac54) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Trim the precision of duration informationRichard Purdie2024-11-231-0/+17
| | | | | | | | | The duration values have pointless amounts of precision. Removing some of the least significant digits reduces result size and makes the results easier to read. (From OE-Core rev: a789a2e6d97bb8efd663226a17db8d1ca6c1e40f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Clean up repoducible build logsRichard Purdie2024-11-231-0/+22
| | | | | | | | | | We've improved the data stored for reproduicible builds. Teach resulttool how to apply those cleanups when reprocessing data so we can reduce results file sizes and make the data easier to process. (From OE-Core rev: b799c57ae6d61c1b1c7035c8a2c4ba6ee08d1a81) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Handle ltp rawlogs as well as ptestRichard Purdie2024-11-231-13/+20
| | | | | | | | | Improve the rawlogs handling to include ltp logs as well as the ptest ones to reduce the size of the results git repos. (From OE-Core rev: a0a1954d559609c2c1ca16936d0d68eb3c4c6b45) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add --logfile-archive option to store modeRichard Purdie2024-11-231-2/+17
| | | | | | | | | | Storing the log files inside the testresults git repo isn't scaling and isn't really appropriate use of a git repository. Allow these to be optionally stored in a separate filesystem location so the git repo can remain managable. (From OE-Core rev: 1afc0f3d7e93fa8496be241e9622d3b9a6904bd5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Fix passthrough of --all files in store modeRichard Purdie2024-11-231-1/+1
| | | | | | | | | When using store mode, --all was broken as not all files were being preserved. Fix this by limiting the scope of the git rm command. (From OE-Core rev: 9604561d2022b6c76b1cb4186d40800d1affdd2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Use single space indentation in json outputRichard Purdie2024-11-233-3/+3
| | | | | | | | | | | Using 4 space indentation in resulted in hundreds of megabytes of extra file size in general use. Reduce this to make filesizes more managable and reduce the processing cost. Some level of indentation and spacing does make the files more readable and allows use of git diff so we need to retain some of it. (From OE-Core rev: a274cdcaf852cca9497f0358f44dda99c06aacbe) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Allow store to filter to specific revisionsRichard Purdie2024-11-141-0/+5
| | | | | | | | | | | We have a challenge on the autobuilder where test results from both OE-Core and poky are being mixed together during result storage which is confusing the data. Add a way to filter to specific revisions as the least worst way to fix the various issues this is causing. (From OE-Core rev: 3f276a0dc65341668788853be2cf27ab6aa12b13) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression.py: skip checking status for ↵Changqing Li2024-10-111-0/+2
| | | | | | | | | | | | | | | | ptestresult.rawlogs/ptestresult.sections ptestresult.rawlogs/ptestresult.sections don't have status is expected, so skip them to avoid following error when running "resulttool regression base target": ERROR: Failed to retrieved base test case status: ptestresult.rawlogs ERROR: Failed to retrieved base test case status: ptestresult.sections (From OE-Core rev: c83a535d1b32f7fd292cd9caea1ec962bc3c735b) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression.py: fix AttributeErrorChangqing Li2024-10-111-0/+1
| | | | | | | | | | | | Fix following AttributeError when running "resulttool regression base target": File "/yocto/poky/scripts/lib/resulttool/regression.py", line 322, in regression_common res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b], args.limit) AttributeError: 'Namespace' object has no attribute 'limit' (From OE-Core rev: d773edde8db2019550916d2552171c45fe31ac2a) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add support to create test report in JUnit XML formatClara Kowalsky2024-08-291-0/+77
| | | | | | | | | | | | | | | | | | | | | | This adds the functionality to convert the results of the testresults.json file to a unit test report in JUnit XML format. The unit test report can be used in the CI/CD pipeline to display the test results. To use the resulttool scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To generate the unit test report, execute the below $ resulttool junit <json_file> By default the unit test report is stored as <build_dir>/tmp/log/oeqa/junit.xml. (From OE-Core rev: 3f9be03946243feaa09b908d7010899769091fe6) Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: group all regressions in regression reportAlexis Lothoré2023-11-051-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit c304fcbe0588b1078373558c2ddf36064bcdf214 introduced a grouping when listing regressions. This grouping has been added only for ptests. It has been observed that any other kind of tests could benefit from it. For example, current regression reports can show the following: 1 regression(s) for oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_help oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED 1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED [...] This output is not so useful in its current state and should be grouped per test type too. Enable grouping for all kind of tests, to make it llok like the following in reports: 5 regression(s) for oescripts oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED (From OE-Core rev: 982798ef96e3a32bf15341bdd3bb7c4356709412) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: make additional info more compactAlexis Lothoré2023-10-231-4/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since "matched" and "improved" tests are not as important as regressions, reduce the place they take in the regression report: - merge "matched" and "improved" tests, while removing the label - add a single line of additional info per pair Those changes make the "Matches and improvements" look like the following sample: oeselftest_almalinux-9.2_qemux86-64_20230910083156 oeselftest_almalinux-8.8_qemux86-64_20231018010951 -> +7 test(s) present oeselftest_almalinux-9.2_qemux86-64_20230911010538 oeselftest_debian-11_qemux86-64_20231017150459 oeselftest_debian-11_qemux86-64_20230910012927 oeselftest_debian-11_qemux86-64_20231017151319 -> +7 test(s) present [...] (From OE-Core rev: 6de4426d9a7da67deed7d3a3918892fb56238ff3) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: rearrange regressions report orderAlexis Lothoré2023-10-231-2/+6
| | | | | | | | | | | | | | Regressions reports currently reports matching pairs and improved pairs first, then regressions. Change order to print regressions first, which is the most valuable info in the report, and then print improvements and matches at the bottom. (From OE-Core rev: 599267467430e70fa4dc8ba6b2a8b126bf6da359) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: limit the number of changes displayed per testAlexis Lothoré2023-10-231-4/+21
| | | | | | | | | | | | | | | | | | | | | | | | Most of the changes list generated in regression reports fall in one of the two following categories: - there is only a few (<10) changes listed and the info is valuable/relevant - the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category suffers the same status (test missing, test failing, test skipped, etc) Prevent those huge, worthless lists by limiting the output for each test group: - current default limit is arbitrarily set to 50 - limit can still be overriden with a new "-l"/"--limit" flag, either with custom value, or with 0 to print the whole lists of changes - limit is applied per test family: currently it distinguishes only types of ptests, but it can be adapted to other kind of tests (From OE-Core rev: cec118406f3ad81cb4709f6e6ae1cef65799658e) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Avoid divide by zeroRichard Purdie2023-08-151-1/+4
| | | | | | | | Avoid a divide by zero traceback if unfortunate test counts are encountered. (From OE-Core rev: c5aeea53dfacb53dedb8445cb3523dc3a8cb6dca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: define custom string for "not found" test resultsAlexis Lothoré2023-08-041-0/+1
| | | | | | | | | | | | | | | | | | | Whenever a test result is present in base test result but absent from target test results, we have an entry in regression report looking like the following one: ptestresult.apr.testfile: PASSED -> None The "None" status may be a bit confusing, so replace it with a custom string which really states what "None" means in this context: ptestresult.apr.testfile: PASSED -> No matching test result (From OE-Core rev: b5f6da6e13f3484f51dfa07f6b3500aea7a21825) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: allow to replace test raw status with custom stringAlexis Lothoré2023-08-041-1/+7
| | | | | | | | | | | | | Add a STATUS_STRINGS dictionnary matching raw statuses to custom strings. Whenever a regression must be reported, raw status is searched in the custom statuses dict (key search is case insensitive). If no custom string is found, raw status is kept and used in regression report (From OE-Core rev: 9d22bfc9d0c4092dba1af0ee11a4c51b7b270786) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: add mention about new detected testsAlexis Lothoré2023-07-251-2/+14
| | | | | | | | | | | | | | | | Some regression reports show a lot of "PASSED->None" transitions. When such big lot of identical transitions are observed, it could be that tests are now failing, but it could also be that some tests has been renamed. To detect such case, add a log in regression report to report the number of new tests (i.e: tests that are present in target results but not in base result). This new log also allows to know about newly added tests bases (From OE-Core rev: 01b5cefd07e01c7407bc663842b8a8d502358a6d) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: allow index generation despite corrupt jsonMichael Halstead2023-07-191-1/+5
| | | | | | | | | | | non-release indexes will continue to generate when test output is corrupted. (From OE-Core rev: 1a9157684a6bff8406c9bb470cb2e16ee006bbe9) Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Improve overlapping ptest result reportingRichard Purdie2023-03-161-5/+3
| | | | | | | | | | With the move to a ptest per image, the multiple ptest log scenario is much more likely. Tweak the handling to only warn if there are overlapping files. (From OE-Core rev: 8391f1668fcbe932ba846299b0ec22de09e06dd4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: add log --list-ptestRoss Burton2023-03-141-0/+5
| | | | | | | | | | Add a convenience argument to the log subcommand to list all of the ptest logs in a testresults file. (From OE-Core rev: cd2d7adf02005d46f19c93a40db1e10ce01ac261) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/resulttool: fix typo breaking resulttool log --ptestRoss Burton2023-03-141-1/+1
| | | | | | | | | | ptestresult_get_log() looked for a key called 'ptestresuls.sections', which should be 'ptestresult.sections' (From OE-Core rev: 7c8c9f7283e54bf8b1521fbaad7dceb66a8fcdbb) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: do not count newly passing tests as regressionsAlexis Lothoré2023-03-121-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | resulttool regression module simply compare a base test status to a target test result status. This approach raises many false positives since all XXX -> PASS transitions (XXX being any status different from PASS) are flagged as regression. - Do not list XXX -> PASS transitions in regression report, instead count them and print a summary of "newly passing tests" - If an inspected pair has only "newly passing tests", do not print detailed list and print it as "Improvement" instead of "Regression" Updated output example looks like the following: [...] Improvement: oeselftest_fedora-37_qemux86-64_20230127010225 oeselftest_ubuntu-22.04_qemux86-64_20230226120516 (+1 test(s) passing) [...] Match: oeselftest_almalinux-8.7_qemuarm64_20230127015830 oeselftest_almalinux-8.7_qemuarm64_20230227015258 [...] Regression: oeselftest_almalinux-9.1_qemumips_20230127000217 oeselftest_opensuseleap-15.4_qemumips_20230226130046 ptestresult.glibc-user.debug/tst-read-chk-cancel: PASS -> None ptestresult.glibc-user.nptl/tst-mutexpi4: PASS -> FAIL ptestresult.glibc-user.nptl/tst-mutexpi5a: PASS -> FAIL Additionally, 44 previously failing test(s) is/are now passing (From OE-Core rev: c335f96f687c73fde443ac330ca3e17113794d9e) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: fix ptests results containing a non reproducible pathAlexis Lothoré2023-03-121-0/+6
| | | | | | | | | | | | | | | | | | | | | | Some ptests results may be wrongly sampled, resulting in some part of the error being propagated in test name. For example: "ptestresult.binutils-ld.in testcase /home/pokybuild/yocto-worker/qemumips/build/build-st-1666126/tmp/work/mips32r2-poky-linux/binutils-cross-testsuite/2.40-r0/git/ld/testsuite/ld-ctf/ctf.exp" "ptestresult.gcc.Couldn't create remote directory /tmp/runtest.455781 on ssh" "ptestresult.gcc-libstdc++-v3.Couldn't create remote directory /tmp/runtest.3814266 on target" While the root errors must be fixed in corresponding ptests packages for those tests, the test results history must still be usable even with this issue Add new filters to detect if temporary test directories (build-st-*, /tmp/runtime.*) are present in name. If so, truncate test name. As a side effect, it will aggregate multiple failing errors into one, but the regression will still be raised (From OE-Core rev: 601eecfddd26bfe2954835a73ed1116bb520235f) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool: call fixup_ptest_names in regression_commonAlexis Lothoré2023-03-121-21/+20
| | | | | | | | | | | | | ptests names not only need to be fixed for regression based on git testresults but also for testsresults provided "manually" Move ptests naming fixup in regression_common to share the fixup between both regression use cases (From OE-Core rev: f772ccd108dc3d618db9d479d672c0f3edd203ca) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Improve matching of poor ptest test names4.2_M3Richard Purdie2023-02-261-0/+21
| | | | | | | | | | | | | | Some test case naming is poor and contains random strings, particularly lttng/babeltrace but also curl. Truncating the test names works since they contain file and line number identifiers which allows us to match them without the random components, or in the case or curl, test IDs. Going forward we may be able to improve the test names but this tweak allows historical test results to work in reports. (From OE-Core rev: 541a2e2683531355e678fd93524a0c4a8c43a8ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/regression: Ensure LTP results are only compared against other ↵Richard Purdie2023-02-261-1/+6
| | | | | | | | | | | LTP runs If a test result contains LTP test results, it should only be compared with other runs containing LTP test results. (From OE-Core rev: 4dbbf2f4a85620a08dc2fa65095dc17fe6c530f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/regression: add metadata filtering for oeselftestAlexis Lothoré2023-02-261-0/+162
| | | | | | | | | | | | | | | | | | | | | When generating regression reports, many false positive can be observed since some tests results are compared while the corresponding tests sets are not the same, as it can be seen for example for oeselftest tests (oeselftest is run multiple time but with different parameters, resulting in different tests sets) Add a filtering mechanism in resulttool regression module to enable a better matching between tests. The METADATA_MATCH_TABLE defines that when the TEST_TYPE is "oeselftest", then resulttool should filter pairs based on OESELFTEST_METADATA appended to test configuration. If metadata is absent from test results, in order to keep compatibility with older results, add a "guessing" mechanism to generate the missing OESELFTEST_METADATA. The guessed data is tightly coupled to the autobuilder configuration, where all oe-selftest executions are described (From OE-Core rev: 94ab7c2b892bf292dd86619ca9c63ddd7bf53f3c) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/regression: remove unused importAlexis Lothoré2023-02-261-1/+0
| | | | | | | (From OE-Core rev: 4d6a74fcee1479562bf66360f20d528107c70887) Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/log: Add ability to dump ltp logs as well as ptestRichard Purdie2020-05-272-15/+28
| | | | | | | | | Currently only ptest logs are accessible with the log command, this adds support so the ltp logs can be extracted too. (From OE-Core rev: 64a2121a875ce128959ee0a62e310d5f91f87b0d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/report: Remove leftover debuggingRichard Purdie2020-05-271-1/+0
| | | | | | | | | I've long since wondered why there was some odd output in result reports, remove the leftover debug which was causing it. (From OE-Core rev: 66e96bf70753933714ff8edcc13a1f35a052656f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Fix unicode error handlingRichard Purdie2020-02-151-4/+1
| | | | | | | | | | | | | | | | | | | | | | This error handling didn't work as expected since upon failure it would inject bytestreams back into the code leading to tracebacks. Instead, ignore the decode errors. Fixes: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module> sys.exit(main()) File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main ret = args.func(args, logger) File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store resultutils.save_resultsdata(results, tempdir, ptestlogs=True) File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata f.write(sectionlog) TypeError: write() argument must be str, not bytes (From OE-Core rev: 3aa0d0fd0bbf8c9c13719251a2fd1e85a22c7336) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/lib/resulttool/report: Enable report selected test case resultYeoh Ee Peng2020-02-131-5/+28
| | | | | | | | | | | | Enable reporting selected test case result given the user provided the selected test case id. If both test result id and test case id were provided, report the selected test case result from the selected test result id. (From OE-Core rev: 7161310ce32d6e0c397d0132808d556bdc80d183) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store.py: Enable add extra test environment dataYeoh Ee Peng2019-11-141-1/+4
| | | | | | | | | | | | | | | Enable the option to add extra test environment data to the configuration of each test result (as optional). Example of optional test environment data include: - custom packages included for runtime test - detail machine specification used as target - detail host environment used for bitbake (From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Add total statistic to test result.Yeoh Ee Peng2019-11-142-1/+7
| | | | | | | | | Add total passed, failed, and skipped statistic to test result. (From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Enable output raw test resultsYeoh Ee Peng2019-11-141-2/+16
| | | | | | | | | | | | | In case of debugging, report user need to acccess the raw test result. Instead of going back to source file/directory/URL to manually pull out the raw result, provide alternative way to let report showing raw test results by providing the result id (optional). (From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/resulttool/report: Enable report to use regression_mapYeoh Ee Peng2019-11-142-7/+13
| | | | | | | | | | | | | | | | | | | | By default, report will use the store_map to generate the key to reference each result set. In some situation when using store_map with multiple set of tests sharing similar test configurations, the report will only showing partial result set for results that having identical result_id (use of multiconfig to run tests where it generate identical result_id). Enable report to have the option to use the regression_map (optional) instead of the default store_map, where it will take larger set of configurations to generate the key to reference each result set, this will prevent the report from only showing partial result set. (From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Encode binary data of logNathan Rossi2019-09-301-1/+5
| | | | | | | | | | | | | | | Do not decode the log content into a string only to re-encode it as binary data again. Some logs might un-intentionally contain bytes that do not decode as utf-8, as such preserve the log file content as it was on disk. Handle the decoding on the resulttool side, but also handle the failure to decode the data. (From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpersNathan Rossi2019-09-162-28/+60
| | | | | | | | | | | | | | | | Add the OEPTestResultTestCase class as a mix-in class to provide helper functions for interacting with ptestresults within the extraresults object generated by the test case. This class also provides default compression of log text and log files. Also add support to resulttool for decoding/decompressing log files embedded in the test results. (From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Handle multiple series containing ptestresultsNathan Rossi2019-09-161-12/+32
| | | | | | | | | | | | Handle multiple results series having ptestresults content. The contents are merged on a per-result basis where duplicates are ignored (with a warning message printed). The 'ptestresults.sections' collection is also merged on a per-suite basis. (From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Add reproducible log extractionJoshua Watt2019-09-061-3/+30
| | | | | | | | | | | | | | | | | Adds an argument to the log subcommand to extract the raw logs from the reproducible selftest. To prevent ambiguity, the "--raw" argument has been renamed "--raw-ptest", although the old "--raw" argument is kept around for compatibility. [YOCTO #13324] (From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/lib/resulttool/report.py: Add more result typesNathan Rossi2019-08-281-3/+3
| | | | | | | | | | | Add additional result types into the dictionary to handle dejagnu style test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED, UNTESTED, UNRESOLVED and ERROR. (From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Prevent multiple results for the same testJon Mason2019-08-211-0/+14
| | | | | | | | | | | | | Currently, if a test occurs multiple times over different series, the code will sum these. This can lead to confusion over the desired results. Change the code to report the redundant tests and skip adding an additional values. (From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/merge: Enable control TESTSERIES and extra configurationsYeoh Ee Peng2019-06-121-5/+15
| | | | | | | | | | | | | Current QA team need to merge test result files from multiple sources. Adding TESTSERIES configuration too early will have negative implication to report and regression. Enable control to add TESTSERIES when needed. Also enable adding EXECUTED_BY configuration when needed. (From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/store: Enable add EXECUTED_BY config to resultsYeoh Ee Peng2019-06-121-2/+7
| | | | | | | | | | | Current results stored does not have information needed to trace who executed the tests. Enable store to add EXECUTED_BY configuration to results file in order to track who executed the tests. (From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/resultutils: Enable add extra configurations to resultsYeoh Ee Peng2019-06-121-7/+12
| | | | | | | | | | | | Current resultutils library always add "TESTSERIES" configuration to results. Enhance this to allow control of adding "TESTSERIES" configuration as well as allow adding extra configurations when needed. (From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool: Remove prints if no tests occurJon Mason2019-06-041-11/+3
| | | | | | | | | | Printing the lack of a test is not necessary (per feedback). Remove this from the template to quieten it. (From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* resulttool/manualexecution: Enable creation of test case configurationsangeeta jain2019-06-041-1/+30
| | | | | | | | | | | | Allow the creation of test case configuration file based on user inputs. Where this testcase configuration file will be used by the the manual execution to run selected test cases for a module rather than compulsory run all test cases in manual json file. (From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-07-21 09:43:20 +0000