summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* git_config: hoist Windows ssh check earlierv2.14.4Mike Frysinger2021-05-041-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | The ssh master logic has never worked under Windows which is why this code always returned False when running there (including cygwin). But the OS check was still done while holding the threading lock. While it might be a little slower than necessary, it still worked. The switch from the threading module to the multiprocessing module changed global behavior subtly under Windows and broke things: the globals previously would stay valid, but now they get cleared. So the lock is reset to None in children workers. We could tweak the logic to pass the lock through, but there isn't much point when the rest of the code is still disabled in Windows. So perform the platform check before we grab the lock. This fixes the crash, and probably speeds things up a few nanoseconds. This shouldn't be a problem on Linux systems as the platform fork will duplicate the existing process memory (including globals). Bug: https://crbug.com/gerrit/14480 Change-Id: I1d1da82c6d7bd6b8cdc1f03f640a520ecd047063 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305149 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: fix print error when handling server errorv2.14.3Mike Frysinger2021-05-041-1/+1
| | | | | | | | | | | When converting this logic from print() to the output buffer, this error codepath should have dropped the use of the file= redirect. Bug: https://crbug.com/gerrit/14482 Change-Id: Ib484924a2031ba3295c1c1a5b9a2d816b9912279 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305142 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: Fix exception in an exsiting clone (without partial-clone).v2.14.2Raman Tenneti2021-04-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Default the partial_clone_exclude argument to an empty set. Fixes the following report by Emil Medve. With this change (up to v2.14.1), on an existing "normal" clone (without partial-clone options) I'm seeing this traceback during `repo selfupdate`: Traceback (most recent call last): File ".../.repo/repo/main.py", line 630, in <module> _Main(sys.argv[1:]) File ".../.repo/repo/main.py", line 604, in _Main result = run() File ".../.repo/repo/main.py", line 597, in <lambda> run = lambda: repo._Run(name, gopts, argv) or 0 File ".../.repo/repo/main.py", line 261, in _Run result = cmd.Execute(copts, cargs) File ".../.repo/repo/subcmds/selfupdate.py", line 54, in Execute if not rp.Sync_NetworkHalf(): File ".../.repo/repo/project.py", line 1091, in Sync_NetworkHalf if self.name in partial_clone_exclude: TypeError: argument of type 'NoneType' is not iterable $ ./run_tests -v Change-Id: I71e744e4ef2a37b13aa9ba42eba3935e78c4e40a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304082 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: fix missing import for -qv2.14.1Mike Frysinger2021-04-211-1/+1
| | | | | | | | | | Some refactors during review dropped this import when it was reworked, but it's still needed when using the --quiet setting. Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: restore default --manifest-nameMike Frysinger2021-04-211-1/+1
| | | | | | | | | | | | | The merge of the repo & init parser missed this default. When running `repo init ...` in an existing checkout but w/out the -m option, then repo would error out complaining that -m is required when it didn't do this before. Change-Id: I58035d48cc413b5d373702b9dc3b9ecd3fd1e900 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303945 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* sync: cleanup sleep+retry logic a bitv2.14Mike Frysinger2021-04-191-14/+18
| | | | | | | | | | | | | | | | | | | Make sure we print a message whenever we retry so it's clear to the user why repo is pausing for a long time, and why repo might have passed even though it displayed some errors earlier. Also unify the sleep logic so we don't have two independent methods. This makes it easier to reason about. Also don't sleep if we're in the last iteration of the for loop. It doesn't make sense to and needlessly slows things down when there are real errors. Bug: https://crbug.com/gerrit/12494 Change-Id: Ifceace5b2dde75c2dac39ea5388527dd37376336 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303402 Reviewed-by: Sam Saccone 🐐 <samccone@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: only print error.GitError, don't raise that exception.Raman Tenneti2021-04-151-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | In _FetchOne & _CheckOne, only print error.GitError exception, but other exceptions are still thrown Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py exiting the repo sync. Tested the code with the following commands and verified repo sync continues after fetch error because of an invalid SHA1. $ ./run_tests -v $ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32 ... error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1 .... An error like the following when jobs=1 error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook Bug: https://crbug.com/gerrit/14392 Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* command: add a helper for the parallel execution boilerplateMike Frysinger2021-04-1510-143/+145
| | | | | | | | | | | Now that we have a bunch of subcommands doing parallel execution, a common pattern arises that we can factor out for most of them. We leave forall alone as it's a bit too complicated atm to cut over. Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* tests: Make ReviewableBranchTests.test_smoke work with git < 2.28.0Peter Kjellerstedt2021-04-141-1/+1
| | | | | | | | Bug: https://crbug.com/gerrit/14380 Change-Id: Id015bd98b008e1530ada2c7e4332c67e8e208e25 Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303325 Reviewed-by: Mike Frysinger <vapier@google.com>
* list: fix help grammarMike Frysinger2021-04-141-3/+3
| | | | | | | Change-Id: Ia642e38532173d59868e0101cc098eab706d715e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303302 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* help: switch from formatter module to textwrapMike Frysinger2021-04-142-8/+14
| | | | | | | | | | | | | | | | Since Python has deprecated the formatter module, switch to the textwrap module instead for reflowing text. We weren't really using any other feature anyways. Verified by diffing the output before & after the change and making sure it was the same. Then made a few tweaks to tighten up the output. Change-Id: I0be1bc2a6661a311b1a4693c80d0f8366320ba55 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303282 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* list: add option to show non-checkedout projects tooMike Frysinger2021-04-131-3/+12
| | | | | | | | | | | | | Currently, list only shows projects that exist in the checkout, and doesn't offer any way to list all projects in the manifest (based on the current settings, or on the options passed to list). This seems to be the opposite of what (at least some) users expect, so let's add an option to show all of them regardless of checkout state. Change-Id: I94bbdc5bd0ff2a411704fa215e7fc2b60fa3360e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301263 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress: hide progress bar when --quietMike Frysinger2021-04-135-8/+16
| | | | | | | | | | | | We want progress bars in the default output mode, but not when the user specifies --quiet. Add a setting to the Progress bar class so it takes care of not displaying anything itself rather than having to update every subcommand to conditionally setup & call the object. Change-Id: I1134993bffc5437bc22e26be11a512125f10597f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* command: make --verbose/--quiet available to all subcommandsMike Frysinger2021-04-1314-40/+41
| | | | | | | | | | | Add new CommonOptions entry points to move the existing --jobs to, and relocate all --verbose/--quiet options to that. This provides both a consistent interface for users as well as for code. Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: Added --partial-clone-exclude option.Raman Tenneti2021-04-136-21/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | partial-clone-exclude option excludes projects during partial clone. This is a comma-delimited project names (from manifest.xml). This option is persisted and it is used by the sync command. A project that has been unparital'ed will remain unpartial if that project's name is specified in the --partial-clone-exclude option. The project name should match exactly. Added $ ./run_tests -v Bug: [google internal] b/175712967 "I can't "unpartial" my androidx-main checkout" $ rm -rf androidx-main/ $ mkdir androidx-main/ $ cd androidx-main/ $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml $ repo_dev sync -c -j8 + Verify a project is partial $ cd frameworks/support/ $ git config -l | grep 'partial' + Unpartial a project. $ /google/bin/releases/android/git_repack/git_unpartial + Verify project is unpartial $ git config -l | grep 'partial' $ cd ../.. + Exclude the project from being unparial'ed after init and sync. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml + Verify project is unpartial $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. + Remove the project from exclude list and verify that project is partially cloned. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: add separate --jobs options for different stepsMike Frysinger2021-04-091-4/+11
| | | | | | | | | | | | | | | The number of jobs one wants to run against the network tends to factor differently from the number of jobs one wants to run when checking out local projects. The former is constrained by your internet connection & server limits while the later is constrained by your local computer's CPU & storage I/O. People with beefier computers probably want to keep the network/server jobs bounded a bit lower than the local/checkout jobs. Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: organize command line options a bitMike Frysinger2021-04-091-18/+28
| | | | | | | | | | We've grown a lot of options in here and it's hard to make sense of them. Add more groups to try and make it easier to pick things out. Change-Id: I6b9dc0e83f96137f974baf82d3fb86992b857bd2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302803 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: add progress bar to garbage collection phaseMike Frysinger2021-04-091-6/+14
| | | | | | | | | | This can take a few seconds, if not a lot more, so add a progress bar so users understand what's going on. Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: rework selfupdate logicMike Frysinger2021-04-091-48/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current logic has a downside in that it doesn't sync to the latest signed version available if the latest commit itself is unsigned. This can come up when using the "main" branch as it is sometimes signed, but often not as it's holding the latest merged commits. When people use the main branch, it's to get early testing on versions tagged but not yet released, and we don't want them to get stuck indefinitely on that old version of repo. For example, this series of events: * "stable" is at v2.12. * "main" is tagged with v2.13. * early testers use --repo-rev main to get v2.13. * new commits are merged to "main". * "main" is tagged with v2.14. * new commits are merged to "main". * devs who had synced in the past to test v2.13 are stuck on v2.13. repo sees "main" is unsigned and so doesn't try to upgrade at all. The only way to get unwedged is to re-run `repo init --repo-rev main`, or to manually sync once with repo verification disabled, or for us to leave "main" signed for a while and hope devs will sync in that window. The new logic is that whenever changes are available, we switch to the latest signed tag. We also replace some of the duplicated verification code in the sync command with the newer wrapper logic. This handles a couple of important scenarios inaddition to above: * rollback (e.g. v2.13.8 -> v2.13.7) * do not trash uncommitted changes (in case of ad-hoc testing) * switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1) Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: merge subcmd & wrapper parsersMike Frysinger2021-04-093-109/+6
| | | | | | | | | | These are manually kept in sync which is a pain. Have the init subcmd reuse the wrapper code directly. Change-Id: Ica73211422c64377bacc9bb3b1d1a8d9d5f7f4ca Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302762 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* repo init: Added --no-partial-clone and made it persist. Bumped version to 2.14.Raman Tenneti2021-04-052-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Saved the repo.partialclone when --no-partial-clone option is passed to init, so repo sync will honor the no-partial-clone option. $ ./run_tests -v Bug: [google internal] b/175712967 $ mkdir androidx-main && cd androidx-main $ repo init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M $ repo sync -c -j32 $ cd frameworks/support/ && /google/bin/releases/android/git_repack/git_unpartial $ git config -l | grep 'partialclonefilter=blob' Observe partialclone is not enabled. $ cd ../.. $ repo init -u https://android.googlesource.com/platform/manifest -b androidx-main $ repo sync -c -j32 $ cd frameworks/support/ && git config -l | grep 'partialclonefilter=blob' Observe partialclone is enabled. $ /google/bin/releases/android/git_repack/git_unpartial Observe partialclone is not enabled. $ cd ../.. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --no-partial-clone $ repo sync -c -j32 $ cd frameworks/support/ && git config -l | grep 'partialclonefilter=blob' Observe partialclone is not enabled. Change-Id: I4400ad7803b106319856bcd0fffe00bafcdf014e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302122 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* tox: enable python 3.5 & 3.9 testingMike Frysinger2021-04-014-5/+7
| | | | | | | | | We still support Python 3.5, so make sure it keeps working. Change-Id: I150158a656b26de6d733316a68a2cbb8b5b99716 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299625 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* run_tests: fix exit code handlingMike Frysinger2021-04-011-1/+1
| | | | | | | | | | | We need to pass back an int, not a CompletedProcess object. Switch to check=False so we don't throw an exception on failure -- we're already showing pytest's stderr, and will return the non-zero status. Change-Id: Ib0d3862a09a3963f25025f39a8e34419cf2a54df Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299624 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* setup: bump major versionMike Frysinger2021-04-011-1/+1
| | | | | | | | | | We don't keep this up-to-date in general, but might as well keep the major version in sync. Change-Id: I20908005b3b393d384da0ef9342d7c9d094550cb Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299622 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: switch network fetch to multiprocessingMike Frysinger2021-04-012-126/+93
| | | | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. This reworks the fetch logic to return results for processing in the main thread instead of leaving every thread to do its own processing. We have to tweak the chunking logic a little here because multiprocessing favors batching over returning immediate results when using a larger value for chunksize. When a single job can be quite slow, this tradeoff is not good UX. Bug: https://crbug.com/gerrit/12389 Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* grep: add --jobs supportMike Frysinger2021-04-011-46/+72
| | | | | | | | | | | | | | | Use multiprocessing to run in parallel. When operating on multiple projects, this can greatly speed things up. Across 1000 repos, it goes from ~40sec to ~16sec with the default -j8. The output processing does not appear to be a significant bottle neck -- it accounts for <1sec out of the ~16sec runtime. Thus we leave it in the main thread to simplify the code. Change-Id: I750b72c7711b0c5d26e65d480738fbaac3a69971 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297984 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* prune: add --jobs supportMike Frysinger2021-03-311-4/+25
| | | | | | | | | | | | | | | | Use multiprocessing to run in parallel. When operating on multiple projects, this can greatly speed things up. Across 1000 repos, it goes from ~10sec to ~4sec with the default -j8. This only does a simple conversion over to get an easy speedup. It is currently written to collect all results before displaying them. If we refactored this module more, we could have it display results as they came in. Change-Id: I5caf4ca51df0b7f078f0db104ae5232268482c1c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298643 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: handle missing project refs betterv2.13.8Mike Frysinger2021-03-251-2/+9
| | | | | | | | | | | | | | If the project exists, but the ref the manifest wants doesn't exist, don't throw an error (and abort the process in general). This can come up with a partially synced tree: the manifest is up-to-date, but not all the projects have yet been synced. Bug: https://crbug.com/gerrit/14289 Change-Id: Iba97413c476544223ffe518198c900c2193a00ed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301262 Reviewed-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: imply -c if --use-superproject option is used.Raman Tenneti2021-03-241-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/183232698 Bug: https://crbug.com/gerrit/13707 $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync --use-superproject $ repo_dev sync real 0m8.046s user 0m2.866s sys 0m2.457s Second time repo sync took only 8 seconds and verified by printing that urrent_branch_only is True in project.py's Sync_NetworkHalf function. Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* start: add a --HEAD aliasMike Frysinger2021-03-241-1/+2
| | | | | | | | | | This makes it easy to use --H as a shortcut, and kind of matches the use of storing HEAD as the revision. Change-Id: I590bf488518f313e7a593853140316df98262d7e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301163 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall/list: delete spurious "Mike Frysinger2021-03-192-2/+2
| | | | | | | Change-Id: I6995d48be1d8fc5d93f4b9fa617fad70b5b3429f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300902 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Added --depth=1 argument to git fetch command.Raman Tenneti2021-03-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Without --depth $ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject real 6m48.086s user 3m27.281s sys 1m1.386s With --depth=1 $ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject real 2m49.637s user 2m51.458s sys 0m39.108s From dwillemsen@: "For me it's the difference between 9m28s using the current code and 16s using --depth=1 while fetching the superproject." Bug: [google internal] b/180451672 Bug: https://crbug.com/gerrit/13707 Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* forall: allow interactive commands with -j1v2.13.7Mike Frysinger2021-03-181-1/+16
| | | | | | | | | | | | | Historically forall has been interactive since it ran in serial. Recent rework in here dropped that to enable parallel processing. Restore support for interactive commands when running -j1 or with an explicit --interactive option. Bug: https://crbug.com/gerrit/14256 Change-Id: I502007186f771914cfd7830846a4e1938b5e1f38 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300722 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* repo: Add a new "command" event type to git trace2 logging in repo.Raman Tenneti2021-03-183-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new "event": "command", which is emitted at when all command arguments have been processed. Additional fields: "name": Name of the primary command (ex: repo, git) "subcommands"': List of the sub-commands once command-line arguments are processed Examples: Command: repo --version Event: {"event": "command", <common fields>, "name": "repo", "subcommands": ["version"] } Bug: [google internal] b/178507266 Testing: - Unit tests - Verified repo git trace2 logs had expected data Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343 Reviewed-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: fix reporting of failed local checkoutsv2.13.6Mike Frysinger2021-03-151-1/+1
| | | | | | | | | | | The refactor to multiprocessing broke status reporting slightly when checking out projects. Make sure we mark the step as failed if any of the projects failed, not just when --fail-fast is set. Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* bash-completion: initial import based on CrOS versionMike Frysinger2021-03-151-0/+121
| | | | | | | | | | | We've had a limited version of this in CrOS for a long time. There's nothing CrOS specific about it, so lets move it to the repo project so everyone can utilize it. Change-Id: I04cd94610c1100f3afcd2baf8c8e7ab13e589490 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299202 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: allow toplevel project checkoutsv2.13.5Mike Frysinger2021-03-123-4/+45
| | | | | | | | | | | | | | Re-allow checking out projects to the top of the repo client checkout. We add checks to prevent checking out files under .repo/ as that path is only managed by us, and projects cannot inject content or settings into it. Bug: https://crbug.com/gerrit/14156 Bug: https://crbug.com/gerrit/14200 Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* document the new manifest restrictions on name & path settingsMike Frysinger2021-03-121-2/+23
| | | | | | | | Bug: https://crbug.com/gerrit/14156 Change-Id: I473edab1173e6a266d0754c29d5dc7ff761f1359 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299403 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* prune: minor optimization & robustificationMike Frysinger2021-03-121-0/+5
| | | | | | | | | | | | | If the current project doesn't have any local branches, then there's nothing to prune, so return right away. This avoids running a few git commands when we aren't actually going to use the results, and it avoids checking repository validity. Since we aren't going to do anything in here, no need to check it. Change-Id: Ie9d5c75a954e42807477299f3e5a63a92fac138b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299742 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: pass groups to ToXml method.Raman Tenneti2021-03-117-27/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added the following methods to XmlManifest class. + GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower() + GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func. + Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr. + Used the above methods to get groups in command.py::GetProjects and part of init.py. TODO: clean up these funcs to take structured group data more instead of passing strings around everywhere that need parsing. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL and verified prebuilts/fullsdk-linux directory has all the folders. Tested repo init and repo sync with --use-superproject and without --use-superproject argument. $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j32 Bug: [google internal] b/181804931 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* project: fix diff printing with embedded %v2.13.4Mike Frysinger2021-03-091-1/+1
| | | | | | | | | | | | The recent commit 84230009ee4282b947482f0d4fc4fe9e9ebc9e01 ("project: make diff tools synchronous") broke repo diff if it includes % formats. Add an explicit format string to fix. Bug: https://crbug.com/gerrit/14208 Change-Id: Ie255a43c5b767488616b2b3dd15abc18f93bfab2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299402 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: expose --worktree optionMike Frysinger2021-03-092-4/+2
| | | | | | | | | | | There's a few rough edges here still, but no known corruption ones, so open it up a bit for people to experiment with. Bug: https://crbug.com/gerrit/11486 Change-Id: I81e0122ab6d3e032c546c8239dd4f03740676e80 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299242 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* Log repo.* config variables in git trace2 logger.Ian Kasprzak2021-03-084-0/+81
| | | | | | | | | | | | | Bug: [google internal] b/181758736 Testing: - Unit tests - Verified repo git trace2 logs had expected data Change-Id: I9af8a574377bd91115f085808c1271e9dee16a36 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299182 Tested-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* superproject: Display status messages during repo init/sync.v2.13.3Raman Tenneti2021-03-043-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Superproject objects accept the optional argument “quiet”. The following progress messages are displayed if quiet is false. Displayed the following message whenever we find we have to make a new folder (aka new remote), because if you started with repo init android and later do googleplex-android that is when it will be slow. "<location>: Performing initial setup for superproject; this might take several minutes.". After fetch completion, added the following notification: "<location>: Initial setup for superproject completed." Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject Bug: [google internal] b/181178282 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* tests: fix duplicate method from copy & paste errorMike Frysinger2021-03-041-2/+2
| | | | | | | | | Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002 Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: fix variable typov2.13.2Mike Frysinger2021-03-031-1/+1
| | | | | | | | | Bug: https://crbug.com/gerrit/11293 Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com> Change-Id: I37bac58aa1dc9ecc10e29253d14ff9e6fb42427c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298942 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: relax include name rules for user-specified pathv2.13.1Mike Frysinger2021-03-023-15/+62
| | | | | | | | | | | | | Allow the user to specify relative or absolute or any other funky path that they want when using `repo init` or `repo sync`. Our goal is to restrict the paths in the remote manifest git repo we cloned from the network, not protect the user from themselves. Bug: https://crbug.com/gerrit/14156 Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: greatly speed up processing overheadv2.13Mike Frysinger2021-03-011-42/+17
| | | | | | | | | | | | | | | | With the recent commit 0501b29e7ae072e0b10ea9ddd913ec6d5975f690 ("status: Use multiprocessing for `repo status -j<num>` instead of threading"), the limitation with project serialization no longer applies. It turns out that ad-hoc logic is expensive. In the CrOS checkout (~1000 projects w/8 jobs by default), it adds about ~7sec overhead to all invocations. With a fast nop run: time repo forall -j8 -c true This goes from ~11sec to ~4sec -- more than 50% speedup. Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: simplify arg passing to worker childrenMike Frysinger2021-03-011-20/+6
| | | | | | | | | | | The ProjectArgs function can be inlined which simplifies it quite a bit. We shouldn't need the custom exception handling here either. This also makes the next commit easier to review. Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: move nested func out to the classMike Frysinger2021-03-011-5/+7
| | | | | | | | | | | This is in preparation for simplifying the jobs support. The nested function is referenced in the options object which can't be pickled, so pull it out into a static method instead. Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-20 18:36:04 +0000