summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
* sync: fix missing import for -qv2.14.1Mike Frysinger2021-04-211-1/+1
| | | | | | | | | | Some refactors during review dropped this import when it was reworked, but it's still needed when using the --quiet setting. Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: only print error.GitError, don't raise that exception.Raman Tenneti2021-04-151-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | In _FetchOne & _CheckOne, only print error.GitError exception, but other exceptions are still thrown Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py exiting the repo sync. Tested the code with the following commands and verified repo sync continues after fetch error because of an invalid SHA1. $ ./run_tests -v $ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32 ... error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1 .... An error like the following when jobs=1 error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook Bug: https://crbug.com/gerrit/14392 Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* command: add a helper for the parallel execution boilerplateMike Frysinger2021-04-151-30/+19
| | | | | | | | | | | Now that we have a bunch of subcommands doing parallel execution, a common pattern arises that we can factor out for most of them. We leave forall alone as it's a bit too complicated atm to cut over. Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* progress: hide progress bar when --quietMike Frysinger2021-04-131-3/+3
| | | | | | | | | | | | We want progress bars in the default output mode, but not when the user specifies --quiet. Add a setting to the Progress bar class so it takes care of not displaying anything itself rather than having to update every subcommand to conditionally setup & call the object. Change-Id: I1134993bffc5437bc22e26be11a512125f10597f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* command: make --verbose/--quiet available to all subcommandsMike Frysinger2021-04-131-11/+3
| | | | | | | | | | | Add new CommonOptions entry points to move the existing --jobs to, and relocate all --verbose/--quiet options to that. This provides both a consistent interface for users as well as for code. Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: Added --partial-clone-exclude option.Raman Tenneti2021-04-131-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | partial-clone-exclude option excludes projects during partial clone. This is a comma-delimited project names (from manifest.xml). This option is persisted and it is used by the sync command. A project that has been unparital'ed will remain unpartial if that project's name is specified in the --partial-clone-exclude option. The project name should match exactly. Added $ ./run_tests -v Bug: [google internal] b/175712967 "I can't "unpartial" my androidx-main checkout" $ rm -rf androidx-main/ $ mkdir androidx-main/ $ cd androidx-main/ $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml $ repo_dev sync -c -j8 + Verify a project is partial $ cd frameworks/support/ $ git config -l | grep 'partial' + Unpartial a project. $ /google/bin/releases/android/git_repack/git_unpartial + Verify project is unpartial $ git config -l | grep 'partial' $ cd ../.. + Exclude the project from being unparial'ed after init and sync. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml + Verify project is unpartial $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. + Remove the project from exclude list and verify that project is partially cloned. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: add separate --jobs options for different stepsMike Frysinger2021-04-091-4/+11
| | | | | | | | | | | | | | | The number of jobs one wants to run against the network tends to factor differently from the number of jobs one wants to run when checking out local projects. The former is constrained by your internet connection & server limits while the later is constrained by your local computer's CPU & storage I/O. People with beefier computers probably want to keep the network/server jobs bounded a bit lower than the local/checkout jobs. Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: add progress bar to garbage collection phaseMike Frysinger2021-04-091-6/+14
| | | | | | | | | | This can take a few seconds, if not a lot more, so add a progress bar so users understand what's going on. Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: rework selfupdate logicMike Frysinger2021-04-091-48/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current logic has a downside in that it doesn't sync to the latest signed version available if the latest commit itself is unsigned. This can come up when using the "main" branch as it is sometimes signed, but often not as it's holding the latest merged commits. When people use the main branch, it's to get early testing on versions tagged but not yet released, and we don't want them to get stuck indefinitely on that old version of repo. For example, this series of events: * "stable" is at v2.12. * "main" is tagged with v2.13. * early testers use --repo-rev main to get v2.13. * new commits are merged to "main". * "main" is tagged with v2.14. * new commits are merged to "main". * devs who had synced in the past to test v2.13 are stuck on v2.13. repo sees "main" is unsigned and so doesn't try to upgrade at all. The only way to get unwedged is to re-run `repo init --repo-rev main`, or to manually sync once with repo verification disabled, or for us to leave "main" signed for a while and hope devs will sync in that window. The new logic is that whenever changes are available, we switch to the latest signed tag. We also replace some of the duplicated verification code in the sync command with the newer wrapper logic. This handles a couple of important scenarios inaddition to above: * rollback (e.g. v2.13.8 -> v2.13.7) * do not trash uncommitted changes (in case of ad-hoc testing) * switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1) Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: switch network fetch to multiprocessingMike Frysinger2021-04-011-124/+91
| | | | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. This reworks the fetch logic to return results for processing in the main thread instead of leaving every thread to do its own processing. We have to tweak the chunking logic a little here because multiprocessing favors batching over returning immediate results when using a larger value for chunksize. When a single job can be quite slow, this tradeoff is not good UX. Bug: https://crbug.com/gerrit/12389 Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: imply -c if --use-superproject option is used.Raman Tenneti2021-03-241-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/183232698 Bug: https://crbug.com/gerrit/13707 $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync --use-superproject $ repo_dev sync real 0m8.046s user 0m2.866s sys 0m2.457s Second time repo sync took only 8 seconds and verified by printing that urrent_branch_only is True in project.py's Sync_NetworkHalf function. Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: fix reporting of failed local checkoutsv2.13.6Mike Frysinger2021-03-151-1/+1
| | | | | | | | | | | The refactor to multiprocessing broke status reporting slightly when checking out projects. Make sure we mark the step as failed if any of the projects failed, not just when --fail-fast is set. Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Display status messages during repo init/sync.v2.13.3Raman Tenneti2021-03-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Superproject objects accept the optional argument “quiet”. The following progress messages are displayed if quiet is false. Displayed the following message whenever we find we have to make a new folder (aka new remote), because if you started with repo init android and later do googleplex-android that is when it will be slow. "<location>: Performing initial setup for superproject; this might take several minutes.". After fetch completion, added the following notification: "<location>: Initial setup for superproject completed." Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject Bug: [google internal] b/181178282 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: switch local checkout to multiprocessingMike Frysinger2021-02-271-124/+52
| | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down to ~4 sec with -j8. Not a big deal, but shows that this actually works to speed things up unlike the threading model. This reworks the checkout logic to return results for processing in the main thread instead of leaving every thread to do its own processing. Bug: https://crbug.com/gerrit/12389 Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: capture all git output by defaultMike Frysinger2021-02-251-0/+7
| | | | | | | | | | | | The default sync output should show a progress bar only for successful commands, and the error output for any commands that fail. Implement that policy here. Bug: https://crbug.com/gerrit/11293 Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress/sync: include active number of jobsMike Frysinger2021-02-251-11/+16
| | | | | | | | | | Provide a bit more info to users that things are actively running. Bug: https://crbug.com/gerrit/11293 Change-Id: Ie8eeaa8804d1ca71cf5c78ad850fa2d17d26208c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297904 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress: always enable always_print_percentageMike Frysinger2021-02-251-2/+1
| | | | | | | | | | | | | | | | | | | The idea for skipping some progress updates was to avoid spending too much time on the progress bar itself. Unfortunately, for large projects (100s if not 1000s) of repos, we get into the situation with large/slow checkouts that we skip showing updates when a repo finishes, but not enough repos finished to increase the percent. Since the progress bar should be relatively fast compared to the actual network & local dick operations, have it show an update whenever the caller requests it. A test with ~1000 repos shows that the progress bar in total adds <100ms. Bug: https://crbug.com/gerrit/11293 Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use superproject if manifest's config has superproject enabled.Raman Tenneti2021-02-251-1/+3
| | | | | | | | | | | | | | | | | | | If --use-superproject is passed as argument to "repo init", then --use-superproject need not be specified during "repo sync". Tested the code with the following commands. $ time repo_dev sync -c -j32 ... WARNING: --use-superproject is experimental and not for general use Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Ian Kasprzak <iankaz@google.com>
* sync: update event is_set APIMike Frysinger2021-02-231-7/+7
| | | | | | | | | Python 3 renamed this method from isSet to is_set. Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* command: unify --job option & default valuesMike Frysinger2021-02-221-5/+4
| | | | | | | | | | | | Extend the Command class to support adding the --jobs option to the parser if the command declares it supports running in parallel. Also pull the default value used for the number of local jobs into the command module so local commands can share it. Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* trim redundant pass statementsMike Frysinger2021-02-161-1/+0
| | | | | | | | | Clean up a few linter warnings. Change-Id: I531d0263a202435d32d83d87ec24998f4051639c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use subprocess.run to verify tagsMike Frysinger2021-02-161-13/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-111-19/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* diffmanifests/sync: simplify repodir lookupMike Frysinger2021-02-111-1/+1
| | | | | | | | | | We have access to repodir on the command object itself, so we don't need to pull it indirectly out of the manifest object. Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-081-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: superproject performance changes.Raman Tenneti2021-02-071-36/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After updating all project’s revsionIds with the SHAs from superproject, write the updated manifest into superproject_override.xml file. Reload that file for future Reloads. This file is created in exp-superproject directory. Moved most of the code that is superproject specific into git_superproject.py and wrote test code. If git pull fails, did a git clone of the superproject. We saw performance gains for consecutive repo sync's. The time to sync went down from around 120 secs to 40 secs when repo sync is executed consecutively. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. First run $ time repo sync --use-superproject ... real 21m3.745s user 97m59.380s sys 19m11.286s After two consecutive sync runs $ time repo sync -c -j8 --use-superproject real 0m39.626s user 0m29.937s sys 0m38.155s Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: Disable info about disabling pruning when quietAnders Björklund2021-01-221-2/+3
| | | | | | | | | | If you have a lot of shared projects, it spams. Bug: https://crbug.com/gerrit/13961 Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049 Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Added --use-superproject option and support for superproject.v2.12Raman Tenneti2021-01-211-1/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Added "--use-superporject" option to sync.py to fetch project SHAs from superproject. If there are any missing projects in superprojects, it prints the missing entries and exits. If there are no missing entries, it will use SHAs from superproject to fetch the projects from git. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and adding <superporject> tag to default.xml. With local modification to the code to print the status, .../WORKING_DIRECTORY$ repo sync --use-superproject repo: executing 'git clone' url: sso://android/platform/superproject repo: executing 'git ls-tree' Success: [] Bug: https://crbug.com/gerrit/13709 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822 Reviewed-by: Mike Frysinger <vapier@google.com>
* drop pyversion & is_python3 checkingMike Frysinger2021-01-061-20/+5
| | | | | | | | | | | We're committed to Python 3 at this point, so purge all the is_python3 related dynamic checks. Bug: https://crbug.com/gerrit/10418 Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* strip python2-only coding:utf-8 & print_function settingsMike Frysinger2021-01-061-4/+0
| | | | | | | | | | We're committed to Python 3 at this point, so clean up boilerplate. Bug: https://crbug.com/gerrit/10418 Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: respect --force-sync when fetching manifest project updatesErwan Yvin2020-11-171-0/+1
| | | | | | | | | | | The --force-sync option was being passed down for all updates except for the manifest project, so add that there too. Bug: https://crbug.com/gerrit/11034 Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* subcmds/sync.py: Fix typo in helpRoger Shimizu2020-06-061-1/+1
| | | | | | | | Change-Id: I70b63477241284249e395b8b0a220cb6f44f836b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/270183 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@digital.ai> Tested-by: David Pursehouse <dpursehouse@digital.ai>
* sync: fix duplicate word in descriptionMike Frysinger2020-05-241-1/+1
| | | | | | | | Bug: https://crbug.com/gerrit/12814 Change-Id: Id722eec9a59dded588f13bc605ce2c94b4047265 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268739 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@digital.ai>
* Make partial clone imply no-clone-bundle by default.Xin Li2020-05-211-2/+6
| | | | | | | | | | | | | | | | | | For large projects, clone bundle is useful because it provided a way to efficiently transfer a large portion of git objects through CDN, without needing to interact with git server. However, with partial clones, the intention is to not download most of the objects, so the use of clone bundles would defeat the space savings normally seen with partial clones, as they are downloaded before the first fetch. A new option, --clone-bundle is added to override this behavior. Add a new repo.clonebundle variable which remembers the choice if explicitly given from command line at repo init. Change-Id: I03638474af303a82af34579e16cd4700690b5f43 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* sync: add retry to fetch operationsGeorge Engelbrecht2020-04-021-0/+5
| | | | | | | | | | | | | Add retries with exponential backoff and jitter to the fetch operations. By default don't change behavior and enable behind the new flag '--fetch-retries'. Bug: https://crbug.com/1061473 Change-Id: I492710843985d00f81cbe3402dc56f2d21a45b35 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/261576 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: George Engelbrecht <engeg@google.com>
* init: rename --repo-branch to --repo-revMike Frysinger2020-03-181-1/+1
| | | | | | | | | | | | We refer to this as "revision" in help text, and in REPO_REV envvar, so rename to --repo-rev to be consistent. We keep --repo-branch for backwards compatibility, but as a hidden option. Bug: https://crbug.com/gerrit/11045 Change-Id: I1ecc282fba32917ed78a63850360c08469db849a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259352 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* sync: warn if not tracking a branchMike Frysinger2020-03-171-0/+7
| | | | | | | | | | | Since tracking a branch prevents repo from updating, make sure we warn people about the situation when using `repo sync`. Bug: https://crbug.com/gerrit/11045 Change-Id: I966513f510827cc93194f8df176c6745946bd739 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258892 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* sync: fix os.environ logic errorsMike Frysinger2020-03-071-3/+3
| | | | | | | | | This is a dict to index, not a function to call. Change-Id: I0117eeaaa8b2ef4762ab6f0d22f9ffdaee961f52 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/258132 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* sync: Fix flake8 E125 and E129 warningsDavid Pursehouse2020-02-201-3/+3
| | | | | | | | | | | | | | - E129 visually indented line with same indent as next logical line - E125 continuation line with same indent as next logical line Fixed automatically by: autopep8 --in-place --select E125,E129 subcmds/sync.py Change-Id: Ia2f82f443e1e6a23ba22c6f9849c8485405aed0e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256092 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* project/sync: move DeleteProject helper to ProjectMike Frysinger2020-02-201-81/+4
| | | | | | | | | | | | Since deleting a source checkout involves a good bit of internal knowledge of .repo/, move the DeleteProject helper out of the sync code and into the Project class itself. This allows us to add git worktree support to it so we can unlock/unlink project checkouts. Change-Id: If9af8bd4a9c7e29743827d8166bc3db81547ca50 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256072 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: set core.repositoryFormatVersion=1 when using extensionsMike Frysinger2020-02-191-2/+1
| | | | | | | | | | | When using extensions, make sure we set the git repo format version so git knows to check the extension compatibility. We can add a helper to the Project API to simplify this and make it foolproof. Change-Id: I9ab6c32d92fe2b8e5df6e2b080ca71556332e909 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/256035 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* add experimental git worktree supportMike Frysinger2020-02-191-6/+18
| | | | | | | | | | | | | | | | | This provides initial support for using git worktrees internally instead of our own ad-hoc symlink tree. It's been lightly tested which is why it's not currently exposed via --help. When people opt-in to worktrees in an existing repo client checkout, no projects are migrated. Instead, only new projects will use the worktree method. This allows for limited testing/opting in without having to completely blow things away or get a second checkout. Bug: https://crbug.com/gerrit/11486 Change-Id: Ic3ff891b30940a6ba497b406b2a387e0a8517ed8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254075 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* assume environment always accepts stringsMike Frysinger2020-02-191-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Different Python & OS versions have different environ behavior wrt accepted types & encoding. Since we're migrating to be Python 3 only, lets change our code to assume strings always work as that's what the newer Python 3 does. This will fail under Python 2 for some env vars, mostly on Windows, but the effort of maintaining shim layers that can handle these edge cases isn't worth it when we're dropping that code. We leave the logic in the `repo` launcher for now as it is simple, and we want it to be able to switch versions a bit longer than the rest of the tree. Here's the support table: | *NIX | Windows | Python 2 | ASCII string | str or bytes, not unicode | Python 3 | str or bytes | str only | Windows uses strings natively in its environment all the time. But it doesn't allow unicode strings under Python 2, so we have to encode. Python 2 on *NIX is funky in that it always lowers to ASCII, so we had to manually encode to avoid errors regardless of unicode or str. Python 3 on Windows & *NIX will accept strings. *NIX will also accept bytes but Windows will not. Bug: https://crbug.com/gerrit/12145 Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* avoid negative variablesMike Frysinger2020-02-191-9/+9
| | | | | | | | | | | Trying to use booleans with names like "no_xxx" are hard to follow due to the double negatives. Invert all of them so we only have positive meanings to follow. Change-Id: Ifd37d0368f97034d94aa2cf38db52c723ac0c6ed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255493 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* Open temporary cookie file as writable in sync.pyCollin Fijalkovich2020-02-181-1/+1
| | | | | | | | | | | | | Named Temporary file defaults to mode 'w+b' which causes repo sync to fail. By opening the tmpcookiefile in PersistentTransport.request as writable, we are able to run sync successfully. Bug: https://crbug.com/gerrit/12370 Test: Ran smartsync successfully Change-Id: I01ddf915fc30eb3ff0e4d440a6f1aa261c63e88d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255692 Tested-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* sync: introduce --verbose optionMike Frysinger2020-02-171-3/+10
| | | | | | | | | | | | | This allows us to control sync output better by having three levels of output: quiet (only errors), default (progress bars), verbose (all the things). For now, we just put the chatty "already have persistent ref" message behind the verbose level. Bug: https://crbug.com/gerrit/11293 Change-Id: Ia61333fd8085719f3e99edb7b466cdb04031b67f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255414 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* sync: add a "finished" messageMike Frysinger2020-02-121-0/+3
| | | | | | | | | | | | | Some people find the existing output to be a bit confusing. It spews a lot of git output before exiting, but it's not exactly clear what the final state is when things pass. Add an explicit message. Bug: https://crbug.com/gerrit/10501 Change-Id: I9de83b595d3185feb820005b8fc81c6adc55b357 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254732 Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* drop old git_require checksMike Frysinger2020-02-121-2/+1
| | | | | | | | | | We've been requiring git-1.7.2 since Oct 2012, so we can safely drop the individual checks sprinkled throughout the code base for older. Change-Id: I1737fff7b3f27f475960b0bff9cb300aefd5d108 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253135 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* Fix usage of bare 'except'David Pursehouse2020-02-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | flake8 reports: E722 do not use bare 'except' Replace them with 'except Exception' per [1] which says: Bare except will catch exceptions you almost certainly don't want to catch, including KeyboardInterrupt (the user hitting Ctrl+C) and Python-raised errors like SystemExit If you don't have a specific exception you're expecting, at least except Exception, which is the base type for all "Regular" exceptions. [1] https://stackoverflow.com/a/54948581 Change-Id: Ic555ea9482645899f5b04040ddb6b24eadbf9062 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254606 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix blank line issues reported by flake8David Pursehouse2020-02-121-0/+9
| | | | | | | | | | | | | | | | | | | | - E301 expected 1 blank line - E302 expected 2 blank lines - E303 too many blank lines - E305 expected 2 blank lines after class or function definition - E306 expected 1 blank line before a nested definition Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E301,E302,E303,E305,E306 Manually fix issues in project.py caused by misuse of block comments. Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-06-26 03:39:31 +0000