summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
...
* sync: dump git output on errorsMike Frysinger2021-07-041-1/+1
| | | | | | | | Bug: https://crbug.com/gerrit/14700 Change-Id: I1ae53dc7f3792b7e8f11d73f706864fb6591eee8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311142 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* Add the ability to administratively enroll repo into using superproject.Xin Li2021-07-011-9/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Repo will remember a choice and an expiration time of the choice, per user, about whether to use superproject by default. When not specified from command line and the choice is not expired, repo would use the user default value. When a user default value is not present and when the system wide enable default is provided in git's system configuration, repo would ask the user for a confirmation which will be valid for two weeks. git_config.py: Add support for system config. When reading system config, we would use --system to avoid hardcoding a path as the value may be different on some other distributions. git_superproject.py: Add a new subroutine, _UseSuperproject(), which returns whether superproject should be used and whether it is from a user configuration. The value is determined in the following order: 1. If the user specifies either --use-superproject or --no-use-superproject, then that choice is being used. 2. If neither is specified, we would then check the saved value (upon repo init) and use that choice when there was a choice. 3. We then check if there is a saved and unexpired value for user's choice in their ~/.gitconfig, and use the unexpired choice, if available. 4. Finally, if all the above didn't give us a decision, and if the git system configuration is providing a rollout hint, present a prompt to user for their decision and save it in ~/.gitconfig. subcmds/sync.py: Make use of the new UseSuperproject() provided by git_superproject.py. While there also silent stderr from git describe when determining the version of repo. Bug: [google internal] b/190688390 Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* superproject: Don't exit if superproject tag doesn't exist in manifest.v2.16Raman Tenneti2021-06-161-6/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't exit if there are missing commit ids in superproject. This change implements the following suggestion from delphij@: "we should note the event (so we know that --use-superproject but there were some errors, e.g. manifest didn't specify commit id for some reason, or if there is no superproject but --use-superproject is used), print out a message telling the use that this is not support, but continue as if --no-use-superproject was specified?" Changes: superproject: + Added git_trace2_event_log as an argument to the constructor. + Sync method returns SyncResult a NamedTuple of ++ success - True if sync of superproject is successful, or False. ++ fatal - True if caller should exit, Or False. + UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of ++ manifest_path - path name of the overriding manifest file instead of None ++ fatal - True if caller should exit, Or False + _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of ++ commit_ids - a dictionary with the projects/commit ids on success, otherwise None ++ fatal - True if caller should exit, Or False + Added _SkipUpdatingProjectRevisionId a helper function to see if a project's revision id needs to be updated or not. This function is used to exclude projects from local manifest file. + Added the following error events into git_trace2_event_log ++ If superproject is missing in a manifest ++ If there are missing commit ids for projects. command.py: + Deleted unused import - platform + Added git_trace2_event_log as a member so all subcmds can log error events. main.py: + Initialized git_trace2_event_log as a member of command object. init.py: + Deleted unused import - optparse init.py: + Called sys.exit only if Sync returns exit=True sync.py: + Called sys.exit only if Superproject's UpdateProjectsRevisionId returns exit=True + Reloaded the manifest only if manifest path is returned by UpdateProjectsRevisionId. If not, fall back to the old way of doing repo sync. test_git_superproject: + Added code to verify error events are being logged. + Added a test for no superproject tag + Added test for UpdateProjectsRevisionId not updating the revision id with the commit ids. Tested the code with the following commands. + Positive test case with aosp-master. $ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in .../android/aosp $ repo_dev sync -j40 --use-superproject remote: Total 12 (delta 4), reused 12 (delta 4) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. ... repo sync has finished successfully. + Negative test case without superproject tag. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml error: Cannot get project commit ids from manifest error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.589s repo sync has finished successfully. + Test for missing commit_id for a project. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint'] error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.364s repo sync has finished successfully. $ ./run_tests -v ... ...== 164 passed in 2.87s ==... Bug: [google internal] b/189371541 Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* commands: document the "common" class attributeMike Frysinger2021-06-151-1/+1
| | | | | | | | | | Switch it to uppercase to make it clear it's a constant, and add documentation so its usage is clear. Change-Id: I6d281a66a90b5908b3131585c9945e88cfe815ea Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309322 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: Sync submodules correctly againv2.15.3Peter Kjellerstedt2021-05-201-3/+9
| | | | | | | | | | | | | Commit b4429439 (sync: refactor main fetch loop) broke syncing of submodules with sync-s="true". The first `repo sync` would just fetch the superrepo, while the second `repo sync` would fetch the submodules. This was due to the new _FetchMain() function not passing back the all_projects variable that it had modified. Change-Id: Ie8187cde7bb894e4e9a6b76c2aed83873d9f69a4 Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/307065 Reviewed-by: Mike Frysinger <vapier@google.com>
* ssh: rewrite proxy management for multiprocessing usagev2.15Mike Frysinger2021-05-101-12/+32
| | | | | | | | | | | | | | | | We changed sync to use multiprocessing for parallel work. This broke the ssh proxy code as it's all based on threads. Rewrite the logic to be multiprocessing safe. Now instead of the module acting as a stateful object, callers have to instantiate a new ProxyManager class that holds all the state, an pass that down to any users. Bug: https://crbug.com/gerrit/12389 Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* ssh: move proxy usage to the sync subcommandMike Frysinger2021-05-101-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | The only time we really need ssh proxies is when we want to run many connections and reuse them. That only happens when running sync. Every other command makes at most two connections, and even then it's only one or none. So the effort of setting up & tearing down ssh proxies isn't worth it most of the time. The big reason we want to move this logic to sync is that it's now using multiprocessing for parallel work. The current ssh proxy code is all based on threads, which means none of the logic is working correctly. The current ssh design makes it hard to fix when all of the state lives in the global/module scope. So the first step to fixing this is top move the setup & teardown to the one place that really needs it: sync. No other commands will use proxies anymore, just direct connections. Bug: https://crbug.com/gerrit/12389 Change-Id: Ibd351acdec39a87562b3013637c5df4ea34e03c6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305485 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: added --no-use-superproject to disable superproject.Raman Tenneti2021-05-081-3/+7
| | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v $ repo_dev sync -c -j8 --no-use-superproject Fetching: 100% (1041/1041), done in 1m22.743s $ repo_dev sync -c -j8 --use-superproject WARNING: --use-superproject is experimental and not for general use .. Bug: [google internal] b/187459275 Change-Id: I3f4269df38cd24a21723e8b2be5a1f013e7b5a91 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305682 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Recommend using --no-use-superproject if sync fails.Raman Tenneti2021-05-071-1/+2
| | | | | | | | | | | | | | | | | | If superproject was not available for a branch, then the next repo sync would also fail because --use-superproject is remembered across repo init. In such cases, hoping the hint to to use --no-use-superproject will help. Tested the code with the following commands and by forcing a failure. $ ./run_tests -v Bug: [google internal] b/187459275 Change-Id: Ie250812b7ba83afc230b5b1d154ba11f245f8b8a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305622 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: refactor main fetch loopMike Frysinger2021-05-061-42/+59
| | | | | | | | | | | | This is a large chunk of code that is largely isolated. Move it into a class method to make it easier to manage & reason about, and in a follow up CL, easier to scope. Bug: https://crbug.com/gerrit/12389 Change-Id: I0c69d95a9e03478d347b761580b2343bffa012d5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305484 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* sync: fix recursive fetchingMike Frysinger2021-05-051-1/+1
| | | | | | | | | | | | | Commit b2fa30a2b891b22c173c960a67bf38ccbba8de1b ("sync: switch network fetch to multiprocessing") accidentally changed the variable passed to the 2nd fetch call from |missing| to |to_fetch| due to a copy & paste of the earlier changed logic. Undo that to fix git submodule fetching. Bug: https://crbug.com/gerrit/14489 Change-Id: I627954f80fd2e80d9d5809b530aa6b0ef9260abb Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305262 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use manifest_name passed inMike Frysinger2021-05-041-1/+1
| | | | | | | | | | | | | | Commit fb527e3f522a9f6ae05ee1fd707fb0b45c9190f2 ("sync: create dedicated manifest project update func") refactored code from the main body into a dedicated method. The manifest_name was passed as an argument, but never used it, and instead reaches back out to the command line options. This ignores the logic in the main loop where manifest_name might have changed (like when using smart sync). Change-Id: I4b84638fbb10c2b6f8f4b555e1475b0669c2daf4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305148 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: switch to multiprocessing.EventMike Frysinger2021-05-041-1/+1
| | | | | | | | | | | We've switched most of this command over to multiprocessing and off of _threading, so do the Event object too. The APIs are the same between the modules, so we shouldn't need to update anything else. Change-Id: I52d31f1c6ef2bcbe7bbc1dd1add79a8d5d08784a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305147 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: fix logic error with linkfile errorsMike Frysinger2021-05-041-2/+2
| | | | | | | | | | Make sure err_update_linkfiles is always initalized. Bug: https://crbug.com/gerrit/11008 Change-Id: I7bdd91f82507608ef967daf0fa0f9c859454e19f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305146 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* subcmds: force consistent help text formatMike Frysinger2021-05-041-4/+5
| | | | | | | | | | | | We're inconsistent with help text as to whether it uses title case and whether it ends in a period. Add a test to enforce a standard, and use the style that Python optparse & argparse use themselves (e.g. with the --help option): always lowercase, and never trailing period. Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init/sync: add --no-tags to match --tagsMike Frysinger2021-05-041-1/+4
| | | | | | | | | | | | | While this provides a way to undo earlier command line options (e.g. `repo sync --tags --no-tags`) which can be helpful for scripting & automation, this more importantly allows the user to override the manifest settings for syncing tags from a project. Bug: https://crbug.com/gerrit/12401 Change-Id: Id4c36cd82e6ca7cb073b5d63a09f6c7ccdebba83 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304904 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* add --no-current-branch option to invert --current-branchMike Frysinger2021-05-041-0/+3
| | | | | | | | | | | | | | | | | For most commands, this is more about providing a way to undo earlier command line options (e.g. `repo info -c --no-current-branch`) which can be helpful for scripting & automation. But for the sync command, this is helpful to undo the setting that exists in the manifest itself. With this in place, tweak the sync current_branch_only logic to only apply the manifest settings when the user hasn't specified a command line option. Bug: https://crbug.com/gerrit/12401 Change-Id: I21e2384624680cc740d1b5d1e49c50589d2fe6a0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304903 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Don't update the commit ids of projects if remote is different.Raman Tenneti2021-05-031-6/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | 1) Skip setting the revision id (commit id) for the projects whose remote doesn't match superproject's remote. 2) exp-superproject/superproject_override.xml includes local_manfiest's projects. When we load this XML, don't reload projects from local.xml (otherwise we will get duplicate projects errors). Tested the code with the following commands. $ ./run_tests -v + Test with local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml .repo/local_manifests/local.xml $ repo_dev sync -c -j8 + Test without local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j8 Bug: [google internal] b/186395810 Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Update copyfile and linkfile if manifest updatedjiajia tang2021-05-011-0/+64
| | | | | | | | | | | | | | | | | | | | | Currently, copyfiles and linkfiles which marked by "<copyfile/>" and "<linkfile/>" in manifest will be created by first exec 'repo sync'. But if some "<copyfile/>" or "<linkfile/>" are removed in manifest, then 'repo sync', these removed item dest can not be removed in the sourcecode workspace. This patch is intent to fix this issue, by save a 'copy-link-files.json' in .repo and then compared with new dest path when next sync. If any "<copyfile/>" or "<linkfile/>" were removed, the dest path will be removed in sourcecode at the same time. Bug: https://crbug.com/gerrit/11008 Change-Id: I6b7b41e94df0f9e6e52801ec755951a4c572d05d Signed-off-by: jiajia tang <tangjiajia@xiaomi.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304202 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: fix missing import for -qv2.14.1Mike Frysinger2021-04-211-1/+1
| | | | | | | | | | Some refactors during review dropped this import when it was reworked, but it's still needed when using the --quiet setting. Change-Id: I6d9302ef5a056e52415ea63f35bad592b9dfa75d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303942 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: only print error.GitError, don't raise that exception.Raman Tenneti2021-04-151-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | In _FetchOne & _CheckOne, only print error.GitError exception, but other exceptions are still thrown Fixes the GitError exceptions from /usr/lib/python3.8/multiprocessing/pool.py exiting the repo sync. Tested the code with the following commands and verified repo sync continues after fetch error because of an invalid SHA1. $ ./run_tests -v $ python3 ~/work/repo/git-repo/repo sync -m manifest_P21623846.xml -j32 ... error.GitError: Cannot fetch platform/vendor/google_devices/redbull/proprietary update-ref: fatal: d5a99e518f09d6abb0c0dfa899594e1ea6232459^0: not a valid SHA1 .... An error like the following when jobs=1 error.GitError: Cannot checkout platform/vendor/qcom/sdm845/proprietary/qcrilOemHook: Cannot initialize work tree for platform/vendor/qcom/sdm845/proprietary/qcrilOemHook Bug: https://crbug.com/gerrit/14392 Change-Id: I8922ad6c07c733125419f5698b0f7e32d70c7905 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303544 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* command: add a helper for the parallel execution boilerplateMike Frysinger2021-04-151-30/+19
| | | | | | | | | | | Now that we have a bunch of subcommands doing parallel execution, a common pattern arises that we can factor out for most of them. We leave forall alone as it's a bit too complicated atm to cut over. Change-Id: I3617a4f7c66142bcd1ab030cb4cca698a65010ac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301942 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* progress: hide progress bar when --quietMike Frysinger2021-04-131-3/+3
| | | | | | | | | | | | We want progress bars in the default output mode, but not when the user specifies --quiet. Add a setting to the Progress bar class so it takes care of not displaying anything itself rather than having to update every subcommand to conditionally setup & call the object. Change-Id: I1134993bffc5437bc22e26be11a512125f10597f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303225 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* command: make --verbose/--quiet available to all subcommandsMike Frysinger2021-04-131-11/+3
| | | | | | | | | | | Add new CommonOptions entry points to move the existing --jobs to, and relocate all --verbose/--quiet options to that. This provides both a consistent interface for users as well as for code. Change-Id: Ifaf83b88872421f4749b073c472b4a67ca6c0437 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303224 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: Added --partial-clone-exclude option.Raman Tenneti2021-04-131-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | partial-clone-exclude option excludes projects during partial clone. This is a comma-delimited project names (from manifest.xml). This option is persisted and it is used by the sync command. A project that has been unparital'ed will remain unpartial if that project's name is specified in the --partial-clone-exclude option. The project name should match exactly. Added $ ./run_tests -v Bug: [google internal] b/175712967 "I can't "unpartial" my androidx-main checkout" $ rm -rf androidx-main/ $ mkdir androidx-main/ $ cd androidx-main/ $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M -m default.xml $ repo_dev sync -c -j8 + Verify a project is partial $ cd frameworks/support/ $ git config -l | grep 'partial' + Unpartial a project. $ /google/bin/releases/android/git_repack/git_unpartial + Verify project is unpartial $ git config -l | grep 'partial' $ cd ../.. + Exclude the project from being unparial'ed after init and sync. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude="platform/frameworks/support,platform/frameworks/support-golden" -m default.xml + Verify project is unpartial $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' $ cd ../.. + Remove the project from exclude list and verify that project is partially cloned. $ repo_dev init -u https://android.googlesource.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --partial-clone-exclude= -m default.xml $ repo_dev sync -c -j8 $ cd frameworks/support/ $ git config -l | grep 'partial' Change-Id: Id5dba418eba1d3f54b54e826000406534c0ec196 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303162 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: add separate --jobs options for different stepsMike Frysinger2021-04-091-4/+11
| | | | | | | | | | | | | | | The number of jobs one wants to run against the network tends to factor differently from the number of jobs one wants to run when checking out local projects. The former is constrained by your internet connection & server limits while the later is constrained by your local computer's CPU & storage I/O. People with beefier computers probably want to keep the network/server jobs bounded a bit lower than the local/checkout jobs. Change-Id: Ia27ab682c62c09d244a8a1427b1c65acf0116c1c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302804 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: add progress bar to garbage collection phaseMike Frysinger2021-04-091-6/+14
| | | | | | | | | | This can take a few seconds, if not a lot more, so add a progress bar so users understand what's going on. Change-Id: I5b4b54c1bbb9ec18728f979521310f7087afaa5c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/302802 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: rework selfupdate logicMike Frysinger2021-04-091-48/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current logic has a downside in that it doesn't sync to the latest signed version available if the latest commit itself is unsigned. This can come up when using the "main" branch as it is sometimes signed, but often not as it's holding the latest merged commits. When people use the main branch, it's to get early testing on versions tagged but not yet released, and we don't want them to get stuck indefinitely on that old version of repo. For example, this series of events: * "stable" is at v2.12. * "main" is tagged with v2.13. * early testers use --repo-rev main to get v2.13. * new commits are merged to "main". * "main" is tagged with v2.14. * new commits are merged to "main". * devs who had synced in the past to test v2.13 are stuck on v2.13. repo sees "main" is unsigned and so doesn't try to upgrade at all. The only way to get unwedged is to re-run `repo init --repo-rev main`, or to manually sync once with repo verification disabled, or for us to leave "main" signed for a while and hope devs will sync in that window. The new logic is that whenever changes are available, we switch to the latest signed tag. We also replace some of the duplicated verification code in the sync command with the newer wrapper logic. This handles a couple of important scenarios inaddition to above: * rollback (e.g. v2.13.8 -> v2.13.7) * do not trash uncommitted changes (in case of ad-hoc testing) * switch tag histories (e.g. v2.13.8 -> v2.13.8-cr1) Change-Id: I5b45ba1dd26a7c582700ee3711f303dc7538579b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300122 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: switch network fetch to multiprocessingMike Frysinger2021-04-011-124/+91
| | | | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. This reworks the fetch logic to return results for processing in the main thread instead of leaving every thread to do its own processing. We have to tweak the chunking logic a little here because multiprocessing favors batching over returning immediate results when using a larger value for chunksize. When a single job can be quite slow, this tradeoff is not good UX. Bug: https://crbug.com/gerrit/12389 Change-Id: I0f0512d15ad7332d1eb28aff52c29d378acc9e1d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298642 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: imply -c if --use-superproject option is used.Raman Tenneti2021-03-241-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/183232698 Bug: https://crbug.com/gerrit/13707 $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync --use-superproject $ repo_dev sync real 0m8.046s user 0m2.866s sys 0m2.457s Second time repo sync took only 8 seconds and verified by printing that urrent_branch_only is True in project.py's Sync_NetworkHalf function. Change-Id: Ic48efb23ea427dfa36e12a5c49973d6ae776d818 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/301182 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: fix reporting of failed local checkoutsv2.13.6Mike Frysinger2021-03-151-1/+1
| | | | | | | | | | | The refactor to multiprocessing broke status reporting slightly when checking out projects. Make sure we mark the step as failed if any of the projects failed, not just when --fail-fast is set. Change-Id: I0efb56ce83b068b2c334046df3fef23d797599c9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299882 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Display status messages during repo init/sync.v2.13.3Raman Tenneti2021-03-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Superproject objects accept the optional argument “quiet”. The following progress messages are displayed if quiet is false. Displayed the following message whenever we find we have to make a new folder (aka new remote), because if you started with repo init android and later do googleplex-android that is when it will be slow. "<location>: Performing initial setup for superproject; this might take several minutes.". After fetch completion, added the following notification: "<location>: Initial setup for superproject completed." Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject Bug: [google internal] b/181178282 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: switch local checkout to multiprocessingMike Frysinger2021-02-271-124/+52
| | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down to ~4 sec with -j8. Not a big deal, but shows that this actually works to speed things up unlike the threading model. This reworks the checkout logic to return results for processing in the main thread instead of leaving every thread to do its own processing. Bug: https://crbug.com/gerrit/12389 Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: capture all git output by defaultMike Frysinger2021-02-251-0/+7
| | | | | | | | | | | | The default sync output should show a progress bar only for successful commands, and the error output for any commands that fail. Implement that policy here. Bug: https://crbug.com/gerrit/11293 Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress/sync: include active number of jobsMike Frysinger2021-02-251-11/+16
| | | | | | | | | | Provide a bit more info to users that things are actively running. Bug: https://crbug.com/gerrit/11293 Change-Id: Ie8eeaa8804d1ca71cf5c78ad850fa2d17d26208c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297904 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress: always enable always_print_percentageMike Frysinger2021-02-251-2/+1
| | | | | | | | | | | | | | | | | | | The idea for skipping some progress updates was to avoid spending too much time on the progress bar itself. Unfortunately, for large projects (100s if not 1000s) of repos, we get into the situation with large/slow checkouts that we skip showing updates when a repo finishes, but not enough repos finished to increase the percent. Since the progress bar should be relatively fast compared to the actual network & local dick operations, have it show an update whenever the caller requests it. A test with ~1000 repos shows that the progress bar in total adds <100ms. Bug: https://crbug.com/gerrit/11293 Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use superproject if manifest's config has superproject enabled.Raman Tenneti2021-02-251-1/+3
| | | | | | | | | | | | | | | | | | | If --use-superproject is passed as argument to "repo init", then --use-superproject need not be specified during "repo sync". Tested the code with the following commands. $ time repo_dev sync -c -j32 ... WARNING: --use-superproject is experimental and not for general use Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Ian Kasprzak <iankaz@google.com>
* sync: update event is_set APIMike Frysinger2021-02-231-7/+7
| | | | | | | | | Python 3 renamed this method from isSet to is_set. Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* command: unify --job option & default valuesMike Frysinger2021-02-221-5/+4
| | | | | | | | | | | | Extend the Command class to support adding the --jobs option to the parser if the command declares it supports running in parallel. Also pull the default value used for the number of local jobs into the command module so local commands can share it. Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* trim redundant pass statementsMike Frysinger2021-02-161-1/+0
| | | | | | | | | Clean up a few linter warnings. Change-Id: I531d0263a202435d32d83d87ec24998f4051639c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use subprocess.run to verify tagsMike Frysinger2021-02-161-13/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-111-19/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* diffmanifests/sync: simplify repodir lookupMike Frysinger2021-02-111-1/+1
| | | | | | | | | | We have access to repodir on the command object itself, so we don't need to pull it indirectly out of the manifest object. Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-081-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: superproject performance changes.Raman Tenneti2021-02-071-36/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After updating all project’s revsionIds with the SHAs from superproject, write the updated manifest into superproject_override.xml file. Reload that file for future Reloads. This file is created in exp-superproject directory. Moved most of the code that is superproject specific into git_superproject.py and wrote test code. If git pull fails, did a git clone of the superproject. We saw performance gains for consecutive repo sync's. The time to sync went down from around 120 secs to 40 secs when repo sync is executed consecutively. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. First run $ time repo sync --use-superproject ... real 21m3.745s user 97m59.380s sys 19m11.286s After two consecutive sync runs $ time repo sync -c -j8 --use-superproject real 0m39.626s user 0m29.937s sys 0m38.155s Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: Disable info about disabling pruning when quietAnders Björklund2021-01-221-2/+3
| | | | | | | | | | If you have a lot of shared projects, it spams. Bug: https://crbug.com/gerrit/13961 Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049 Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Added --use-superproject option and support for superproject.v2.12Raman Tenneti2021-01-211-1/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Added "--use-superporject" option to sync.py to fetch project SHAs from superproject. If there are any missing projects in superprojects, it prints the missing entries and exits. If there are no missing entries, it will use SHAs from superproject to fetch the projects from git. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and adding <superporject> tag to default.xml. With local modification to the code to print the status, .../WORKING_DIRECTORY$ repo sync --use-superproject repo: executing 'git clone' url: sso://android/platform/superproject repo: executing 'git ls-tree' Success: [] Bug: https://crbug.com/gerrit/13709 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822 Reviewed-by: Mike Frysinger <vapier@google.com>
* drop pyversion & is_python3 checkingMike Frysinger2021-01-061-20/+5
| | | | | | | | | | | We're committed to Python 3 at this point, so purge all the is_python3 related dynamic checks. Bug: https://crbug.com/gerrit/10418 Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* strip python2-only coding:utf-8 & print_function settingsMike Frysinger2021-01-061-4/+0
| | | | | | | | | | We're committed to Python 3 at this point, so clean up boilerplate. Bug: https://crbug.com/gerrit/10418 Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: respect --force-sync when fetching manifest project updatesErwan Yvin2020-11-171-0/+1
| | | | | | | | | | | The --force-sync option was being passed down for all updates except for the manifest project, so add that there too. Bug: https://crbug.com/gerrit/11034 Change-Id: I33818b652f828c6b847dbc70f1fedfac5ac17bbe Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/228146 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-06-26 14:42:13 +0000