summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
* pylint: remove unused importsDaniel Kutik2022-12-131-1/+1
| | | | | | | | | Removed unused imports accross multiple files. Change-Id: Ib5ae4cebf9660e7339b11e3fa592d99f8d51e8d8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354700 Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: fix manifest sync-j handlingMike Frysinger2022-12-081-29/+42
| | | | | | | | | | | | | Since --jobs defaults to 0, not None, we never pull the value out of the manifest. Treat values of 0 and None the same to fix. Bug: http://b/239712300 Bug: http://b/260908907 Change-Id: I9b1026682072366616825fd72f90bd90c10a252f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354254 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Sam Saccone <samccone@google.com>
* sync: Fix undefined variable in _FetchOneKarsten Tausche2022-12-081-2/+3
| | | | | | | | | | | | | | | | If syncing in _FetchOne fails with GitError, sync_result does not get set. There's already a separate local variable for success; do the same for remote_fetched instead of referring to the conditionally defined named tuple. This bug is originally caused by a combination of ad8aa697 "sync: only print error.GitError, don't raise that exception." and 1eddca84 "sync: use namedtuples for internal return values". Change-Id: I0f9dbafb97f8268044e5a56a6f92cf29bc23ca6a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354176 Tested-by: Karsten Tausche <karsten@fairphone.com> Reviewed-by: LaMont Jones <lamontjones@google.com>
* sync: finish marking REPO_AUTO_GC=1 as deprecated.LaMont Jones2022-12-051-5/+3
| | | | | | | | | | The wrong revision of the change was submitted as d793553804c76677444709ebefd70f6e01c29525. Change-Id: I6f3e4993cf40c30ccf0d69020177db8fe5f76b8c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353934 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Sam Saccone <samccone@google.com>
* sync: cleanup output when not doing GCLaMont Jones2022-12-021-13/+14
| | | | | | | | | | | | | Do not use a progress bar when not doing GC, and restrict activity in that case to only repairing preciousObject state. This also includes additional cleanup based on review comments from previous changes. Change-Id: I48581c9d25da358bc7ae15f40e98d55bec142331 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353514 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Revert "sync: save any cruft after calling git gc."LaMont Jones2022-12-021-45/+4
| | | | | | | | | | | | This bug-cacher related code is no longer needed. This reverts commit 891e8f72ce3551a19c377456574bbfbeac5c8b8e. Change-Id: Ia94a2690ff149427fdcafacd39f5008cd60827d5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353774 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Sam Saccone <samccone@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: mark REPO_AUTO_GC=1 as deprecated.LaMont Jones2022-12-021-1/+5
| | | | | | | | | | | | | REPO_AUTO_GC was introduced as a way for users to restore the previous default behavior, since the default changed at the same time as the option was added. As such, it should be marked as deprecated, and removed entirely in a future release. Change-Id: Ib73d98fbea693e7057cc4587928c225a9e4beab2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/353734 Reviewed-by: Sam Saccone <samccone@google.com> Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* subcmds: display correct path multitree messagesLaMont Jones2022-11-151-1/+1
| | | | | | | | | Correct usage of project.relpath for multi manifest workspaces. Change-Id: Idc32873552fcdae6eec7b03dde2b2f31134b72fd Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/347534 Reviewed-by: Xin Li <delphij@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: REPO_AUTO_GC=1 to restore old behavior.LaMont Jones2022-11-101-2/+10
| | | | | | | | | | Add an environment variable to restore previous behavior, since the older version of repo does not support `--auto-gc`. Change-Id: I874dfb8fc3533a97b8adfd52125eb3d1d75e2f3c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/351194 Reviewed-by: Sam Saccone <samccone@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: no garbage collection by defaultLaMont Jones2022-11-081-1/+19
| | | | | | | | | | | Adds --auto-gc and --no-auto-gc (default) options to control sync's behavior around calling `git gc`. Bug: b/184882274 Change-Id: I4d6ca3b233d79566f27e876ab2d79f238ebc12a9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344535 Reviewed-by: Xin Li <delphij@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: clear preciousObjects when set in error.LaMont Jones2022-11-031-21/+75
| | | | | | | | | | | | | | If this is a project that is not using object sharing (there is only one copy of the remote project) then clear preciousObjects. To override this for a project, run: git config --replace-all repo.preservePreciousObjects true Change-Id: If3ea061c631c5ecd44ead84f68576012e2c7405c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350235 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* Set tracing to always on and save to .repo/TRACE_FILE.Joanna Wang2022-11-031-8/+5
| | | | | | | | | | | | - add `--trace_to_stderr` option so stderr will include trace outputs and any other errors that get sent to stderr - while TRACE_FILE will only include trace outputs piggy-backing on: https://gerrit-review.googlesource.com/c/git-repo/+/349154 Change-Id: I3895a84de4b2784f17fac4325521cd5e72e645e2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/350114 Reviewed-by: LaMont Jones <lamontjones@google.com> Tested-by: Joanna Wang <jojwang@google.com>
* sync: uninitialized variable on mirror sync failureLaMont Jones2022-10-261-0/+1
| | | | | | | | | | | | When repo sync fails, if the workspace is a mirror, an uninitialized variable is referenced. Bug: crbug.com/gerrit/16356 Change-Id: I1dba9f92319b9cbfd18460327560a395c88a089f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349654 Reviewed-by: Sam Saccone <samccone@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: do not require python 3.9v2.29.4LaMont Jones2022-10-251-5/+5
| | | | | | | | | | | | Use pre-3.9 syntax for NamedTuple, so that users do not need to have python 3.9 or later installed. Bug: b/255632143, crbug.com/gerrit/16355 Test: manually verified with python 3.8 Change-Id: I488d2d5267ed98d5c55c233cc789e629f1911c9d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/349395 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* sync: only use --cruft when git supports it.v2.29.3LaMont Jones2022-09-221-3/+9
| | | | | | | | | | | git gc --cruft was added in 2.37.0. Bug: https://crbug.com/gerrit/16270 Change-Id: I71e46741e33472a92f16d6f11c51a23e1e55d869 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/346577 Reviewed-by: Emily Shaffer <emilyshaffer@google.com> Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: use namedtuples for internal return valuesLaMont Jones2022-09-191-12/+83
| | | | | | | | | | | | | Replace tuple returns with namedtuples, to simplify adding new fields. Extend the Sync_NetworkHalf return value to: - success: True if successful (the former return value) - remote_fetched: True if we called `git fetch` Change-Id: If63c24c2f849523f77fa19c05bbf23a5e9a20ba9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/344534 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: incorporate review feedback.LaMont Jones2022-09-191-31/+32
| | | | | | | | | | This incorporates feedback from https://gerrit-review.googlesource.com/c/git-repo/+/345114 Change-Id: I04433d6435b967858f1ffb355217d90bc48c1e5d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345894 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: save any cruft after calling git gc.v2.29.2LaMont Jones2022-09-091-4/+44
| | | | | | | | | | | | | | | | | | | This is ENABLED BY DEFAULT due to data corruption potential. To disable it, set REPO_BACKUP_OBJECTS=0 in the environment. While the workspace will grow over time, this provides a recovery path for an issue where objects are erroneously deleted from the workspace, resulting in lost work. Once the root cause is determined, we will be able to stop saving backups again. Backups are kept in .git/objects/.repo/pack.bak Bug: https://crbug.com/gerrit/16247 Change-Id: Ib8b5c9b4bf0dfa9e29606e0f5c881d65996b2a40 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/345114 Reviewed-by: Jonathan Tan <jonathantanmy@google.com> Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Xin Li <delphij@google.com>
* sync: Correctly sync multi manifest workspacesv2.29.1LaMont Jones2022-08-241-6/+5
| | | | | | | | | | | When actually fetching the manifests, start at the correct (sub) manifest. Bug: https://crbug.com/gerrit/16198 Change-Id: I39fdd726f1917ef4277a0b7c83663c8f49167466 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/343914 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: handle smartsync HTML responses betterMike Frysinger2022-07-281-4/+10
| | | | | | | | | | | If the server responds with an HTML page, we should show that to the user instead of crashing with XML errors. Bug: https://crbug.com/gerrit/15936 Change-Id: I52e6b781c3bb6a6c9f6ecbe2e0907044876cdc8d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337519 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: start clearing git settings that disabled pruningLaMont Jones2022-07-251-12/+18
| | | | | | | | | | | | | For projects that no longer share their per-project objects directly, we no longer have to disable the git settings that disable pruning. See commit "project: stop directly sharing objects/ between shared projects" for more details. Bug: https://crbug.com/gerrit/15553 Change-Id: Ica0e83c3002716424c2bc9111b3b3d3a76c30973 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/337535 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: rework --jobs to provide better defaultsMike Frysinger2022-07-251-26/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For --jobs-network, the logic is now: * If the user specifies --jobs-network, use that. * Else, if the user specifies --jobs, use that. * Else, if the manifest specifies sync-j, use that. * Else, default to 1. Then we limit the jobs count based on the softlimit RLIMIT_NOFILE. For --jobs-checkout, the logic is now: * If the user specifies --jobs-checkout, use that. * Else, if the user specifies --jobs, use that. * Else, if the manifest specifies sync-j, use that. * Else, default to DEFAULT_LOCAL_JOBS which is based on user's ncpus. Then we limit the jobs count based on the softlimit RLIMIT_NOFILE. For garbage collecting, the logic is now: * If the user specifies --jobs, use that. * Else, if the manifest specifies sync-j, use that. * Else, default to the user's ncpus. Then we limit the jobs count based on the softlimit RLIMIT_NOFILE. Having to factor in the manifest settings makes this more complicated which is why we delay processing of defaults until after we've synced the manifest projects. Bug: http://b/239712300 Change-Id: Id27cda63c76c156f1d63f6a20cb2c4ceeb3d547c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/341394 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: LaMont Jones <lamontjones@google.com>
* Only sync superproject if it will be used.LaMont Jones2022-06-081-1/+1
| | | | | | | | | | | | If the user says `--no-use-superproject`, then do not bother syncing the superproject. Also add/update docstrings and comments throughout. Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: add multi-manifest supportv2.26LaMont Jones2022-05-261-98/+176
| | | | | | | | | | | | | With this change, partial syncs (sync with a project list) are again supported. If the updated manifest includes new sub manifests, download them inheriting options from the parent manifestProject. Change-Id: Id952f85df2e26d34e38b251973be26434443ff56 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334819 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: refactor use of self.manifestLaMont Jones2022-04-141-68/+82
| | | | | | | | | | | We need to iterate over multiple manifests, and generally use the outer_client.manifest for multi-manifest support. This refactors the use of self.manifest into a chosen manifest. Change-Id: I992f21d610c929675e99555ece9c38df4b635839 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334699 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Stop passing optparse.Values to git_superprojectLaMont Jones2022-04-141-3/+5
| | | | | | | | | | | | | | Make git_superproject independent of the command line by passing the specific value instead of requiring the caller to have an optparse.Values object to pass in. Flag --use-superproject and --archive as incompatible in subcmds/init.py Change-Id: Ied7c874b312e151038df903c8af4328f070f387c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335135 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* manifest_xml: use Superproject to hold XML contentv2.23LaMont Jones2022-04-121-6/+5
| | | | | | | | | | | | | | | Always create Superproject when there is a <superproject> tag, and have it hold the XML content, similar to how other manifest elements are handled. This also adds SetQuiet and SetPrintMessages to Superproject consistent with manifest.SetUseLocalManifests. Change-Id: I522bf3da542006575799f0640c67f7052704f266 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* sync: respect `sync-c` manifest optionDaniel Andersson2022-04-081-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The documentation states that a `sync-c` attribute in the manifest file can set a default for whether only the current branch should be fetched or all branches. This seems to have been broken for some time. Commit 7356114 introduced the `--no-current-branch` CLI option and relied on getting `None` via `optparse` if neither `--current-branch` nor `--no-current-branch` was set to distinguish it from a boolean value. If `None` was received, it would read the value from the manifest option `sync-c`. The parsing went through the utility function `_GetCurrentBranchOnly` which returned `True` if `--current-branch` had been given on the command-line, or fell back on the "superproject" setting, which would either return `True` or `None`. This would incorrectly make `repo` fall back to the manifest setting even if the user had given `--no-current-branch` if no superproject was requested -- the manifest became "too powerful": Command-line Using superproject → `current_branch_only` ------------ ------------------ ----------------------- No From manifest Yes True --current-branch No True --current-branch Yes True --no-current-branch No From manifest ← wrong --no-current-branch Yes True In commit 0cb6e92 the superproject configuration value reading changed from something that could return `None` to something that always returned a boolean. If it returned `False`, this would then incorrectly make `repo` ignore the manifest option even if neither `--current-branch` nor `--no-current-branch` had been given. The manifest default became useless: Command-line Using superproject → `current_branch_only` ------------ ------------------ ----------------------- No False ← wrong Yes True --current-branch No True --current-branch Yes True --no-current-branch No False --no-current-branch Yes True By swapping the order in which the command-line option target and the superproject setting is evaluated, things should work as documented: Command-line Using superproject → `current_branch_only` ------------ ------------------ ----------------------- No From manifest Yes True --current-branch No True --current-branch Yes True --no-current-branch No False --no-current-branch Yes True Change-Id: I933c232d2fbecc6b9bdc364ebac181798bce9175 Tested-by: Daniel Andersson <daniel.r.andersson@volvocars.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334270 Reviewed-by: Mike Frysinger <vapier@google.com>
* manifest_xml: Add Load and Unload methodsLaMont Jones2022-04-081-2/+2
| | | | | | | | | | | | - do not call the internal method from subcmds/sync.py. - use the correct default groups for submanifests. - only sync the superproject when we are told to. Change-Id: I81e4025058f1ee564732b9e17aecc522f6b5f626 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334639 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* Move manifest config logic into ManifestProjectLaMont Jones2022-04-061-1/+1
| | | | | | | | | Use ManifestProject properties for config values. Change-Id: Ib4ad90b0d9a089916e35615b8058942e6d01dc04 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334519 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Add multi-manifest support with <submanifest> elementv2.22LaMont Jones2022-02-171-2/+6
| | | | | | | | | | | To be addressed in another change: - a partial `repo sync` (with a list of projects/paths to sync) requires `--this-tree-only`. Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: With --mirror option, don't display no-use-superproject... message.Raman Tenneti2021-12-071-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | + Display 'Defaulting to no-use-superproject because there is no working tree.' message if --use-superproject option is used and we are not using superproject because manifest is either a mirror or is an archive. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init -u https://android.googlesource.com/mirror/manifest --mirror $ repo_dev sync Receiving objects: 100% (3/3), done.eiving objects: 33% (1/3) $ repo_dev sync --use-superproject Defaulting to no-use-superproject because there is no working tree. Fetching: 0% (0/2158) warming up Bug: https://crbug.com/gerrit/15368 Change-Id: I16b87ee9623315dbc3100b612b1decdaab7ac1dc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/325797 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* init, sync: fixed flake8 warnings.v2.18Raman Tenneti2021-11-181-4/+4
| | | | | | | | | | | | Tested: + run_tests + flake8 subcmds/init.py + flake8 subcmds/sync.py Change-Id: Ie337481d8a210bfc49b0745f75c05a308a0e74d3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/324155 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Inherit --no-use-superproject with --mirror option.Raman Tenneti2021-11-181-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | init.py + Similar to opt.archive, gave an error if --mirror option is used with --use-superproject. sync.py + Defaulted to --no-use-superproject if manifest is a mirror or archive (similar to error at line# 1067). Tested: + run_tests + flake8 (will fix known errors in another CL). $ repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest --use-superproject --mirror Usage: repo init [options] [manifest url] main.py: error: --mirror and --use-superproject cannot be used together. + repo init and repo sync with --mirror and without --mirror options. $ repo_dev init -u https://android.googlesource.com/platform/manifest $ repo_dev sync ...superproject.git: Initial setup for superproject completed. + With --mirror option, verfied there are no exceptions in git_superproject.py Bug: [google internal] b/206537893 Change-Id: I059f20e76f0ab36f0587f29779bb53ede4663bd4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323955 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: link the internal-fs-layout doc into checkoutsMike Frysinger2021-11-151-0/+9
| | | | | | | | | This should make it easy to discover for people poking around .repo/. Change-Id: Ie5051551f25127c0592df5e36efba7bb2263e5d4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323701 Reviewed-by: Jack Neus <jackneus@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: make --prune the defaultMike Frysinger2021-11-051-2/+7
| | | | | | | | | | | | | | | | | | | | | | If a remote deletes a ref, and it points to an object that doesn't exist locally, we can get into a bad state, and the only way for the user to recover is to run `repo sync --prune` (and to know that is the option they need). The error message is not helpful: fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects This situation can also come up when the remote renames refs in a UNIX FS incompatible way. For example, replacing refs/heads/foo with refs/heads/foo/bar. Also add a --no-prune option for users to disable the behavior. Bug: https://issuetracker.google.com/203366450 Change-Id: Icf45d838a10938feb091d29800f7e49240830ec3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322956 Reviewed-by: Andrew Lamb <andrewlamb@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: fix --tags optionMike Frysinger2021-11-051-3/+2
| | | | | | | | | | | | This has been broken since it was added where --tags was actually the same as --no-tags. Oddly, it was copied from init where the logic is correct. Bug: https://crbug.com/gerrit/12401 Change-Id: I15b89da1a655176a11bebc22573b25c728055328 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322955 Reviewed-by: Andrew Lamb <andrewlamb@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: added 'implies -c' in the help of --use-superproject option.Raman Tenneti2021-11-011-2/+1
| | | | | | | | | | | | | | sync.py: deleted unused import errno. Tested: $ ./run_tests $ flake8 repo subcmds/sync.py Bug: https://crbug.com/gerrit/15208 Change-Id: I2bb3098f5602ded3861e000100766041ad93b53d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322555 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: properly handle standalone manifests for `sync` commandv2.17.2Jack Neus2021-10-151-2/+4
| | | | | | | | | | | | | | sync should not attempt to sync the manifest project if it was created from a standalone manifest. The current work around is to run sync with --nmu. BUG=none TEST=manual runs Change-Id: I2e121af0badf9642143e77c7af89d1c2d993b0f3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/321195 Tested-by: Jack Neus <jackneus@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* subcmds/sync: Use pack-refs instead of gc for redundant gitdirs.Allen Webb2021-10-141-10/+24
| | | | | | | | | | | | | Previously `git gc` was being run on every gitdir even when they shared the same objects. Instead only call it once and use pack-refs for the gitdirs that were not gc'ed. Bug: https://crbug.com/gerrit/15113 Test: repo sync -j # and check that git pack-refs is called Change-Id: Icff37ab3ec78cfb44391d8cc7f2d875991532320 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/320275 Tested-by: Allen Webb <allenwebb@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* subcmds/sync: Disable autoDetach for git gc.Allen Webb2021-10-051-0/+1
| | | | | | | | | | | | | | | gc.autoDetach is enabled by default which makes 'git gc --auto' return immediately and run in background. This can lead to a pile up of operations all using large amounts of memory at the same time. To avoid this set gc.autoDetach to false so that the garbage collect task waits for instances to finish before spawning more. Bug: https://crbug.com/gerrit/15113 Test: repo sync -j # and check the number of 'git gc' processes Change-Id: Ic0815156ba3db03972968f33f6f9f51e4928f23b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319835 Tested-by: Allen Webb <allenwebb@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* make file removal a bit more robustMike Frysinger2021-09-281-15/+5
| | | | | | | | | | | | Some of the file removal calls are subject to race conditions (if something else deletes the file), so extend our remove API to have an option to ignore ENOENT errors. Then update a bunch of random call sites to use this new functionality. Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195 Reviewed-by: Sean McAllister <smcallis@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: Log repo sync state events as 'data' events.Raman Tenneti2021-09-141-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git_trace2_event_log.py: + Added LogDataConfigEvents method to log 'data' events. Sync's current_sync_state and previous_sync_state are logged as 'data' events in the current log. It logs are key/value in the |config| argument. Each key is prefixed with |prefix| argument. The following are sample events that are logged during repo sync. {"event":"data", "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0", "thread":"MainThread", "time":"2021-09-14T18:16:19.935846Z", "key":"previous_sync_state/repo.syncstate.main.synctime", "value":"2021-09-14T17:27:11.573717Z"} {"event":"data", "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0", "thread":"MainThread", "time":"2021-09-14T18:16:19.955546Z", "key":"current_sync_state/repo.syncstate.main.synctime", "value":"2021-09-14T18:16:19.935979Z"} tests/test_git_trace2_event_log.py: + Added unit tests sync.py: + Changed logging calls to LogDataConfigEvents. Tested: $ ./run_tests Tested it by running the following command multiple times. $ repo_dev sync -j 20 repo sync has finished successfully Verified config data is looged in trace2 event logs. Bug: [google internal] b/199758376 Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823 Reviewed-by: Josh Steadmon <steadmon@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: print messages if the manifest has superproject tag.v2.16.5Raman Tenneti2021-08-131-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1) If the manifest has superproject tag (git_master, etc), then display error/warning messages (as it is doing today) 2) If the manifest doesn't have superproject tag (nest, chromeos manifests), then don't display any error/warning messages about superrproject (behave as though user has specified --no-use-superproject). 3) Print error/warning messages if --use-superproject passed as argument to repo sync. 4) No change in behavior for the repo init command. git_superproject.py: + Fixed typo in _WriteManifestFile method name + Superproject accepts print_message as an argument and it defaults to True. All messages that are printed to stderr are controlled by this flag. If it is True, then messages get printed. + Added PrintMessages function which return true if either --use-superproject is specified on the command line or if the manifest has a superproject tag. sync.py: + Displays the warning message if PrintMessgages are enabled and passes that as argument to superproject object. + Added 'hassuperprojecttag' trace2 log entry for analysis. We can find users/branches that are using superproject, but the manifest is missing the superproject tag. Tested: $ ./run_tests + Verified printing of messages with and without superproject tag, with with --use-superproject option. + aosp-master $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest $ repo_dev sync + A manifest without superproject tag. $ repo_dev init -m $(pwd)/manifest_7482982.xml $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml Bug: [google internal] b/196411099 Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: Added logging of repo sync state and config options for analysis.Raman Tenneti2021-07-291-3/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git_config.py: + Added SyncAnalysisState class, which saves the following data into the config object. ++ sys.argv, options, superproject's logging data. ++ repo.*, branch.* and remote.* parameters from config object. ++ current time as synctime. ++ Version number of the object. + All the keys for the above data are prepended with 'repo.syncstate.' + Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods to GitConfig object to save/get the above data. git_trace2_event_log.py: + Added LogConfigEvents method with code from DefParamRepoEvents to log events. sync.py: + superproject_logging_data is a dictionary that collects all the superproject data that is to be logged as trace2 event. + Sync at the end logs the previously saved syncstate.* parameters as previous_sync_state. Then it calls config's UpdateSyncAnalysisState to save and log all the current options, superproject logged data. docs/internal-fs-layout.md: + Added doc string explaining [repo.syncstate ...] sections of .repo/manifests.git/config file. test_git_config.py: + Added unit test for the new methods of GitConfig object. Tested: $ ./run_tests $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest Tested it by running the following command multiple times. $ repo_dev sync -j 20 repo sync has finished successfully Verified config file has [syncstate ...] data saved. Bug: [google internal] b/188573450 Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Xin Li <delphij@google.com>
* superproject: Skip updating of superproject when -l is used with sync.Raman Tenneti2021-07-281-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Skip updating the superproject when -l is present and use the existing superproject, if available (this would make sync -l work as it's intended to do), and fall back to sync without superproject when not (this would catch the case when superproject is enabled by automatic rollout). Tested: $ repo sync -j 20 -n NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` /usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. Fetching: 100% (1032/1032), done in 41.184s ... $ repo_dev sync -j 20 -l prebuilts/asuite/: discarding 1 commits prebuilts/runtime/: discarding 1 commits ... repo sync has finished successfully. + With superproject-override.xml and test it. $ ls -l .repo/exp-superproject/ total 176 drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l ... repo sync has finished successfully. + Rename the file superproject-override.xml and test it. $ ls -l .repo/exp-superproject/ total 176 drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml $ repo_dev sync -j 20 -l Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits prebuilts/remoteexecution-client/: discarding 1 commits Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/.... .... Checking out: 100% (1031/1031), done in 5.478s repo sync has finished successfully. Bug: [google internal] b/184368268 Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* Update manifest_name with the returned manifest name for superproject.v2.16.2Xin Li2021-07-141-3/+1
| | | | | | | | | | | This is similar to smart sync, allowing sync to benefit from the patched manifest. Bug: [google internal] b/190688390 Change-Id: I158a80afceca606dcd81ec76b2caede369f7ed03 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312142 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Xin Li <delphij@google.com>
* superproject - More friendly user message when superproject failed.v2.16.1Raman Tenneti2021-07-141-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | superproject is going to be default for some users. This change doesn't fail for repo init or repo sync if source couldn't be synced because of errors in superproject and superproject=true in the config file. The commands will fail if --use-superproject is specified on the command line explicitly. The error messages are logged with trace2 event logs and will be monitored. + sync - When there are errors with superproject and git_superproject says it is fatal failure, sync will exit only when --use-superproject option is specified on the command line. + init - command doesn't fail *if there are any superproject errors), but it will print a warning message and logs message via trace2 event logs. For fatal errors, init will exit only when --use-superproject option is specified on the command line. + All git commands log the command that is being executed so trace2 event logs will know the manifest, remote url and the branch name. There is no functional change other than fatal errors are honored with --use-supeproject option with init/sync commands. Tested the code with the following commands. $ ./run_tests -v Test 1 - sync'ing without errors -------------------------------- Added the following lines to '~/.repoconfig/config [repo] superproject = true $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test $ repo_dev sync remote: Total 4 (delta 1), reused 4 (delta 1) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` /usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. ... Test 2 - init and sync fail when --use-superproject option is passed -------------------------------------------------------------------- $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject remote: Total 57 (delta 16), reused 56 (delta 16) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: Cannot get project commit ids from manifest warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning. Test 3 - git fetch command fails and git command is printed ----------------------------------------------------------- With config change $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 ... NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes. repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in .... Test 4 - no superproject tag ----------------------------- $ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev ... NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in ... $ repo_dev sync NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml warning: Cannot get project commit ids from manifest warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning. Bug: [google internal] b/192614798 Bug: [google internal] b/Bug: [google internal] b/192614798 Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502 Reviewed-by: Amith Dsouza <amithds@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* Fix an issue that repo can't see projects declared in a local manifest.Xin Li2021-07-131-1/+1
| | | | | | | | | | | | | | | When loading of superproject failed, we were resetting the manifest to None, and later code would reload the manifest to see if there are submodules, which would load the non-local manifest, causing sync with superproject to fail. Address this by setting the manifest_name to opt.manifest_name instead. Bug: [google internal] b/189139268 Change-Id: I3616512e1c4b73e7eca0d83fd1fc474b825adbbf Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311102 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* sync: dump git output on errorsMike Frysinger2021-07-041-1/+1
| | | | | | | | Bug: https://crbug.com/gerrit/14700 Change-Id: I1ae53dc7f3792b7e8f11d73f706864fb6591eee8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311142 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-06-25 15:12:06 +0000