summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* sync: Refresh index before updating repoJosip Sokcevic2025-03-131-0/+2
| | | | | | | | | | | | | If the repo index is stale, reset --keep will refuse to reset workspace. An index can be stale if there are any modifications to file node, including mtime, atime, ownership changes, etc. Bug: b/375423099 Change-Id: Ibef03d9d8d2babbb107041707281687342ab7a77 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/460022 Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Tested-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Scott Lee <ddoman@google.com>
* gc: Add tags to remote pack listv2.52Josip Sokcevic2025-02-051-0/+2
| | | | | | | | | | | | | | | | | | | | If tags are omitted from the remote pack list, they must be present in local pack. However, local packs don't have promisor objects, meaning that all blobs must be available locally, and therefore all missing blobs will be downloaded during rev-list phase. Git downloads those sequentially, by invokving fetch operation (rev-list/fetch). Instead of downloading tags' blobs, instruct Git to include all tags in remote rev-list operation. This change was tested with `git fsck --all`. R=yiwzhang@google.com Bug: b/392732561 Change-Id: Id94a40aebbe4f084c952329583d559d296db1a11 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/451422 Reviewed-by: Yiwei Zhang <yiwzhang@google.com> Tested-by: Josip Sokcevic <sokcevic@chromium.org> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
* Activate submodulesKaushik Lingarkar2025-02-041-0/+22
| | | | | | | | | | | | | | This change moves further towards ensuring Git can understand repo's submodules. 'submodule init' is used to make the submodules active[1]. [1] https://git-scm.com/docs/gitsubmodules#_active_submodules Change-Id: I0c20ff1991101fc5be171e566d8fb644aab47200 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446182 Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com> Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Mike Frysinger <vapier@google.com>
* Use 'gitfile' in submodule checkoutsKaushik Lingarkar2025-02-041-7/+42
| | | | | | | | | | | | | | | | | | This change takes another step towards ensuring Git can understand repo's submodules to some extent. Replace the old '.git' symlink with gitfile[1] pointing to the bare checkout of the submodule. This is required for Git's 'recurse submodules' opts to work with repo's submodules as '.git' is expected to be writable by Git when recursing over submodules. [1] https://git-scm.com/docs/gitrepository-layout#_description Change-Id: I52d15451768ee7bd6db289f4d2b3be5907370d42 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446181 Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
* Update internal filesystem layout for submodulesKaushik Lingarkar2025-02-043-2/+34
| | | | | | | | | | | | | | Change the bare checkout directory for submodules from 'subprojects' to 'modules'. Git expects bare submodule checkouts to be in the 'modules' directory. If old subproject directories are found, they will be migrated to the new modules directory. This change is the first step in ensuring Git can understand repo's submodules to some extent. Change-Id: I385029f1bb55d040616d970d6ffb4bb856692520 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444881 Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
* Revert "sync: skip network half on repo upgrade"Josip Sokcevic2025-01-311-1/+1
| | | | | | | | | | | | | | | This reverts commit 61224d01fa29bcf54dd6d521e09e09a8c0da77fe. Reason for revert: the manifest will be updated during in the post-upgrade process, and that can result in a missing object in LocalHalf, since NetworkHalf is not skipped. Bug: b/392979411 Change-Id: I8a46e5b54093ed78285c8b30f000bb08a8244179 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450181 Tested-by: Josip Sokcevic <sokcevic@chromium.org> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Scott Lee <ddoman@google.com>
* docs: smart-sync: split out & expand detailsMike Frysinger2025-01-303-20/+131
| | | | | | | | | | | | The existing documentation on smart-sync behavior is a bit light on details, and out of date wrt what the code actually does. Start a dedicated document and fill it out more. Change-Id: I1a8a3ac6edf9291d72182ad55db865035d9b683e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450002 Commit-Queue: Mike Frysinger <vapier@google.com> Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
* run_tests: update to python 3.11 & pytest 8.3.4Mike Frysinger2025-01-301-17/+11
| | | | | | | | Change-Id: Iffe45d85a54dc380cdd37bbbbe64b058eacad0a9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/449901 Reviewed-by: Josip Sokcevic <sokcevic@chromium.org> Tested-by: Mike Frysinger <vapier@google.com> Commit-Queue: Mike Frysinger <vapier@google.com>
* git_config: prefer XDG config locationflexagoon2025-01-231-0/+14
| | | | | | | | | | | | | Currently, repo ignores the XDG path for the git config file, and creates a new one in the user's home directory. This commit changes the behavior to prefer the XDG path if it exists, which matches git behavior and avoids littering the home directory. Bug: 40012443 Change-Id: Icd3ec6db6b0832f47417bbe98ff9461306b51297 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/448385 Tested-by: lmaor xenix <25misha52@gmail.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Recover from errors during read-treev2.51Josip Sokcevic2025-01-161-5/+12
| | | | | | | | | | | | | | | | | | | | | When repo is initializing a git repository, it calls `git read-tree`. During such operation, git is restoring workspace based on the current index. However, some things can go wrong: a user can run out of disk space, or, in case of partial clone, user may no longer reach the remote host. That will leave affected repository in a bad state with partially checked out workspace. The follow up repo sync won't try to fix such state. This change removes .git symlink, which will force the next `repo sync` to redo Git repository setup. Bug: b/363171216 Bug: b/390161127 Change-Id: I57db4b6cae0ef21826dc7cede4d3bf02cfc3d955 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447801 Reviewed-by: Scott Lee <ddoman@google.com> Tested-by: Josip Sokcevic <sokcevic@chromium.org> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
* gc: Add repack optionJosip Sokcevic2025-01-141-2/+160
| | | | | | | | | | | | | When a repository is partially cloned, no longer needed blobs are never removed. To reclaim some of disk space, allow user to pass --repack which affects only repositories with filter=blob:none and if projects are not shared. Change-Id: I0608172c9eff82fb8a6b6ef703eb109fedb7a6cc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447722 Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Tested-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Scott Lee <ddoman@google.com>
* gc: extract deletion from Execute methodJosip Sokcevic2025-01-141-7/+14
| | | | | | | | Change-Id: Icef4f28fbdb9658892611def7589f5eba43c952c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447721 Reviewed-by: Scott Lee <ddoman@google.com> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Tested-by: Josip Sokcevic <sokcevic@chromium.org>
* sync: Handle KeyboardInterrupt during checkoutJosip Sokcevic2025-01-081-0/+2
| | | | | | | | | | | KeyboardInterrupt is handled during NetworkHalf. This patch handles KeyboardInterrupt during LocalHalf. Bug: b/372069163 Change-Id: I26847f7ca3cdf1fe57b265b4f6b18cc8102d2921 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447401 Tested-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Gavin Mak <gavinmak@google.com>
* sync: skip network half on repo upgradeJosip Sokcevic2024-12-181-1/+1
| | | | | | | | | | | | | | | When repo upgrades itself, it will restart itself and rerun sync command. At that point, we know that network half is already done and we can just proceed with local half. R=ddoman@google.com Bug: b/377567091 Change-Id: I77205b1f2df19891597347d55283a617de3c6634 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446201 Reviewed-by: Scott Lee <ddoman@google.com> Tested-by: Josip Sokcevic <sokcevic@chromium.org> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
* gc: Introduce new command to remove old projectsv2.50.1Josip Sokcevic2024-12-184-2/+193
| | | | | | | | | | | | | | | | | | | When projects are removed from manifest, they are only removed from worktree and not from .repo/projects and .repo/project-objects. Keeping data under .repo can be desired if user expects deleted projects to be restored (e.g. checking out a release branch). Android has ongoing effort to remove many stale projects and this change allows users to easily free-up their disk space. Bug: b/344018971 Bug: 40013312 Change-Id: Id23c7524a88082ee6db908f9fd69dcd5d0c4f681 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445921 Reviewed-by: Mike Frysinger <vapier@google.com> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@chromium.org>
* sync: Delete symlinks relative to client topdirv2.50.0v2.50Josip Sokcevic2024-12-161-1/+4
| | | | | | | | | | | | | If repo sync is invoked outside the repo root, and the latest manifest removes symlinks, repo incorrectly tries to remove symlink - it starts from `cwd` instead of the repo root. Bug: b/113935847 Bug: 40010423 Change-Id: Ia50ea70a376e38c94389880f020c80da3c3f453c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445901 Tested-by: Josip Sokcevic <sokcevic@chromium.org> Reviewed-by: Gavin Mak <gavinmak@google.com>
* Add smoke test for subcmd forallFredrik de Groot2024-12-111-0/+156
| | | | | | | | | | | | | | | | | | | After some refactoring earlier, the forall command was broken briefly, returning after only one run instead of after all projects. This test, albeit simple in nature, would have caught that. Due to the somewhat demanding nature of forall, a lot more setup was needed than expected but seems to do its job now so hopefully it catches similar stuff in the future. Change-Id: I51e161ff0e7e31a65401211c376f319bda504532 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445461 Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Fredrik de Groot <fredrik.de.groot@haleytek.com>
* Remove gitc support from repoJosip Sokcevic2024-12-037-168/+12
| | | | | | | | | | gitc is no longer available. Change-Id: I0cbfdf936832f2cdd4876104ae3cc5a6e26154e2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444841 Reviewed-by: Gavin Mak <gavinmak@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com>
* branches: Escape percent signs in branch namesJosip Sokcevic2024-12-031-1/+4
| | | | | | | | | | | | | | If a branch name contains a percent sign, it will be interpreted as a placeholder and color.py will fail to format it. To avoid this, escape the percent signs prior to calling Coloring method. Bug: b/379090488 Change-Id: Id019c776bbf8cbed5c101f2773606f1d32c9e057 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443801 Reviewed-by: Scott Lee <ddoman@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* sync: Do not fail to sync a manifest with no projectsPeter Kjellerstedt2024-11-261-1/+1
| | | | | | | | | | | | | | | | Since commit 454fdaf1191c87e5c770ab865a911e10e600e178 (v2.48), syncing a manifest without any projects would result in: Repo command failed: RepoUnhandledExceptionError Number of processes must be at least 1 Bug: 377546300 Change-Id: Iaa2f6a3ac64542ad65a19c0eef449f53c09cae67 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443442 Reviewed-by: Erik Elmeke <erik@haleytek.corp-partner.google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
* Fix event log command event hierarchy.Josip Sokcevic2024-11-223-18/+16
| | | | | | | | | | | | command should be cmd_name, to match what git is emitting. This also fixes arguments, so that only relevant arguments are passed instead of the entire sys.args, which will contain wrapper information Change-Id: Id436accfff511292ec2c56798fffb2306dda38fc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443741 Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com>
* Make repo installation work without .gitJosip Sokcevic2024-11-181-4/+5
| | | | | | | | | | | Some tools like jj and cog will not have .git. This change makes it possible to run all repo commands in such setups. Change-Id: I7f3845dc970fbaa731c31e0aa48355a4b56ed3a6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/442821 Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* sync: fix connection error on macOSv2.49.3Kuang-che Wu2024-11-061-0/+11
| | | | | | | | | | | | | | | | With a large number of sync workers, the sync process may fail on macOS due to connection errors. The root cause is that multiple workers may attempt to connect to the multiprocessing manager server at the same time when handling the first job. This can lead to connection failures if there are too many pending connections, exceeding the socket listening backlog. Bug: 377538810 Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541 Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
* upload: Return correct tuple values in _ProcessResultsv2.49.2Josip Sokcevic2024-10-311-2/+2
| | | | | | | | | | | | | | | Incorrect tuple values were returned with http://go/grev/440221 - instead of returning (Project, ReviewableBranch), _ProcessResults was returning (int, ReviewableBranch). R=jojwang@google.com Bug: 376731172 Change-Id: I75205f42fd23f5ee6bd8d0c15b18066189b42bd9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441121 Reviewed-by: Sam Saccone <samccone@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com>
* worktree: Do not try to fix relative pathsv2.49.1Allen Webb2024-10-301-18/+23
| | | | | | | | | | | | | --worktree was broken with incorrect paths in the .git files whenever the local copy of git populated gitdir with relative paths instead of absoulte paths. Bug: 376251410 Change-Id: Id32dc1576315218967de2a9bfe43bf7a5a0e7aa6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440801 Commit-Queue: Allen Webb <allenwebb@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Allen Webb <allenwebb@google.com>
* forall: Fix returning results earlyJosip Sokcevic2024-10-301-3/+3
| | | | | | | | | | | | | rc should be returned only after all results are processed. R=jojwang@google.com Bug: b/376454189 Change-Id: I8200b9954240dd3e8e9f2ab82494779a3cb38627 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440901 Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Joanna Wang <jojwang@google.com>
* Use full name of the revision when checking dest-branchv2.49joehsu2024-10-281-2/+5
| | | | | | | | | | | | | | | | | | The manifest usually doesn't sepecify the revision with the full name (e.g. refs/heads/REV). However, when checking if the name of the merge branch, full name is used on the merge branch. The CL use full name of revision when comparing it with the merge branch. Bug: b/370919047 Test: repo upload on a project with `dest-branch` set Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Joe Hsu <joehsu@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* Add REPO_SKIP_SELF_UPDATE check in syncFredrik de Groot2024-10-282-1/+6
| | | | | | | | | | | | | | | | | | | The command _PostRepoFetch will try to self update during repo sync. That is beneficial but adds version uncertainty, fail potential and slow downs in non-interactive scenarios. Conditionally skip the update if env variable REPO_SKIP_SELF_UPDATE is defined. A call to selfupdate works as before, meaning even with the variable set, it will run the update. Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967 Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
* manifest: add optional base check on remove and extendFredrik de Groot2024-10-283-0/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds an optional, built-in checker for guarding against patches hanging on wrong base revisions, which is useful if a lower layer of the manifest changes after a patch was done. When adding a patch with a new revision using extend-project or remove-project/project: C---D---E patches in project bla / A---B project bla in manifest state 1 <extend-project name="bla" revision="E" base-rev="B"> If project bla gets updated, in a new snap ID or by a supplier or similar, to a new state: C---D---E patches in project bla / A---B---F---G project bla in manifest state 2 Parsing will fail because revision of bla is now G, giving the choice to create a new patch branch from G and updating base-rev, or keeping previous branch for some reason and only updating base-rev. Intended for use in a layered manifest with hashed revisions. Named refs like branches and tags also work fine when comparing, but will be misleading if a branch is used as base-rev. Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* [event_log] Stop leaking semaphore resourcesJosip Sokcevic2024-10-241-2/+10
| | | | | | | | | | | | With the global state and fork, we are left with uncleaned resources. Isolate mulitprocessing.Value in a function so we stop the leak. Bug: 353656374 Change-Id: If50bb544bda12b72f00c02bc1d2c0d19de000b88 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440261 Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com>
* progress: always show done messageKuang-che Wu2024-10-241-9/+4
| | | | | | | | | | | | The done message was omitted if the task is shorter than 0.5s. This might confuse users. Bug: b/371638995 Change-Id: I3fdd2cd8daea16d34fba88457d09397fff71af15 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440222 Tested-by: Kuang-che Wu <kcwu@google.com> Commit-Queue: Kuang-che Wu <kcwu@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
* subcmds: reduce multiprocessing serialization overheadKuang-che Wu2024-10-2311-172/+228
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Follow the same approach as 39ffd9977e to reduce serialization overhead. Below benchmarks are tested with 2.7k projects on my workstation (warm cache). git tracing is disabled for benchmark. (seconds) | v2.48 | v2.48 | this CL | this CL | | -j32 | | -j32 ----------------------------------------------------------- with clean tree state: branches (none) | 5.6 | 5.9 | 1.0 | 0.9 status (clean) | 21.3 | 9.4 | 19.4 | 4.7 diff (none) | 7.6 | 7.2 | 5.7 | 2.2 prune (none) | 5.7 | 6.1 | 1.3 | 1.2 abandon (none) | 19.4 | 18.6 | 0.9 | 0.8 upload (none) | 19.7 | 18.7 | 0.9 | 0.8 forall -c true | 7.5 | 7.6 | 0.6 | 0.6 forall -c "git log -1" | 11.3 | 11.1 | 0.6 | 0.6 with branches: start BRANCH --all | 21.9 | 20.3 | 13.6 | 2.6 checkout BRANCH | 29.1 | 27.8 | 1.1 | 1.0 branches (2) | 28.0 | 28.6 | 1.5 | 1.3 abandon BRANCH | 29.2 | 27.5 | 9.7 | 2.2 Bug: b/371638995 Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Kuang-che Wu <kcwu@google.com> Commit-Queue: Kuang-che Wu <kcwu@google.com>
* sync: reduce multiprocessing serialization overheadKuang-che Wu2024-10-232-87/+134
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Background: - Manifest object is large (for projects like Android) in terms of serialization cost and size (more than 1mb). - Lots of Project objects usually share only a few manifest objects. Before this CL, Project objects were passed to workers via function parameters. Function parameters are pickled separately (in chunk). In other words, manifests are serialized again and again. The major serialization overhead of repo sync was O(manifest_size * projects / chunksize) This CL uses following tricks to reduce serialization overhead. - All projects are pickled in one invocation. Because Project objects share manifests, pickle library remembers which objects are already seen and avoid the serialization cost. - Pass the Project objects to workers at worker intialization time. And pass project index as function parameters instead. The number of workers is much smaller than the number of projects. - Worker init state are shared on Linux (fork based). So it requires zero serialization for Project objects. On Linux (fork based), the serialization overhead is O(projects) --- one int per project On Windows (spawn based), the serialization overhead is O(manifest_size * min(workers, projects)) Moreover, use chunksize=1 to avoid the chance that some workers are idle while other workers still have more than one job in their chunk queue. Using 2.7k projects as the baseline, originally "repo sync" no-op sync takes 31s for fetch and 25s for checkout on my Linux workstation. With this CL, it takes 12s for fetch and 1s for checkout. Bug: b/371638995 Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921 Tested-by: Kuang-che Wu <kcwu@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Kuang-che Wu <kcwu@google.com>
* Fix incremental syncs for prjs with submodulesKaushik Lingarkar2024-10-181-1/+3
| | | | | | | | | | | | | | | | | | | When performing an incremental sync (re-running repo init with an updated manifest revision) with --fetch-submodules or sync-s=true, there is an attempt to get a list of all projects (including submodules) before projects are actually fetched. However, we can only list submodules of a project if we have already fetched its revision. Instead of throwing an error when we don't have the revision, assume there are no submodules for that project. In the sync cmd, we already update the list of projects to include submodules after fetching superprojects. Change-Id: I48bc68c48b5b10117356b18f5375d17f9a89ec05 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439761 Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org> Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org>
* sync: Always use WORKER_BATCH_SIZEv2.48Josip Sokcevic2024-10-073-27/+43
| | | | | | | | | | | | | | | | | | | With 551285fa35ccd0836513e9cf64ee8d3372e5e3f4, the comment about number of workers no longer stands - dict is shared among multiprocesses and real time information is available. Using 2.7k projects as the baseline, using chunk size of 4 takes close to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of rougly 8 times which matches the increase. R=gavinmak@google.com Bug: b/371638995 Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523 Commit-Queue: Josip Sokcevic <sokcevic@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com>
* project: Handle git sso auth failures as repo exitJosip Sokcevic2024-10-032-0/+19
| | | | | | | | | | | | | | | | If a user is not authenticated, repo continues execution and it will likely result in more of the same errors being printed. A user is also likely to SIGTERM the process resulting in more errors. This change stops repo sync if any of repositories can't be fetched to Git authentcation using sso helper. We could extend this to all Git authentication Change-Id: I9e471e063450c0a51f25a5e7f12a83064dfb170c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438522 Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* superproject: Remove notice about betaJosip Sokcevic2024-10-031-12/+0
| | | | | | | | | | | It's been the default for Android for over a year now, and it's no longer useful notice. Change-Id: I53c6f1e7cee8c1b2f408e67d3a6732db3b272bee Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438521 Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* project: run fetch --refetch on unable to not parse commitJosip Sokcevic2024-10-031-1/+4
| | | | | | | | | | | | | | | Similarly to e59e2ae757623e64f625a9cdadf1c2010ef82b34, handle missing gc'ed commits by running `git fetch --refetch`. R=jojwang@google.com Bug: b/360889369 Bug: b/371000949 Change-Id: I108b870b855d3b9f23665afa134c6e35f7cd2830 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438461 Reviewed-by: Joanna Wang <jojwang@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* Disable git terminal prompt during fetch/cloneJosip Sokcevic2024-09-262-6/+22
| | | | | | | | | | | | | | | | | | | | git fetch operation may prompt user to enter username and password. This won't be visible to user when repo sync operation since stdout and stderr are redirected. If that happens, user may think repo is doing work and likely won't realize it's stuck on user's input. This patch disables prompt for clone and fetch operations, and repo will fail fast. R=gavinmak@google.com Bug: b/368644181 Change-Id: I2efa88ae66067587a00678eda155d861034b9127 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438001 Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com>
* init: add --manifest-upstream-branchKaushik Lingarkar2024-09-263-2/+28
| | | | | | | | | | | | | | | When a sha1 is provided to '--manifest-branch', the ref which is expected to contain that sha1 can be provided using the new '--manifest-upstream-branch' option. This is useful with '--current-branch' to avoid having to sync all heads and tags, or with a commit that comes from a non-head/tag ref (like a Gerrit change ref). Change-Id: I46a3e255ca69ed9e809039e58b0c163e02af94ef Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436717 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Kaushik Lingarkar <kaushik.lingarkar@linaro.org> Tested-by: Kaushik Lingarkar <kaushik.lingarkar@linaro.org>
* man: regenerate man pagesKaushik Lingarkar2024-09-252-2/+10
| | | | | | | | | Change-Id: Icf697eda7d5dcdc87854ad6adf607353c7ba5ac2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437941 Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Nasser Grainawi <nasser.grainawi@linaro.org> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
* project: Copy and link files even with local branchesBrian Norris2024-09-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the winding maze that constitutes Sync_LocalHalf(), there are paths in which we don't copy-and-link files. Examples include something like: cd some/project/ repo start head . # do some work, make some commit, upload that commit to Gerrit [[ ... in the meantime, someone addes a <linkfile ...> for some/project/ in the manifest ... ]] cd some/project/ git pull --rebase repo sync In this case, we never hit a `repo rebase` case, which might have saved us. Instead, the developer is left confused why some/project/ never had its <linkfile>s created. Notably, this opens up one more corner case in which <linkfile ... /> or <copyfile ... /> could potentially clobber existing work in the destination directory, but there are existing cases where that's true, and frankly, those seem like bigger holes than this new one. Change-Id: I394b0e4529023a8ee319dc25d03d513a19251a4a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437421 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Brian Norris <briannorris@google.com> Commit-Queue: Brian Norris <briannorris@google.com>
* sync: include TARGET_RELEASE when constructing smart sync target.v2.47Navil2024-09-121-0/+13
| | | | | | | | | | | | | | | | | | | | | | When using the smart sync option, we try to construct the target that was "lunched" from the TARGET_PRODUCT and TARGET_BUILD_VARIANT envvars. However, an android target is now made of three parts, {TARGET_PRODUCT}-{TARGET_RELEASE}-{TARGET_BUILD_VARIANT}. I am leaving the option of creating a target if a TARGET_RELEASE is not specified in case there are other consumers who depend on that option. BUG=b:358101714 TEST=./run_tests TEST=smart sync on android repo and manually inspecting smart_sync_override.xml Change-Id: I556137e33558783a86a0631f29756910b4a93d92 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436977 Tested-by: Yiwei Zhang <yiwzhang@google.com> Reviewed-by: Yiwei Zhang <yiwzhang@google.com> Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
* color: fix have_fg not re assign to truebright2024-09-123-0/+83
| | | | | | | | | | | | | | | | In method _parse the value of this variable 'have_fg ' is always False, Maybe reassign it to True is lost. I guess the author’s original intention was: if set some value in gitconfig file(for ex: text = black red ul), the first is bg color, the second is fg color, and the last one is attr. Change-Id: I372698fe625db4c1fdaa94ea7f193a80a850ecb9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425997 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Bright Ma <mmh1989@foxmail.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* tox.ini: Make the lint and format environments run black for all codePeter Kjellerstedt2024-09-121-2/+2
| | | | | | | | | | This matches the extra files specified in run_tests. Change-Id: Ic8999383a17b3ec7ae27322323ea44eeaa40c968 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/434998 Tested-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* tox.ini, constraints.txt: Lock the version of black to <24Peter Kjellerstedt2024-09-122-0/+4
| | | | | | | | | | | The formatting produced by black versions before 24 matches the current formatting of the code. Change-Id: I045f22d2f32a09d4683867293e81512f2abd1036 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/434997 Tested-by: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* project: run fetch --refetch onacould not parse commitJosip Sokcevic2024-09-121-0/+11
| | | | | | | | | | | | | | | | git may gc reachable objects in partial clone repository due to a bug (report: https://lore.kernel.org/git/20240802073143.56731-1-hanyang.tony@bytedance.com/ ). Until git is properly patched and released, force --refetch iff "could not parse commit" is part of git output. --refetch will will ensure that gc'ed objects are retrieved. Bug: b/360889369 Change-Id: I0fc911c591060f859235dcd8d019881106f0858e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437017 Commit-Queue: Josip Sokcevic <sokcevic@google.com> Reviewed-by: Sam Saccone <samccone@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com>
* Add a `--rebase` option to sync commandJeroen Dhollander2024-08-302-3/+21
| | | | | | | | | | | | | | | | Previously repo would abort a sync if there were published changes not merged upstream. The --rebase option allows the modification of published commits. This is a copy of http://go/grev/369694 with the merge conflicts resolved. Bug: 40014610 Change-Id: Idac8199400346327b530abea33f1ed794e5bb4c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435838 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Tested-by: Jeroen Dhollander <jeroendh@google.com> Commit-Queue: Jeroen Dhollander <jeroendh@google.com>
* sync: Fix git command for aborting rebase being called incorrectly.Erik Elmeke2024-08-281-1/+1
| | | | | | | | | | | | The argument list was incorrectly destructured so only the first element of the list was considered a git-cmd, split up by each character in the string. Change-Id: Idee8a95a89a7da8b8addde07135354fc506c2758 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435839 Reviewed-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Erik Elmeke <erik@haleytek.corp-partner.google.com> Tested-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
* project: ignore more curl failure modesv2.46Mike Frysinger2024-07-021-4/+13
| | | | | | | | | | | | | Current clone bundle fetches from Google storage results HTTP/404 and curl exiting 56. This is basically WAI, so stop emitting verbose error output whenever that happens. Also add a few more curl exit statuses based on chromite history, and document them. Change-Id: I3109f8a8a19109ba9bbd62780b40bbcd4fce9b76 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432197 Commit-Queue: Mike Frysinger <vapier@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-14 12:31:21 +0000