summaryrefslogtreecommitdiffstats
path: root/git_superproject.py
Commit message (Collapse)AuthorAgeFilesLines
* superproject: Remove notice about betaJosip Sokcevic2024-10-031-12/+0
| | | | | | | | | | | It's been the default for Android for over a year now, and it's no longer useful notice. Change-Id: I53c6f1e7cee8c1b2f408e67d3a6732db3b272bee Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438521 Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Josip Sokcevic <sokcevic@google.com> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
* git_superproject: tell git that superproject is bareEmily Shaffer2023-11-091-5/+7
| | | | | | | | | | | | | | | | | | | | | | The superproject is initialized as a bare repo in Superproject:_Init(). That means that later operations must treat it as a bare repository, specifying the gitdir and setting 'bare' appropriately when launching GitCommand()s. It's also OK not to specify cwd here because GitCommand() will drop cwd if bare == True anyways. With this change, it's possible to run `repo init` and `repo sync` with the Git config 'safe.bareRepository' set to 'explicit'. This config strengthens Git's security posture against embedded bare repository attacks like https://github.com/justinsteven/advisories/blob/main/2022_git_buried_bare_repos_and_fsmonitor_various_abuses.md. Bug: b/227257481 Change-Id: I954a64c6883d2ca2af9c603e7076fd83b52584e9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/389794 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Jason R. Coombs <jaraco@google.com> Tested-by: Emily Shaffer <emilyshaffer@google.com> Reviewed-by: Emily Shaffer <emilyshaffer@google.com> Commit-Queue: Jason R. Coombs <jaraco@google.com>
* cleanup: convert exceptions to OSErrorJason R. Coombs2023-10-211-1/+1
| | | | | | | | | | | In Python 3, these exceptions were merged into OSError, so switch everything over to that. Change-Id: If876a28b692de5aa5c62a3bdc8c000793ce52c63 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390376 Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com> Commit-Queue: Mike Frysinger <vapier@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* delete Python 2 (object) compatMike Frysinger2023-10-201-1/+1
| | | | | | | | | Bug: 302871152 Change-Id: I39636d73a6e1d69efa8ade74f75c5381651e6dc8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390054 Commit-Queue: Mike Frysinger <vapier@google.com> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* isort: format codebasev2.36Mike Frysinger2023-08-221-2/+4
| | | | | | | | Change-Id: I6f11d123b68fd077f558d3c21349c55c5f251019 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/383715 Reviewed-by: Gavin Mak <gavinmak@google.com> Tested-by: Mike Frysinger <vapier@google.com> Commit-Queue: Mike Frysinger <vapier@google.com>
* Format codebase with black and check formatting in CQGavin Mak2023-03-221-395/+485
| | | | | | | | | | | | Apply rules set by https://gerrit-review.googlesource.com/c/git-repo/+/362954/ across the codebase and fix any lingering errors caught by flake8. Also check black formatting in run_tests (and CQ). Bug: b/267675342 Change-Id: I972d77649dac351150dcfeb1cd1ad0ea2efc1956 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/363474 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Gavin Mak <gavinmak@google.com> Commit-Queue: Gavin Mak <gavinmak@google.com>
* git_superproject: Log actual error fmt instead of the entire error message.Joanna Wang2023-02-011-24/+27
| | | | | | | | Bug: b/258492341 Change-Id: I00678d572712791190ae1ad4e1bcf3cbe04cc1c0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/357114 Tested-by: Joanna Wang <jojwang@google.com> Reviewed-by: Gavin Mak <gavinmak@google.com>
* pylint: remove unused importsDaniel Kutik2022-12-131-1/+1
| | | | | | | | | Removed unused imports accross multiple files. Change-Id: Ib5ae4cebf9660e7339b11e3fa592d99f8d51e8d8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354700 Tested-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Use --negotiation-tip in superproject fetches.Joanna Wang2022-12-091-1/+11
| | | | | | | | Bug: b/260645739 Change-Id: Ib0cdbb13f130b91ab14df9c60a510f1e27cca8e0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/354354 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Joanna Wang <jojwang@google.com>
* Omit local_manifest groups from superproject override.v2.28LaMont Jones2022-07-151-1/+2
| | | | | | | | | | | | | | When we create superproject_override.xml, do not include projects that are present from local_manifests/*. Such projects are fully under the control of the local_manifests/ file. Bug: b/238934278 Test: manual, ./run_tests Change-Id: I40382ceb82d9cf7b8dc7b5f2abed3f6d4d80017e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/340877 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Xin Li <delphij@google.com> Reviewed-by: Sam Saccone 🐐 <samccone@google.com>
* Only sync superproject if it will be used.LaMont Jones2022-06-081-5/+12
| | | | | | | | | | | | If the user says `--no-use-superproject`, then do not bother syncing the superproject. Also add/update docstrings and comments throughout. Change-Id: I9cdad706130501bab9a22d3099a1dae605e9c194 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/338975 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Stop passing optparse.Values to git_superprojectLaMont Jones2022-04-141-7/+17
| | | | | | | | | | | | | | Make git_superproject independent of the command line by passing the specific value instead of requiring the caller to have an optparse.Values object to pass in. Flag --use-superproject and --archive as incompatible in subcmds/init.py Change-Id: Ied7c874b312e151038df903c8af4328f070f387c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/335135 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* git_superproject: only print beta notice once.LaMont Jones2022-04-141-2/+9
| | | | | | | | | This eliminates duplicate notices during multi-manifest syncs. Change-Id: Idcb038ddeb363368637c58c11346ebf8fd2b27ac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334939 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* manifest_xml: use Superproject to hold XML contentv2.23LaMont Jones2022-04-121-28/+41
| | | | | | | | | | | | | | | Always create Superproject when there is a <superproject> tag, and have it hold the XML content, similar to how other manifest elements are handled. This also adds SetQuiet and SetPrintMessages to Superproject consistent with manifest.SetUseLocalManifests. Change-Id: I522bf3da542006575799f0640c67f7052704f266 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334641 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: LaMont Jones <lamontjones@google.com>
* Move manifest config logic into ManifestProjectLaMont Jones2022-04-061-1/+1
| | | | | | | | | Use ManifestProject properties for config values. Change-Id: Ib4ad90b0d9a089916e35615b8058942e6d01dc04 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/334519 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Add multi-manifest support with <submanifest> elementv2.22LaMont Jones2022-02-171-1/+2
| | | | | | | | | | | To be addressed in another change: - a partial `repo sync` (with a list of projects/paths to sync) requires `--this-tree-only`. Change-Id: I6c7400bf001540e9d7694fa70934f8f204cb5f57 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/322657 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Move local-manifest check to manifest_xml.pyLaMont Jones2022-02-151-2/+1
| | | | | | | | | | This removes the need for git_superproject to include manifest_xml, and puts the logic for local_manifest detection in one place. Change-Id: I4d33ded0542ceea4606a1ea24304f678de20c59e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/330499 Tested-by: LaMont Jones <lamontjones@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* Revert "sync: dropped "NOTICE: --use-superproject is in beta ..." message."Raman Tenneti2022-01-061-0/+2
| | | | | | | | | | | | | | | | | This reverts commit d53cb9549a0c57939b12c5c35a6b581aec2ca36f. As long as repo's reference docs treat this feature as a work in progress and don't cover it well enough to allow all repo maintainers to easily support it, it is inconsistent to report to users that it is no longer in beta. Thanks for vapier@google.com for noticing. https://crbug.com/gerrit/15527 tracks the required documentation changes before we'd be ready to roll forward again. Change-Id: Ic9bd951cfb3c1abf6e1bfa30dfe4afa1c9b7bec6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327337 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Jonathan Nieder <jrn@google.com>
* sync: dropped "NOTICE: --use-superproject is in beta ..." message.Raman Tenneti2021-12-291-2/+0
| | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/209511230 Change-Id: Ia3c6de47709f5276e324a5bb608383aba3b2c562 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327197 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Only trigger enrollment logic when manifest have it.Xin Li2021-09-281-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The current code would check for enrollment status when the user did not explicitly specify --[no-]use-superproject and do not have a remembered value in their repo client. However, because superproject only makes sense for manifests that have one specified, we should skip the enrollment logic in that case. Address this by checking manifest.superproject prior to proceeding. This would avoid showing the greeting message of superproject enrollment which can be confusing for developers. Tested: For manifest without superproject: - repo sync --use-superproject will still show message for superproject; - repo sync will not show message regardless of enrollment state For manifest with superproject: - repo sync will show message and perform enrollment if not previously enrolled Bug: https://crbug.com/gerrit/15039 Change-Id: Ic2be9f9d037f0e7cf3446da474a5a0d0e4bd88da Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319255 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Fix indent error which would have prevented choice expiration to work.Xin Li2021-09-271-1/+1
| | | | | | | | Change-Id: I077e05eea23ad58d1dde2c9fe5608660a56d03e5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318815 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Amith Dsouza <amithds@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* superproject: support a new revision attribute.Xin Li2021-09-271-12/+2
| | | | | | | | | | | | | | | | | | | | | | | Tested: $ ./run_tests Verified that a manifest that specified superproject revision would use the specified revision, and superproject will use the default revision. Note that this is a slight behavior change from earlier repo versions, which would always use the branch name of the manifest itself. However, the new behavior would be more consisitent with regular "project" element and would allow superproject be used even if it is not enabled for the particular manifest branch, so we have decided to make the change as it would provide more flexibility and better matches what other elements would do. Bug: [google internal] b/187868160 Change-Id: I35255ee347aff6e65179f7879d52931f168b477e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: Move enrollment to opt-out when enabled globallyv2.16.8Xin Li2021-09-201-36/+10
| | | | | | | | | | | | | | | | Our internal experiments was a success so far and we are enrolling 100% users now. Instead of asking every two weeks, simply consider a lack of unexpired choice as accepting the system default. With this change the user would still be able to override the system default with --no-use-superproject, or to permanently set the choice in user's profile with git config --global repo.superprojectchoice. Bug: [google internal] b/190688390 Change-Id: Idc77a9cbf88a169d90304169e91f0d722dc4ac8b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317975 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* superproject: Log branch and remote url with every log message.Raman Tenneti2021-09-161-14/+18
| | | | | | | | | | | | | | | | | | | | Saved superproject's remote URL in _remote_url data and used it in the _Fecth function. Tested: $ ./run_tests $ flake8 git_superproject.py $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest $ repo_dev sync Verified the all log messages have the following format. repo superproject branch: <branch> url: <url> warning: <message> Bug: [google internal] b/200072098 Change-Id: Iac6af7c99225479fd50bc6909396b22e0ce5f76b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318177 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Provide accurate feedback for user choicev2.16.7Xin Li2021-09-081-3/+8
| | | | | | | | | | | | | | Currently the code would give a message that would appear like the user have enrolled the experiment regardless of the actual choice. For users who choose to not enroll in the experiment, we should give them instructions to override (enable) superproject once instead of how to disable it, which is what the code already behave. Bug: [google internal] b/199167992 Change-Id: Iba3314cb510aedf024375a26baa8bc1d5e2846cf Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317382 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* superoject: log error message in the 'fmt' field also.v2.16.6Raman Tenneti2021-09-031-1/+1
| | | | | | | | | | | Tested: + Verified error messages are being collected. Bug: [google internal] b/193711236 Change-Id: I6c608a2af332ccd38722b7f83a82e5ac8fa143db Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317162 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: print messages if the manifest has superproject tag.v2.16.5Raman Tenneti2021-08-131-13/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1) If the manifest has superproject tag (git_master, etc), then display error/warning messages (as it is doing today) 2) If the manifest doesn't have superproject tag (nest, chromeos manifests), then don't display any error/warning messages about superrproject (behave as though user has specified --no-use-superproject). 3) Print error/warning messages if --use-superproject passed as argument to repo sync. 4) No change in behavior for the repo init command. git_superproject.py: + Fixed typo in _WriteManifestFile method name + Superproject accepts print_message as an argument and it defaults to True. All messages that are printed to stderr are controlled by this flag. If it is True, then messages get printed. + Added PrintMessages function which return true if either --use-superproject is specified on the command line or if the manifest has a superproject tag. sync.py: + Displays the warning message if PrintMessgages are enabled and passes that as argument to superproject object. + Added 'hassuperprojecttag' trace2 log entry for analysis. We can find users/branches that are using superproject, but the manifest is missing the superproject tag. Tested: $ ./run_tests + Verified printing of messages with and without superproject tag, with with --use-superproject option. + aosp-master $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest $ repo_dev sync + A manifest without superproject tag. $ repo_dev init -m $(pwd)/manifest_7482982.xml $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml Bug: [google internal] b/196411099 Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: prepend messages with - "repo superproject"Raman Tenneti2021-08-121-17/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | Changed _LogError method to _LogWarning. Replaced 'repo error:' with "repo superproject warning:"(except IOError message, which is still an "repo superproject error:" message) Tested: $ ./run_tests Tested the errors and warnings by forcing the error/warning. $ repo_dev sync -j 20 --use-superproject platform/packages/apps/Music ... repo superproject warning: please file a bug using go/repo-bug to report missing commit_ids for: [] ... repo superproject error: cannot write manifest to : /sdc/android/src/aosp/.repo/exp-superproject/superproject_override.xml ... Bug: [google internal] b/193711236 Change-Id: Ia0b6c830e04cf18dfc1a2ce325181a5b1160e054 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314642 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Xin Li <delphij@google.com>
* superproject: Skip updating of superproject when -l is used with sync.Raman Tenneti2021-07-281-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Skip updating the superproject when -l is present and use the existing superproject, if available (this would make sync -l work as it's intended to do), and fall back to sync without superproject when not (this would catch the case when superproject is enabled by automatic rollout). Tested: $ repo sync -j 20 -n NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` /usr/local/google/home/rtenneti/work/android/src/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. Fetching: 100% (1032/1032), done in 41.184s ... $ repo_dev sync -j 20 -l prebuilts/asuite/: discarding 1 commits prebuilts/runtime/: discarding 1 commits ... repo sync has finished successfully. + With superproject-override.xml and test it. $ ls -l .repo/exp-superproject/ total 176 drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 superproject_override.xml rtenneti@rtenneti:~/work/android/src/aosp$ repo_dev sync -j 20 -l ... repo sync has finished successfully. + Rename the file superproject-override.xml and test it. $ ls -l .repo/exp-superproject/ total 176 drwxr-xr-x 7 rtenneti primarygroup 4096 Jul 27 14:10 925043f706ba64db713e9bf3b55987e2-superproject.git -rw-r--r-- 1 rtenneti primarygroup 172742 Jul 27 14:10 temp.xml $ repo_dev sync -j 20 -l Checking out: 1% (12/1031) platform/external/rust/crates/fallible-streaming-iteexternal/linux-kselftest/: discarding 1 commits prebuilts/remoteexecution-client/: discarding 1 commits Checking out: 51% (536/1031) platform/prebuilts/gcc/darwin-x86/aarch64/.... .... Checking out: 100% (1031/1031), done in 5.478s repo sync has finished successfully. Bug: [google internal] b/184368268 Change-Id: I3aba5872e4f7c299977b92c2a39847ef28698c5a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312962 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject - More friendly user message when superproject failed.v2.16.1Raman Tenneti2021-07-141-28/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | superproject is going to be default for some users. This change doesn't fail for repo init or repo sync if source couldn't be synced because of errors in superproject and superproject=true in the config file. The commands will fail if --use-superproject is specified on the command line explicitly. The error messages are logged with trace2 event logs and will be monitored. + sync - When there are errors with superproject and git_superproject says it is fatal failure, sync will exit only when --use-superproject option is specified on the command line. + init - command doesn't fail *if there are any superproject errors), but it will print a warning message and logs message via trace2 event logs. For fatal errors, init will exit only when --use-superproject option is specified on the command line. + All git commands log the command that is being executed so trace2 event logs will know the manifest, remote url and the branch name. There is no functional change other than fatal errors are honored with --use-supeproject option with init/sync commands. Tested the code with the following commands. $ ./run_tests -v Test 1 - sync'ing without errors -------------------------------- Added the following lines to '~/.repoconfig/config [repo] superproject = true $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in /usr/local/google/home/rtenneti/work/drive2/android/test $ repo_dev sync remote: Total 4 (delta 1), reused 4 (delta 1) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` /usr/local/google/home/rtenneti/work/drive2/android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. ... Test 2 - init and sync fail when --use-superproject option is passed -------------------------------------------------------------------- $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 --use-superproject remote: Total 57 (delta 16), reused 56 (delta 16) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal, and you can continue to run repo sync, please run repo init with the --no-use-superproject option to stop seeing this warning rtenneti@rtenneti2:~/work/drive2/android/test$ repo_dev sync --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo: error: git fetch call failed, command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: Cannot get project commit ids from manifest warning: Update of revisionId from superproject has failed, repo sync will not use superproject to fetch the source. Please resync with the --no-use-superproject option to avoid this repo warning. Test 3 - git fetch command fails and git command is printed ----------------------------------------------------------- With config change $ repo_dev init -u https://android.googlesource.com/platform/manifest -b android-s-beta-2 ... NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/test/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Performing initial setup for superproject; this might take several minutes. repo: error: git fetch call failed,command: git ['fetch', 'https://android.googlesource.com/platform/superproject', '--depth', '1', '--force', '--no-tags', '--filter', 'blob:none', 'android-s-beta-2:android-s-beta-2'], return code: 128, stderr: fatal: couldn't find remote ref android-s-beta-2 warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in .... Test 4 - no superproject tag ----------------------------- $ repo_dev init -u https://android.googlesource.com/platform/manifest -b pie-dev ... NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: .../android/pie_dev/.repo/manifest.xml warning: git update of superproject failed, repo sync will not use superproject to fetch source; while this error is not fatal and you can continue to run repo sync please run repo init with the --no-use-superproject option to avoid the repo warning Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in ... $ repo_dev sync NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: /usr/local/google/home/rtenneti/work/drive2/android/pie_dev/.repo/manifest.xml warning: Cannot get project commit ids from manifest warning: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option to avoid the repo warning. Bug: [google internal] b/192614798 Bug: [google internal] b/Bug: [google internal] b/192614798 Change-Id: I9a97a0e7d9e609fad151bd7dd9cfc523eaa887cd Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/311502 Reviewed-by: Amith Dsouza <amithds@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* Add the ability to administratively enroll repo into using superproject.Xin Li2021-07-011-0/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Repo will remember a choice and an expiration time of the choice, per user, about whether to use superproject by default. When not specified from command line and the choice is not expired, repo would use the user default value. When a user default value is not present and when the system wide enable default is provided in git's system configuration, repo would ask the user for a confirmation which will be valid for two weeks. git_config.py: Add support for system config. When reading system config, we would use --system to avoid hardcoding a path as the value may be different on some other distributions. git_superproject.py: Add a new subroutine, _UseSuperproject(), which returns whether superproject should be used and whether it is from a user configuration. The value is determined in the following order: 1. If the user specifies either --use-superproject or --no-use-superproject, then that choice is being used. 2. If neither is specified, we would then check the saved value (upon repo init) and use that choice when there was a choice. 3. We then check if there is a saved and unexpired value for user's choice in their ~/.gitconfig, and use the unexpired choice, if available. 4. Finally, if all the above didn't give us a decision, and if the git system configuration is providing a rollout hint, present a prompt to user for their decision and save it in ~/.gitconfig. subcmds/sync.py: Make use of the new UseSuperproject() provided by git_superproject.py. While there also silent stderr from git describe when determining the version of repo. Bug: [google internal] b/190688390 Change-Id: Iad3ee03026342ee500e5d65e2f0fa600d7637613 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309762 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* superproject - don't update the commit ids of projects that have revisionId.Raman Tenneti2021-06-291-0/+3
| | | | | | | | | | | | | | | | | | | | Pinned manifests and release manifest have revisionId set for all projects. For such projects don't update the commit ids. Tested the code with the following commands. $ ./run_tests -v $ repo_dev sync --use-superproject -j8 $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml Bug: [google internal] b/191995372 Change-Id: I4681135b1d15f4a63527b6f0356d76ec842485d6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310582 Reviewed-by: Xin Li <delphij@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Don't exit if superproject tag doesn't exist in manifest.v2.16Raman Tenneti2021-06-161-31/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't exit if there are missing commit ids in superproject. This change implements the following suggestion from delphij@: "we should note the event (so we know that --use-superproject but there were some errors, e.g. manifest didn't specify commit id for some reason, or if there is no superproject but --use-superproject is used), print out a message telling the use that this is not support, but continue as if --no-use-superproject was specified?" Changes: superproject: + Added git_trace2_event_log as an argument to the constructor. + Sync method returns SyncResult a NamedTuple of ++ success - True if sync of superproject is successful, or False. ++ fatal - True if caller should exit, Or False. + UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of ++ manifest_path - path name of the overriding manifest file instead of None ++ fatal - True if caller should exit, Or False + _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of ++ commit_ids - a dictionary with the projects/commit ids on success, otherwise None ++ fatal - True if caller should exit, Or False + Added _SkipUpdatingProjectRevisionId a helper function to see if a project's revision id needs to be updated or not. This function is used to exclude projects from local manifest file. + Added the following error events into git_trace2_event_log ++ If superproject is missing in a manifest ++ If there are missing commit ids for projects. command.py: + Deleted unused import - platform + Added git_trace2_event_log as a member so all subcmds can log error events. main.py: + Initialized git_trace2_event_log as a member of command object. init.py: + Deleted unused import - optparse init.py: + Called sys.exit only if Sync returns exit=True sync.py: + Called sys.exit only if Superproject's UpdateProjectsRevisionId returns exit=True + Reloaded the manifest only if manifest path is returned by UpdateProjectsRevisionId. If not, fall back to the old way of doing repo sync. test_git_superproject: + Added code to verify error events are being logged. + Added a test for no superproject tag + Added test for UpdateProjectsRevisionId not updating the revision id with the commit ids. Tested the code with the following commands. + Positive test case with aosp-master. $ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in .../android/aosp $ repo_dev sync -j40 --use-superproject remote: Total 12 (delta 4), reused 12 (delta 4) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. ... repo sync has finished successfully. + Negative test case without superproject tag. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml error: Cannot get project commit ids from manifest error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.589s repo sync has finished successfully. + Test for missing commit_id for a project. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint'] error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.364s repo sync has finished successfully. $ ./run_tests -v ... ...== 164 passed in 2.87s ==... Bug: [google internal] b/189371541 Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: add projects from local manifest to local::<filename> group.Raman Tenneti2021-06-101-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With repo sync --use-superproject, don't update the commit ids of every project that comes from local manifest. Tested the code with the following commands. $ ./run_tests -v + Test with local.xml 1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest 2. cd .repo cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests cd .. local$ time repo_dev sync --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed. ... Bug: [google internal] b/189360443 Bug: [google internal] b/189139268 Bug: https://crbug.com/gerrit/14499 Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* superproject: revert not updating commit ids if remote is different.Raman Tenneti2021-06-081-10/+0
| | | | | | | | | | | | | | superproject supports multiple remotes. Get all commit ids from superproject for all projects that are in the manifest. $ ./run_tests -v Bug: [google internal] b/186395810 Change-Id: I6edce3918853a7a3a65aec5528e6a43a544eff53 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308862 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: change the warning message to say it is beta.Raman Tenneti2021-06-031-2/+2
| | | | | | | | | | | | | | | $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest remote: Total 3 (delta 0), reused 3 (delta 0) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` ... $ ./run_tests -v Bug: [google internal] b/189946009 Change-Id: Ifb3ef266a72b67f3c4a2a3ac2033b10e03b789d4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308522 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: require git version 2.28.0 or higher.v2.15.4Raman Tenneti2021-06-031-1/+4
| | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ repo init --use-superproject -u https://android.googlesource.com/platform/manifest + Tested for the wrong git version. $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest WARNING: --use-superproject is experimental and not for general use superproject requires a git version 2.28 or later error: git update of superproject failed $ ./run_tests -v Bug: https://crbug.com/gerrit/14617 Bug: [google internal] b/189846687 Change-Id: I5cd4158ea29b3b3c8c81234f4e818165de346e63 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308442 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: Use bugurl from contactinfo in the missing commits error message.v2.15.1Raman Tenneti2021-05-181-2/+1
| | | | | | | | | | | | | | | | | | + In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL. + contactinfo returns a namedtuple. + bug_url can be accessed as self._manifest.contactinfo.bugurl. Tested the code with the following commands. $ ./run_tests -v Added contactinfo tag to default.xml and verified that bugurl is used. Bug: [google internal] b/186220520. Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: skip updating commit ids if remote's fetchUrl don't match.Raman Tenneti2021-05-041-2/+2
| | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v + Test with local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml .repo/local_manifests/local.xml $ repo_dev sync -c -j8 + Test without local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j8 Bug: [google internal] b/186395810 Change-Id: Id618113a91c12bcb90a30a3c23d3d6842bcb49e1 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304942 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Don't update the commit ids of projects if remote is different.Raman Tenneti2021-05-031-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | 1) Skip setting the revision id (commit id) for the projects whose remote doesn't match superproject's remote. 2) exp-superproject/superproject_override.xml includes local_manfiest's projects. When we load this XML, don't reload projects from local.xml (otherwise we will get duplicate projects errors). Tested the code with the following commands. $ ./run_tests -v + Test with local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml .repo/local_manifests/local.xml $ repo_dev sync -c -j8 + Test without local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j8 Bug: [google internal] b/186395810 Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* help/version: sprinkle bug report URL aroundMike Frysinger2021-04-261-2/+2
| | | | | | | | | Make it a bit easier for people to locate bug reporting info. Change-Id: If9c8939c84ebd52eb96b353c1797afa25868bb85 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303943 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* superproject: Added --depth=1 argument to git fetch command.Raman Tenneti2021-03-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Without --depth $ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject real 6m48.086s user 3m27.281s sys 1m1.386s With --depth=1 $ time repo_dev init -u sso://googleplex-android.git.corp.google.com/platform/manifest -b master --repo-rev=main --use-superproject real 2m49.637s user 2m51.458s sys 0m39.108s From dwillemsen@: "For me it's the difference between 9m28s using the current code and 16s using --depth=1 while fetching the superproject." Bug: [google internal] b/180451672 Bug: https://crbug.com/gerrit/13707 Change-Id: I1c3e4aef4414c4e9dd259fb6e4619da0421896b0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300922 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: pass groups to ToXml method.Raman Tenneti2021-03-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added the following methods to XmlManifest class. + GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower() + GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func. + Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr. + Used the above methods to get groups in command.py::GetProjects and part of init.py. TODO: clean up these funcs to take structured group data more instead of passing strings around everywhere that need parsing. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL and verified prebuilts/fullsdk-linux directory has all the folders. Tested repo init and repo sync with --use-superproject and without --use-superproject argument. $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j32 Bug: [google internal] b/181804931 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Display status messages during repo init/sync.v2.13.3Raman Tenneti2021-03-041-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Superproject objects accept the optional argument “quiet”. The following progress messages are displayed if quiet is false. Displayed the following message whenever we find we have to make a new folder (aka new remote), because if you started with repo init android and later do googleplex-android that is when it will be slow. "<location>: Performing initial setup for superproject; this might take several minutes.". After fetch completion, added the following notification: "<location>: Initial setup for superproject completed." Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo_dev init -u persistent-https://googleplex-android.git.corp.google.com/platform/manifest -b rvc-dev --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject Bug: [google internal] b/181178282 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia7fb85c6fb934faaa90c48fc0c55e7f41055f48a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299122 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: superproject - support for switching hosts and switching branches.Raman Tenneti2021-02-251-38/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-111-65/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Pass branch to git ls-tree.Raman Tenneti2021-02-081-4/+8
| | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/179702819 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I7d2b609ac2f927c94701757aa1502ba236afe7c0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296342 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-081-16/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: superproject performance changes.Raman Tenneti2021-02-071-15/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After updating all project’s revsionIds with the SHAs from superproject, write the updated manifest into superproject_override.xml file. Reload that file for future Reloads. This file is created in exp-superproject directory. Moved most of the code that is superproject specific into git_superproject.py and wrote test code. If git pull fails, did a git clone of the superproject. We saw performance gains for consecutive repo sync's. The time to sync went down from around 120 secs to 40 secs when repo sync is executed consecutively. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. First run $ time repo sync --use-superproject ... real 21m3.745s user 97m59.380s sys 19m11.286s After two consecutive sync runs $ time repo sync -c -j8 --use-superproject real 0m39.626s user 0m29.937s sys 0m38.155s Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: Added --filter=blob:none for git clone of superproject.Raman Tenneti2021-02-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | + This is without --depth option. This is done for reachability. Server doesn't know what you know about in the history so it always sends you the whole thing Which is very slow. If we have the full history it can send you incremental update history which is very small and fast. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. .../WORKING_DIRECTORY$ repo sync --use-superproject Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362 Reviewed-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-14 06:52:38 +0000