summaryrefslogtreecommitdiffstats
path: root/tests
Commit message (Collapse)AuthorAgeFilesLines
* git_config: Fixed test.gitconfig getting updated when running tests.v2.17Raman Tenneti2021-09-302-29/+19
| | | | | | | | | | | | | | | | | | Moved test_GetSyncAnalysisStateData to GitConfigReadWriteTests class. Deleted [repo "syncstate*..] data from tests/fixtures/test.gitconfig. Tested: ./run_tests ... tests/test_git_config.py::GitConfigReadWriteTests::test_GetSyncAnalysisStateData PASSED [ 84%] ... Bug: https://crbug.com/gerrit/15103 Change-Id: I8cb89ce10b025994a045106c9c66dd243ae8ba50 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319557 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* trace2_event: Add remove_prefix to fix failing tests on Linux & macOS.Raman Tenneti2021-09-291-1/+9
| | | | | | | | | | | | | | | | | | removeprefix is available i python 3.9. Mac and Linux are running in a version below 3.9. Thus tests are failing with the following error: "AttributeError: 'str' object has no attribute 'removeprefix' " Replaced the removeprefix with custom function which we will delete once Linux and macOS versions are updated. Tested: $ ./run_tests Bug: [google internal] b/201453085 Change-Id: I9b4d564ff1176e1b4471805ef05472c1914cd9f9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319375 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Add 'dest-path' to extend-project to support changing pathMichael Kelly2021-09-281-0/+46
| | | | | | | | | | This allows us to move the repository to a new location in the source tree without having to remove-project + add a new project tag. Change-Id: I4dba6151842e57f6f2b8fe60cda260ecea68b7b4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310962 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Michael Kelly <mkelly@arista.com>
* superproject: Log syncstate's parameter as data-json it it is an array.Raman Tenneti2021-09-281-1/+1
| | | | | | | | | | | | | | | | | | | All the values of syncstate are strings, check the first byte and last byte to see if it is an array. For syncstate data, there were no false positives. Tested: $ repo_dev sync Verified event logged for argv is "data-json". $./run_tests Bug: [google internal] b/201102002 Change-Id: Id56adb532b80267f08d09147ac663cdd5987ce87 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319075 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* make file removal a bit more robustMike Frysinger2021-09-281-0/+50
| | | | | | | | | | | | Some of the file removal calls are subject to race conditions (if something else deletes the file), so extend our remove API to have an option to ignore ENOENT errors. Then update a bunch of random call sites to use this new functionality. Change-Id: I31a9090e135452033135337a202a4fc2dbf8b63c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319195 Reviewed-by: Sean McAllister <smcallis@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Log argv parameter of syncstate as 'data-json'.Raman Tenneti2021-09-271-4/+7
| | | | | | | | | | | | | | | | | | | | Fixed: "we need to make a special case for logging the argv; it should probably be a "data-json" event so that we log this directly as an array rather than an encoded string. Tested: $ repo_dev sync Verified event logged for argv is "data-json". $./run_tests Bug: [google internal] b/201102002 Change-Id: I18ccec79c73c8dc931cb8afc472b2361db8aea4c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/319055 Reviewed-by: Josh Steadmon <steadmon@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: support a new revision attribute.Xin Li2021-09-272-1/+70
| | | | | | | | | | | | | | | | | | | | | | | Tested: $ ./run_tests Verified that a manifest that specified superproject revision would use the specified revision, and superproject will use the default revision. Note that this is a slight behavior change from earlier repo versions, which would always use the branch name of the manifest itself. However, the new behavior would be more consisitent with regular "project" element and would allow superproject be used even if it is not enabled for the particular manifest branch, so we have decided to make the change as it would provide more flexibility and better matches what other elements would do. Bug: [google internal] b/187868160 Change-Id: I35255ee347aff6e65179f7879d52931f168b477e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317643 Tested-by: Xin Li <delphij@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* manifest: make repo-hooks more robust wrt element orderingJack Neus2021-09-231-0/+13
| | | | | | | | | | | | | Currently, repo will fail to sync to a manifest if the definition of the repo-hooks project comes after the repo-hooks element. BUG=none TEST=new test, run_tests Change-Id: I0bf85625173492af6c6404d4b67543e96e670562 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/318520 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Jack Neus <jackneus@google.com>
* sync: Log repo sync state events as 'data' events.Raman Tenneti2021-09-142-1/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git_trace2_event_log.py: + Added LogDataConfigEvents method to log 'data' events. Sync's current_sync_state and previous_sync_state are logged as 'data' events in the current log. It logs are key/value in the |config| argument. Each key is prefixed with |prefix| argument. The following are sample events that are logged during repo sync. {"event":"data", "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0", "thread":"MainThread", "time":"2021-09-14T18:16:19.935846Z", "key":"previous_sync_state/repo.syncstate.main.synctime", "value":"2021-09-14T17:27:11.573717Z"} {"event":"data", "sid":"repo-20210914T181545Z-P000330c0/repo-20210914T181545Z-P000330c0", "thread":"MainThread", "time":"2021-09-14T18:16:19.955546Z", "key":"current_sync_state/repo.syncstate.main.synctime", "value":"2021-09-14T18:16:19.935979Z"} tests/test_git_trace2_event_log.py: + Added unit tests sync.py: + Changed logging calls to LogDataConfigEvents. Tested: $ ./run_tests Tested it by running the following command multiple times. $ repo_dev sync -j 20 repo sync has finished successfully Verified config data is looged in trace2 event logs. Bug: [google internal] b/199758376 Change-Id: I75fd830e90c1811ec28510538c99a2632b104e85 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/317823 Reviewed-by: Josh Steadmon <steadmon@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: print messages if the manifest has superproject tag.v2.16.5Raman Tenneti2021-08-132-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1) If the manifest has superproject tag (git_master, etc), then display error/warning messages (as it is doing today) 2) If the manifest doesn't have superproject tag (nest, chromeos manifests), then don't display any error/warning messages about superrproject (behave as though user has specified --no-use-superproject). 3) Print error/warning messages if --use-superproject passed as argument to repo sync. 4) No change in behavior for the repo init command. git_superproject.py: + Fixed typo in _WriteManifestFile method name + Superproject accepts print_message as an argument and it defaults to True. All messages that are printed to stderr are controlled by this flag. If it is True, then messages get printed. + Added PrintMessages function which return true if either --use-superproject is specified on the command line or if the manifest has a superproject tag. sync.py: + Displays the warning message if PrintMessgages are enabled and passes that as argument to superproject object. + Added 'hassuperprojecttag' trace2 log entry for analysis. We can find users/branches that are using superproject, but the manifest is missing the superproject tag. Tested: $ ./run_tests + Verified printing of messages with and without superproject tag, with with --use-superproject option. + aosp-master $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest $ repo_dev sync + A manifest without superproject tag. $ repo_dev init -m $(pwd)/manifest_7482982.xml $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml Bug: [google internal] b/196411099 Change-Id: I92166dcad15a4129fab82edcf869e7c8db3efd4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314982 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: prepend messages with - "repo superproject"Raman Tenneti2021-08-121-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | Changed _LogError method to _LogWarning. Replaced 'repo error:' with "repo superproject warning:"(except IOError message, which is still an "repo superproject error:" message) Tested: $ ./run_tests Tested the errors and warnings by forcing the error/warning. $ repo_dev sync -j 20 --use-superproject platform/packages/apps/Music ... repo superproject warning: please file a bug using go/repo-bug to report missing commit_ids for: [] ... repo superproject error: cannot write manifest to : /sdc/android/src/aosp/.repo/exp-superproject/superproject_override.xml ... Bug: [google internal] b/193711236 Change-Id: Ia0b6c830e04cf18dfc1a2ce325181a5b1160e054 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/314642 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Xin Li <delphij@google.com>
* sync: Remove '_' from the repo.syncstate.* keys when saved to config.v2.16.4Raman Tenneti2021-07-292-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | GitConfig doesn't save keys if the keys contain "_" characters. Some of the options like mp_update, use_superproject have underscores. This fixes issue with previous_sync_state missing some of the options. Tested: $ ./run_tests $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest Tested it by running the sync command multiple times and verifing previous_sync_state and current_sync_state have the same keys. $ repo_dev sync -j 20 repo sync has finished successfully Verified config file has [syncstate ...] data saved. Bug: [google internal] b/188573450 Change-Id: I16b52a164f9dd1633d7dad1d8cf6b151c629fcb1 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313242 Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* sync: Added logging of repo sync state and config options for analysis.Raman Tenneti2021-07-292-0/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git_config.py: + Added SyncAnalysisState class, which saves the following data into the config object. ++ sys.argv, options, superproject's logging data. ++ repo.*, branch.* and remote.* parameters from config object. ++ current time as synctime. ++ Version number of the object. + All the keys for the above data are prepended with 'repo.syncstate.' + Added GetSyncAnalysisStateData and UpdateSyncAnalysisState methods to GitConfig object to save/get the above data. git_trace2_event_log.py: + Added LogConfigEvents method with code from DefParamRepoEvents to log events. sync.py: + superproject_logging_data is a dictionary that collects all the superproject data that is to be logged as trace2 event. + Sync at the end logs the previously saved syncstate.* parameters as previous_sync_state. Then it calls config's UpdateSyncAnalysisState to save and log all the current options, superproject logged data. docs/internal-fs-layout.md: + Added doc string explaining [repo.syncstate ...] sections of .repo/manifests.git/config file. test_git_config.py: + Added unit test for the new methods of GitConfig object. Tested: $ ./run_tests $ repo_dev init --use-superproject -u https://android.googlesource.com/platform/manifest Tested it by running the following command multiple times. $ repo_dev sync -j 20 repo sync has finished successfully Verified config file has [syncstate ...] data saved. Bug: [google internal] b/188573450 Change-Id: I1f914ce50f3382111b72940ca56de7c41b53d460 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/313123 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Xin Li <delphij@google.com>
* repo: properly handle remote annotations in manifest_xmlJack Neus2021-07-231-1/+28
| | | | | | | | | | BUG=b:192664812 TEST=tests/ Change-Id: I1aa50260f4a00d3cebbd531141e1626825e70127 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/312643 Tested-by: Jack Neus <jackneus@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Gracefully ignore bad remove-project lineMichael Kelly2021-07-081-0/+50
| | | | | | | | | | | | | | | | Sometimes, we don't care if the remove project is referring to a non-existing project and we can just ignore it. This change allows us to ignore remove-project entries if the project that they refer to doesn't exist, making them effectively a no-op. Because this change breaks existing configuration, we allow this to be configuration controlled using the `optional` attribute in the remove-project tag. Change-Id: I6313a02983e81344eadcb4e47d7d6b037ee7420e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310964 Tested-by: Michael Kelly <mkelly@arista.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* Fix an issue when syncing with --use-superproject and clone bundles.Xin Li2021-06-302-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | It is possible that a clone bundle contained the object referenced by the branch in the manifest and in the superproject, but not the branch itself (for example, the branch may be newly created from an existing branch, or is not vislble to the user downloading the clone bundle). When --use-superproject is enabled, because we are overriding revisionExpr with the SHA1 revision provided by the superproject, the verification step would succeed, but because the expected branch do not exist, it would confuse git-repo at a later time, as it is expecting the remote branch to exist in the local clone. In project.py, fix this by making SetRevisionId() to always remember the actual branch name and verify it in _CheckForImmutableRevision() so that we only skip the fetch step when both objects exists locally. Bug: [google internal] b/191974277 Change-Id: I49d3ca0667f524c8c45f416492faf95b1dd822fb Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310802 Reviewed-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* superproject - don't update the commit ids of projects that have revisionId.Raman Tenneti2021-06-291-2/+49
| | | | | | | | | | | | | | | | | | | | Pinned manifests and release manifest have revisionId set for all projects. For such projects don't update the commit ids. Tested the code with the following commands. $ ./run_tests -v $ repo_dev sync --use-superproject -j8 $ repo_dev sync -n -c -j32 -m $(pwd)/manifest_7482982.xml Bug: [google internal] b/191995372 Change-Id: I4681135b1d15f4a63527b6f0356d76ec842485d6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/310582 Reviewed-by: Xin Li <delphij@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: Don't exit if superproject tag doesn't exist in manifest.v2.16Raman Tenneti2021-06-161-18/+116
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't exit if there are missing commit ids in superproject. This change implements the following suggestion from delphij@: "we should note the event (so we know that --use-superproject but there were some errors, e.g. manifest didn't specify commit id for some reason, or if there is no superproject but --use-superproject is used), print out a message telling the use that this is not support, but continue as if --no-use-superproject was specified?" Changes: superproject: + Added git_trace2_event_log as an argument to the constructor. + Sync method returns SyncResult a NamedTuple of ++ success - True if sync of superproject is successful, or False. ++ fatal - True if caller should exit, Or False. + UpdateProjectsRevisionId returns UpdateProjectsResult a NamedTuple of ++ manifest_path - path name of the overriding manifest file instead of None ++ fatal - True if caller should exit, Or False + _GetAllProjectsCommitIds returns CommitIdsResult a NamedTuple of ++ commit_ids - a dictionary with the projects/commit ids on success, otherwise None ++ fatal - True if caller should exit, Or False + Added _SkipUpdatingProjectRevisionId a helper function to see if a project's revision id needs to be updated or not. This function is used to exclude projects from local manifest file. + Added the following error events into git_trace2_event_log ++ If superproject is missing in a manifest ++ If there are missing commit ids for projects. command.py: + Deleted unused import - platform + Added git_trace2_event_log as a member so all subcmds can log error events. main.py: + Initialized git_trace2_event_log as a member of command object. init.py: + Deleted unused import - optparse init.py: + Called sys.exit only if Sync returns exit=True sync.py: + Called sys.exit only if Superproject's UpdateProjectsRevisionId returns exit=True + Reloaded the manifest only if manifest path is returned by UpdateProjectsRevisionId. If not, fall back to the old way of doing repo sync. test_git_superproject: + Added code to verify error events are being logged. + Added a test for no superproject tag + Added test for UpdateProjectsRevisionId not updating the revision id with the commit ids. Tested the code with the following commands. + Positive test case with aosp-master. $ repo_dev init -u persistent-https://android.git.corp.google.com/platform/manifest -b master --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. Your identity is: Raman Tenneti <rtenneti@google.com> If you want to change this, please re-run 'repo init' with --config-name repo has been initialized in .../android/aosp $ repo_dev sync -j40 --use-superproject remote: Total 12 (delta 4), reused 12 (delta 4) NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. ... repo sync has finished successfully. + Negative test case without superproject tag. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` repo error: superproject tag is not defined in manifest: .../android/aosp/.repo/manifest.xml error: Cannot get project commit ids from manifest error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.589s repo sync has finished successfully. + Test for missing commit_id for a project. $ repo_dev sync -j40 --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../android/aosp/.repo/exp-superproject/925043f706ba64db713e9bf3b55987e2-superproject.git: Initial setup for superproject completed. error: please file a bug using go/repo-bug to report missing commit_ids for: ['build/blueprint'] error: Update of revsionId from superproject has failed. Please resync with --no-use-superproject option ... Checking out: 100% (1022/1022), done in 3.364s repo sync has finished successfully. $ ./run_tests -v ... ...== 164 passed in 2.87s ==... Bug: [google internal] b/189371541 Change-Id: I5ea49f87e8fa41be590fc0c914573e16c8cdfcfa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/309162 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* repo: properly handle NoneType in Default/Remote equality checksJack Neus2021-06-111-0/+31
| | | | | | | | | | BUG=none TEST=none Change-Id: I4ccdbbc7ba4b6f6e20c6959db1b46fdb44ea2819 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308982 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Jack Neus <jackneus@google.com>
* superproject: add projects from local manifest to local::<filename> group.Raman Tenneti2021-06-101-4/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With repo sync --use-superproject, don't update the commit ids of every project that comes from local manifest. Tested the code with the following commands. $ ./run_tests -v + Test with local.xml 1. repo init --use-superproject -u persistent-https://googleplex-android.git.corp.google.com/a/platform/manifest 2. cd .repo cp -r /google/src/head/depot/google3/wireless/android/build_tools/translations/pipeline/local_manifests local_manifests cd .. local$ time repo_dev sync --use-superproject NOTICE: --use-superproject is in beta; report any issues to the address described in `repo version` .../local/.repo/exp-superproject/feb2c2847da5e274f3d530d5ab438af8-superproject.git: Initial setup for superproject completed. ... Bug: [google internal] b/189360443 Bug: [google internal] b/189139268 Bug: https://crbug.com/gerrit/14499 Change-Id: Ideaf268c294e9b500b2b9726ffbd733dd8d63004 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308822 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* trace2_event_log: Added logging of error events.Raman Tenneti2021-06-091-0/+24
| | | | | | | | | | | | | | | Added error event in preperation for superproject to log errors. Testing: + Unit tests ./run_tests -v Bug: [google internal] b/189371541 Change-Id: Ife1dd28d52d9e9925b7b34ae913f8eb5fa19037c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308863 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Xin Li <delphij@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* superproject: revert not updating commit ids if remote is different.Raman Tenneti2021-06-081-45/+0
| | | | | | | | | | | | | | superproject supports multiple remotes. Get all commit ids from superproject for all projects that are in the manifest. $ ./run_tests -v Bug: [google internal] b/186395810 Change-Id: I6edce3918853a7a3a65aec5528e6a43a544eff53 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/308862 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* tests: Make the tests pass for Python < 3.8Peter Kjellerstedt2021-05-262-28/+55
| | | | | | | | | | | | | | Before Python 3.8, xml.dom.minidom sorted the attributes of an element when writing it to a file, while later versions output the attributes in the order they were created. Avoid these differences by sorting the attributes for each element before comparing the generated manifests with the expected ones. Bug: https://crbug.com/gerrit/14382 Change-Id: Ie2597727afcc48f9063a7261ad970e8a549f0587 Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303326 Reviewed-by: Mike Frysinger <vapier@google.com>
* superproject: Use bugurl from contactinfo in the missing commits error message.v2.15.1Raman Tenneti2021-05-181-1/+1
| | | | | | | | | | | | | | | | | | + In XmlManifest._Unload set 'bugurl' to Wrapper().BUG_URL. + contactinfo returns a namedtuple. + bug_url can be accessed as self._manifest.contactinfo.bugurl. Tested the code with the following commands. $ ./run_tests -v Added contactinfo tag to default.xml and verified that bugurl is used. Bug: [google internal] b/186220520. Change-Id: Iaafd6465e072b2e47a0a0b548bf6cb608a0b0a04 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/306342 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* ssh: rewrite proxy management for multiprocessing usagev2.15Mike Frysinger2021-05-101-4/+26
| | | | | | | | | | | | | | | | We changed sync to use multiprocessing for parallel work. This broke the ssh proxy code as it's all based on threads. Rewrite the logic to be multiprocessing safe. Now instead of the module acting as a stateful object, callers have to instantiate a new ProxyManager class that holds all the state, an pass that down to any users. Bug: https://crbug.com/gerrit/12389 Change-Id: I4b1af116f7306b91e825d3c56fb4274c9b033562 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305486 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* ssh: move all ssh logic to a common placeMike Frysinger2021-05-062-32/+52
| | | | | | | | | | | | We had ssh logic sprinkled between two git modules, and neither was quite the right home for it. This largely moves the logic as-is to its new home. We'll leave major refactoring to followup commits. Bug: https://crbug.com/gerrit/12389 Change-Id: I300a8f7dba74f2bd132232a5eb1e856a8490e0e9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305483 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_command: switch version caches to functoolsMike Frysinger2021-05-061-2/+7
| | | | | | | | | Simplifies the code a bit to use the stdlib cache helper. Change-Id: I778e90100ce748a71cc3a5a5d67dda403334315e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305482 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest_xml: cleanup of contactinfo test for readability with f-strings.Raman Tenneti2021-05-051-1/+3
| | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/186220520. Change-Id: I1c0f8958ff4c615707eec218250e8de753ec6562 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305282 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* manifest_xml: initial support for <contactinfo>Raman Tenneti2021-05-041-21/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | It will be used to let manifest authors self-register contact info. This element can be repeated, and any later entries will clobber earlier ones. This would allow manifest authors who extend manifests to specify their own contact info. It would have 1 required attribute: bugurl. "bugurl" specifies the URL to file a bug against the manifest owner. <contactinfo bugurl="bug-url"/> TODO: This CL only implements the parsing logic and further work will be in followup CLs. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v Bug: [google internal] b/186220520. Change-Id: I47e765ba2dab5cdf850191129f4d4cd6b803f451 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305203 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* subcmds: force consistent help text formatMike Frysinger2021-05-041-0/+30
| | | | | | | | | | | | We're inconsistent with help text as to whether it uses title case and whether it ends in a period. Add a test to enforce a standard, and use the style that Python optparse & argparse use themselves (e.g. with the --help option): always lowercase, and never trailing period. Change-Id: Ic1defae23daeac0ac9116aaf487427f50b34050d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/305144 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Don't update the commit ids of projects if remote is different.Raman Tenneti2021-05-031-13/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | 1) Skip setting the revision id (commit id) for the projects whose remote doesn't match superproject's remote. 2) exp-superproject/superproject_override.xml includes local_manfiest's projects. When we load this XML, don't reload projects from local.xml (otherwise we will get duplicate projects errors). Tested the code with the following commands. $ ./run_tests -v + Test with local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --use-superproject --partial-clone --clone-filter=blob:limit=10M && mkdir -p .repo/local_manifests && (gcertstatus -quiet=true || gcert) && ln -s /google/src/head/depot/google3/wireless/android/build_tools/aosp/manifests/mirror-aosp-master-with-vendor/local.xml .repo/local_manifests/local.xml $ repo_dev sync -c -j8 + Test without local.xml $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j8 Bug: [google internal] b/186395810 Change-Id: I4e9d4ac2d94a9fc0cef0ccd787b6310758009e86 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304882 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* manifest_xml: ban use of newlines in pathsMike Frysinger2021-04-301-0/+16
| | | | | | | | | | | | There should be no valid use of these anywhere, so just ban them to make things easier for people. Bug: https://crbug.com/gerrit/14156 Bug: https://crbug.com/gerrit/14200 Change-Id: I8d2cf988c510c98194c43a329a2b9bf313a3f0a8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/304662 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* tests: Make ReviewableBranchTests.test_smoke work with git < 2.28.0Peter Kjellerstedt2021-04-141-1/+1
| | | | | | | | Bug: https://crbug.com/gerrit/14380 Change-Id: Id015bd98b008e1530ada2c7e4332c67e8e208e25 Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/303325 Reviewed-by: Mike Frysinger <vapier@google.com>
* repo: Add a new "command" event type to git trace2 logging in repo.Raman Tenneti2021-03-181-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new "event": "command", which is emitted at when all command arguments have been processed. Additional fields: "name": Name of the primary command (ex: repo, git) "subcommands"': List of the sub-commands once command-line arguments are processed Examples: Command: repo --version Event: {"event": "command", <common fields>, "name": "repo", "subcommands": ["version"] } Bug: [google internal] b/178507266 Testing: - Unit tests - Verified repo git trace2 logs had expected data Change-Id: I825bd0ecedee45135382461a4ba10f987f09aef3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/300343 Reviewed-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* manifest: allow toplevel project checkoutsv2.13.5Mike Frysinger2021-03-121-2/+28
| | | | | | | | | | | | | | Re-allow checking out projects to the top of the repo client checkout. We add checks to prevent checking out files under .repo/ as that path is only managed by us, and projects cannot inject content or settings into it. Bug: https://crbug.com/gerrit/14156 Bug: https://crbug.com/gerrit/14200 Change-Id: Id6bf9e882f5be748442b2c35bbeaee3549410b25 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299623 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: pass groups to ToXml method.Raman Tenneti2021-03-112-4/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added the following methods to XmlManifest class. + GetDefaultGroupsStr() - return 'default,platform-' + platform.system().lower() + GetGroupsStr() - Same as gitc_utils.py's _manifest_groups func. + Replaced gitc_utils.py's_manifest_groups calls with GetGroupsStr. + Used the above methods to get groups in command.py::GetProjects and part of init.py. TODO: clean up these funcs to take structured group data more instead of passing strings around everywhere that need parsing. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL and verified prebuilts/fullsdk-linux directory has all the folders. Tested repo init and repo sync with --use-superproject and without --use-superproject argument. $ repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b androidx-main --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject $ repo_dev sync -c -j32 Bug: [google internal] b/181804931 Bug: https://crbug.com/gerrit/13707 Change-Id: Ia98585cbfa3a1449710655af55d56241794242b6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299422 Reviewed-by: Jonathan Nieder <jrn@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* Log repo.* config variables in git trace2 logger.Ian Kasprzak2021-03-081-0/+49
| | | | | | | | | | | | | Bug: [google internal] b/181758736 Testing: - Unit tests - Verified repo git trace2 logs had expected data Change-Id: I9af8a574377bd91115f085808c1271e9dee16a36 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299182 Tested-by: Ian Kasprzak <iankaz@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com>
* tests: fix duplicate method from copy & paste errorMike Frysinger2021-03-041-2/+2
| | | | | | | | | Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002 Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: relax include name rules for user-specified pathv2.13.1Mike Frysinger2021-03-021-2/+30
| | | | | | | | | | | | | Allow the user to specify relative or absolute or any other funky path that they want when using `repo init` or `repo sync`. Our goal is to restrict the paths in the remote manifest git repo we cloned from the network, not protect the user from themselves. Bug: https://crbug.com/gerrit/14156 Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: validate project name & path and include name attributesMike Frysinger2021-02-281-71/+159
| | | | | | | | | | | These attribute values are used to construct local filesystem paths, so apply the existing filesystem checks to them. Bug: https://crbug.com/gerrit/14156 Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: refactor the filesystem checking logic for more reuseMike Frysinger2021-02-281-0/+2
| | | | | | | | | | | | | | | | | | | | | This function is currently written with copyfile & linkfile in mind. Generalize the logic & function arguments slightly so we can reuse in more places that make sense. This changes the validation logic slightly too in that we no longer allow "." for the dest attribute with copyfile & linkfile, nor for the src attribute with copyfile. We already rejected those later on when checking against the active filesystem, but now we reject them a little sooner when parsing. The empty path check isn't a new requirement exactly -- repo used to crash on it, so it was effectively blocked, but now we diagnosis it. Bug: https://crbug.com/gerrit/14156 Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: superproject - support for switching hosts and switching branches.Raman Tenneti2021-02-251-15/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* tests: fix init subcmd after url changeMike Frysinger2021-02-251-1/+1
| | | | | | | | | | My recent 401c6f072564966437a74dc2f33280a85d79dc84 ("init: make --manifest-url flag optional") commit broke the unittest. Change-Id: I19ad0e8c8cbb84ab5474ebc370e00acfe957e136 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298223 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* tests: refactor manifest testsMike Frysinger2021-02-251-90/+98
| | | | | | | | | | | The XmlManifestTests class is getting to be large and we're only adding more to it. Factor out the core logic into a new TestCase so we can reuse it to better group more tests. Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* error: fix pickling of all exceptionsMike Frysinger2021-02-191-0/+53
| | | | | | | | | | | | | | Make sure all our custom exceptions can be pickled so that if they get thrown in a multiprocess subprocess, we don't crash & hang due to multiprocessing being unable to pickle+unpickle the exception. Details/examples can be seen in Python reports like: https://bugs.python.org/issue13751 Change-Id: Iddf14d3952ad4e2867cfc71891d6b6559130df4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297382 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-111-51/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* git_config: add SetBoolean helperMike Frysinger2021-02-111-3/+67
| | | | | | | | | A little sugar simplifies the code a bit. Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-081-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: superproject performance changes.Raman Tenneti2021-02-071-7/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After updating all project’s revsionIds with the SHAs from superproject, write the updated manifest into superproject_override.xml file. Reload that file for future Reloads. This file is created in exp-superproject directory. Moved most of the code that is superproject specific into git_superproject.py and wrote test code. If git pull fails, did a git clone of the superproject. We saw performance gains for consecutive repo sync's. The time to sync went down from around 120 secs to 40 secs when repo sync is executed consecutively. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. First run $ time repo sync --use-superproject ... real 21m3.745s user 97m59.380s sys 19m11.286s After two consecutive sync runs $ time repo sync -c -j8 --use-superproject real 0m39.626s user 0m29.937s sys 0m38.155s Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* manifest: set revisionId as revision attribute it it is not being set in ToXml.Raman Tenneti2021-02-071-0/+20
| | | | | | | | | | | | | | | | | As we were testing superproject setting revisionId attribute to SHA and reloading the manifest, we found out revisionId attribute is not being saved. Made the change to save the revisionId if it is not being saved. Tested the code with the following commands. $ ./run_tests -v Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182 Reviewed-by: Ian Kasprzak <iankaz@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-19 16:57:38 +0000