summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* docs: add warnings about repos data modelMike Frysinger2021-02-251-2/+22
| | | | | | | | | | For people coming across these docs and thinking that repo's methods are good to replicate, add a note warning them against doing so. Change-Id: I443a783794313851a6e7ba1c39baebac988bff9a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298164 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* platform_utils: delete unused FileDescriptorStreams APIsMike Frysinger2021-02-241-158/+0
| | | | | | | | | | | | | | Now that we've converted the few users of this over to subprocess APIs, we don't need this anymore. It's been a bit hairy to maintain across different operating systems, so there's no desire to bring it back. Using multiprocessing Pool to batch things has been working better in general anyways. Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: update event is_set APIMike Frysinger2021-02-231-7/+7
| | | | | | | | | Python 3 renamed this method from isSet to is_set. Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_command: switch process capturing over to subprocessMike Frysinger2021-02-231-33/+7
| | | | | | | | | | | | Now that these code paths are all synchronous, there's no need to run our own poll loop to read & pass thru/save output. Delete all of that and just let the subprocess module take care of it all. Change-Id: Ic27fe71b6f964905cf280ce2b183bb7ee46f4a0d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297422 Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* diff: add --jobs supportMike Frysinger2021-02-232-5/+41
| | | | | | | | | Use multiprocessing to run diff in parallel. Change-Id: I61e973d9c2cde039d5eebe8d0fe8bb63171ef447 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297483 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* forall: rewrite parallel logicMike Frysinger2021-02-221-66/+40
| | | | | | | | | | | | | | | | | This fixes intermingling of parallel jobs and simplifies the code by switching to subprocess.run. This also provides stable output in the order of projects by returning the output as a string that the main loop outputs. This drops support for interactive commands, but it's unclear if anyone was relying on that, and the default behavior (-j2) made that unreliable. If it turns out someone still wants this, we can look at readding it. Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: improve pool logicMike Frysinger2021-02-221-15/+10
| | | | | | | | | | Use a pool contextmanager to take care of the messy details like properly cleaning it up when aborting. Change-Id: I264ebb591c2e67c9a975b6dcc0f14b29cc66a874 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297243 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* status: improve parallel execution stabilityMike Frysinger2021-02-223-13/+22
| | | | | | | | | | | | | | | | | The status command runs a bunch of jobs in parallel, and each one is responsible for writing to stdout directly. When running many noisy jobs in parallel, output can get intermingled. Pass down a StringIO buffer for writing to so we can return the entire output as a string so the main job can handle displaying it. This fixes interleaved output as well as making the output stable: we always display results in the same project order now. By switching from map to imap, this ends up not really adding any overhead. Bug: https://crbug.com/gerrit/12231 Change-Id: Ic18b07c8074c046ff36e306eb8d392fb34fb6eca Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297242 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* command: unify --job option & default valuesMike Frysinger2021-02-225-25/+27
| | | | | | | | | | | | Extend the Command class to support adding the --jobs option to the parser if the command declares it supports running in parallel. Also pull the default value used for the number of local jobs into the command module so local commands can share it. Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* git_command: make execution synchronousMike Frysinger2021-02-201-6/+6
| | | | | | | | | | | | | Every use of GitCommand in the tree just calls Wait as soon as it's instantiated. Move the bulk of the logic into the init path to make the call synchronous to simplify. We'll cleanup the users of the Wait API to follup commits -- having this split makes it easier to track down regressions. Change-Id: I1e8c519efa912da723749ff7663558c04c1f491c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297244 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* use simpler super() magicMike Frysinger2021-02-194-17/+16
| | | | | | | | | | Python 3 has a simpler super() style so switch to it to make the code a little simpler and to stop pylint warnings. Change-Id: I1b3ccf57ae968d56a9a0bcfc1258fbd8bfa3afee Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297383 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* error: fix pickling of all exceptionsMike Frysinger2021-02-192-8/+61
| | | | | | | | | | | | | | Make sure all our custom exceptions can be pickled so that if they get thrown in a multiprocess subprocess, we don't crash & hang due to multiprocessing being unable to pickle+unpickle the exception. Details/examples can be seen in Python reports like: https://bugs.python.org/issue13751 Change-Id: Iddf14d3952ad4e2867cfc71891d6b6559130df4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297382 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: make --manifest-url flag optionalMike Frysinger2021-02-182-8/+23
| | | | | | | | | | | Since the --manifest-url flag is always required when creating a new checkout, allow the url to be specified via a positional argument. This brings it a little closer to the `git clone` UI. Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* gitc_utils: rewrite to use multiprocessingMike Frysinger2021-02-181-21/+29
| | | | | | | | | | | This is the only code in the tree that uses GitCommand asynchronously. Rewrite it to use multiprocessing.Pool as it makes the code a little bit easier to understand and simpler. Change-Id: I3ed3b037f24aa1e9dfe8eec9ec21815cdda7678a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297143 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com>
* project: make diff tools synchronousMike Frysinger2021-02-181-52/+44
| | | | | | | | | | | | | | | | | | | These are the only users in the tree that process the output as it's produced. All others capture all the output first and then process the results. However, these functions still don't fully return until it's finished processing, and these funcs are in turn used in other synchronous code paths. So it's unclear whether anyone will notice that it's slightly slower or less interactive. Let's try it out and see if users report issues. This will allow us to simplify our custom GitCommand code and move it over to Python's subprocess.run, and will help fix interleaved output when running multiple commands in parallel (e.g. `repo diff -j8`). Change-Id: Ida16fafc47119d30a629a8783babeba890515de0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297144 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* git_command: rework stdin handlingMike Frysinger2021-02-172-10/+8
| | | | | | | | | | | We only provide input to GitCommand in one place, so inline the logic to be more synchronous and similar to subprocess.run. This makes the code simpler and easier to understand. Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* trim redundant pass statementsMike Frysinger2021-02-163-3/+0
| | | | | | | | | Clean up a few linter warnings. Change-Id: I531d0263a202435d32d83d87ec24998f4051639c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_command: use subprocess.run for version infoMike Frysinger2021-02-161-5/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: If125ea7d776cdfa38a0440a2b03583de079c4839 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297023 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use subprocess.run to verify tagsMike Frysinger2021-02-161-13/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-116-136/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* diffmanifests/sync: simplify repodir lookupMike Frysinger2021-02-112-3/+3
| | | | | | | | | | We have access to repodir on the command object itself, so we don't need to pull it indirectly out of the manifest object. Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_config: add SetBoolean helperMike Frysinger2021-02-114-14/+81
| | | | | | | | | A little sugar simplifies the code a bit. Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Pass branch to git ls-tree.Raman Tenneti2021-02-081-4/+8
| | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/179702819 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I7d2b609ac2f927c94701757aa1502ba236afe7c0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296342 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-083-26/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: superproject performance changes.Raman Tenneti2021-02-073-58/+212
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After updating all project’s revsionIds with the SHAs from superproject, write the updated manifest into superproject_override.xml file. Reload that file for future Reloads. This file is created in exp-superproject directory. Moved most of the code that is superproject specific into git_superproject.py and wrote test code. If git pull fails, did a git clone of the superproject. We saw performance gains for consecutive repo sync's. The time to sync went down from around 120 secs to 40 secs when repo sync is executed consecutively. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. First run $ time repo sync --use-superproject ... real 21m3.745s user 97m59.380s sys 19m11.286s After two consecutive sync runs $ time repo sync -c -j8 --use-superproject real 0m39.626s user 0m29.937s sys 0m38.155s Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* manifest: set revisionId as revision attribute it it is not being set in ToXml.Raman Tenneti2021-02-072-0/+22
| | | | | | | | | | | | | | | | | As we were testing superproject setting revisionId attribute to SHA and reloading the manifest, we found out revisionId attribute is not being saved. Made the change to save the revisionId if it is not being saved. Tested the code with the following commands. $ ./run_tests -v Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I95fdf655b19648ad3e9aba10b9bed8bb9439deb6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296182 Reviewed-by: Ian Kasprzak <iankaz@google.com>
* Update _CheckForImmutableRevision to use git rev-listIan Kasprzak2021-02-051-1/+3
| | | | | | | | | | | | | | | | | | | | | _CheckForImmutableRevision is used to see if repo can skip fetching a project, but 'git rev-parse' with partial clone does a data fetch to accomplish this. Changed to use: 'git rev-list -1 --missing=allow-any <SHA>^0' which checks the local ref without fetching from the server first. Bug: [google internal] b/179477822 Testing: - Unit tests - Verified init/sync working on aosp-master - Verified wwith a pinned manifest that local ref check works (no fetch) Change-Id: If327b893c6658421f41df1f58c337f53b4c60ce6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296142 Reviewed-by: Dan Willemsen <dwillemsen@google.com> Tested-by: Ian Kasprzak <iankaz@google.com>
* sync: Added --filter=blob:none for git clone of superproject.Raman Tenneti2021-02-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | + This is without --depth option. This is done for reachability. Server doesn't know what you know about in the history so it always sends you the whole thing Which is very slow. If we have the full history it can send you incremental update history which is very small and fast. Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. .../WORKING_DIRECTORY$ repo sync --use-superproject Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I239de6d8f1c2ed6b4c69e7a78b8aa95338fa838c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/295362 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Do a git pull with --use-superproject if superproject tree already exists.Raman Tenneti2021-02-012-1/+8
| | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. .../WORKING_DIRECTORY$ repo sync --use-superproject Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I7e4b1e51ca1d18b836a5fa8d139a0765262ba500
* sync: Added --filter=blob:none (and no-depth) wduring git clone of superproject.Raman Tenneti2021-02-011-8/+29
| | | | | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and doing a repo sync --use-superproject twice. .../WORKING_DIRECTORY$ repo sync --use-superproject Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Ieea31445ca89ba1d217e779ec7a7a2ebe81ac518
* git_command.py: Handle unicode decode errorGaurav Pathak2021-01-281-1/+1
| | | | | | | | | | | | | | | | | repo diffmanifests saves git commit messages in buf and uses default utf-8 decoding, in some scenarios git commit message can itself contain a non UTF-8 character due to a typo or incorrect i18n.commitEncoding. e.g. d354d9afe923 [PATCH] fbcon: don\xb4t call set_par() in fbcon_init() if vc_mode == KD_GRAPHICS Convert the buf containing git commits to string if decoding to utf-8 encounters an error. Signed-off-by: Gaurav Pathak <gaurav.pathak@pantacor.com> Change-Id: If818562f0faaa5062c765fbea11dc0e1c86a24d7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294742 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Disable info about disabling pruning when quietAnders Björklund2021-01-221-2/+3
| | | | | | | | | | If you have a lot of shared projects, it spams. Bug: https://crbug.com/gerrit/13961 Change-Id: If3f5baef65930830af9a2cd01a1b593dd518ab09 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294049 Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: Added --use-superproject option and support for superproject.v2.12Raman Tenneti2021-01-216-7/+285
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Added "--use-superporject" option to sync.py to fetch project SHAs from superproject. If there are any missing projects in superprojects, it prints the missing entries and exits. If there are no missing entries, it will use SHAs from superproject to fetch the projects from git. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v tests/test_git_superproject.py $ ./run_tests -v Tested the sync code by copying all the repo changes into my Android AOSP checkout and adding <superporject> tag to default.xml. With local modification to the code to print the status, .../WORKING_DIRECTORY$ repo sync --use-superproject repo: executing 'git clone' url: sso://android/platform/superproject repo: executing 'git ls-tree' Success: [] Bug: https://crbug.com/gerrit/13709 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822 Reviewed-by: Mike Frysinger <vapier@google.com>
* launcher: bump version for new releaseMike Frysinger2021-01-211-1/+1
| | | | | | | Change-Id: I8a39630d482fc389cf497399102f795d7e576ff9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294122 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* Update "evt" field to be logged as a string type.Ian Kasprzak2021-01-192-1/+3
| | | | | | | | | | | | Testing: - Unit tests - Verified git trace log has "evt": "2" (vs "evt": 2 previously) Bug: https://crbug.com/gerrit/13966 Change-Id: I2e0c98dda0cccdd5cb6328105c11b93cd42676eb Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/294123 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Ian Kasprzak <iankaz@google.com>
* launcher: add a requirements framework to declare version dependenciesMike Frysinger2021-01-193-0/+222
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently we don't have a way for the checked out repo version to declare the version of tools it needs before we start running it. For somethings, like git, it's not a big deal as it can handle all the asserts itself. But for things like Python, it's impossible to reliably check before executing. We're in this state now: - we've been allowing Python 3.4, so the launcher accepts it - the repo codebase starts using Python 3.6 features - launcher tries to import us but hits syntax errors - user is left confused and assuming new repo is broken because they're seeing syntax errors This scenario is playing out with old launchers that still accept Python 2, and will continue to play out as time goes on and we want to require newer versions of Python 3. Lets create a JSON file to declare all these system requirements. That file format is extremely stable, so loading & parsing from even ancient versions of Python shouldn't be a problem. Then the launcher can read these settings and check the system state before attempting to execute any code. If the tools are too old, it can clearly diagnose & display information to the user as to the real problem (and not emit tracebacks or syntax errors). We have a couple of different tool version checks already (git, python, ssh) and can harmonize them in a single place. This also allows us to assert a reverse dependency if the need ever comes up: force the user to upgrade their `repo` launcher before we'll let them run us. Even though the launcher warns whenever a newer release is available, some users seem to ignore that, or they don't use repo that often (on the scale of years), and their upgrade jump is so dramatic that they fall back into the syntax error pit. Hopefully by the end of the year we can assume enough people have upgraded their launcher such that we can delete all of the duplicate version checks in the codebase. But until then, we'll keep them to maintain coverage. Change-Id: I5c12bbffdfd0a8ce978f39aa7f4674026fe9f4f8 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293003 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest_xml: - Added doc and testing of unknown tags/elements.Raman Tenneti2021-01-122-0/+27
| | | | | | | | | | | | | | | | Added this test to verify that older versions of repo can handle "<superproject" element. Tested by adding "<iankaz" unknown element. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v Bug: https://crbug.com/gerrit/13709 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I858d56f38cefcfcd14474efdd631a5a940c3ce47 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293482 Reviewed-by: Mike Frysinger <vapier@google.com>
* version: add remote tracking informationMike Frysinger2021-01-081-0/+2
| | | | | | | | | | This tells us what --repo-rev the user is using. Bug: https://crbug.com/1164415 Change-Id: Idb6c48e6ca5a4783c529717e6be38266bf7038b0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293143 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest_xml: initial support for <superproject>Raman Tenneti2021-01-083-0/+123
| | | | | | | | | | | | | | | | | | | | | | | | | At most one superproject may be specified. It will be used to specify the URL of superproject. It would have 3 attributes: remote, name, and default. Only "name" is required while the others have reasonable defaults. <remote name="superproject-url" review="<url>" /> <superproject remote="superproject-url" name="platform/superproject"/> TODO: This CL only implements the parsing logic and further work will be in followup CLs. Tested the code with the following commands. $ ./run_tests tests/test_manifest_xml.py $ ./run_tests -v Bug: https://crbug.com/gerrit/13709 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I5b4bba02c8b59601c754cf6b5e4d07a1e16ce167 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292982 Reviewed-by: Mike Frysinger <vapier@google.com>
* launcher: bump version for new releasev2.11.1Mike Frysinger2021-01-071-1/+1
| | | | | | | Change-Id: Ie0abee81e86046f412b42f08100041cfd3689c4a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292682 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* Fix bug in git trace2 event Write() function when no config present.Ian Kasprzak2021-01-072-21/+52
| | | | | | | | | | | | | | | | | | See https://bugs.chromium.org/p/gerrit/issues/detail?id=13706#c9 Added additional unit tests for Write() for additional test coverage. Testing: - Unit tests - Verified repo works with: - Valid trace2.eventtarget - Invalid trace2.eventtarget Bug: https://crbug.com/gerrit/13706 Tested-by: Ian Kasprzak <iankaz@google.com> Change-Id: I6b027cb2399bd03e453a132ad82e022a1f48476e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292762 Reviewed-by: Mike Frysinger <vapier@google.com>
* drop pyversion & is_python3 checkingMike Frysinger2021-01-0613-156/+20
| | | | | | | | | | | We're committed to Python 3 at this point, so purge all the is_python3 related dynamic checks. Bug: https://crbug.com/gerrit/10418 Change-Id: I4c8b405d6de359b8b83223c9f4b9c8ffa18ea1a2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292383 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* strip python2-only coding:utf-8 & print_function settingsMike Frysinger2021-01-0657-174/+0
| | | | | | | | | | We're committed to Python 3 at this point, so clean up boilerplate. Bug: https://crbug.com/gerrit/10418 Change-Id: Ib1719ba2eb65c53b94881a1a1bf203ddfcaaafed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292382 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* launcher: add test for version requirementsv2.11Mike Frysinger2021-01-061-0/+11
| | | | | | | | | | | Make sure the modules stay in sync in case one is updated but we forgot to update the other. Bug: https://crbug.com/gerrit/13795 Change-Id: I6de9533d45c083e5f7ad792ee6d541e23647de3f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292444 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* main: require Python 3.5 nowMike Frysinger2021-01-061-1/+1
| | | | | | | | | | | | | | | | | | | | | We've been warning about Python 3.4 for almost a year. This drops support for these systems: * Ubuntu Trusty: released Apr 2014, EOL Apr 2022 * Debian Jessie: released Apr 2015, EOL Jun 2020 So the min required distros would now be: * Ubuntu Xenial: released Sep 2015 w/Python 3.5 * Debian Stretch: released Jun 2017 w/Python 3.6 I don't think we're quite ready to drop Python 3.5 which would affect Ubuntu Xenial -- we'd have to update to Ubuntu Bionic from Apr 2018. Let's see how much the community reacts to loss of Python 3.4 first. Bug: https://crbug.com/gerrit/10418 Change-Id: Ib24a57818fdca49e23db53e1bdd1f4c76b4963f7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291502 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* Enable git trace2 event format logging.Ian Kasprzak2021-01-063-0/+375
| | | | | | | | | | | | | | | | | | | | | Ways to enable logging: 1) Set git's trace2.eventtarget config variable to desired logging path 2) Specify path via --git_trace2_event_log option A unique logfile name is generated per repo execution (based on the repo session-id). Testing: 1) Verified git config 'trace2.eventtarget' and flag enable logging. 2) Verified version/start/end events are expected format: https://git-scm.com/docs/api-trace2#_event_format 3) Unit tests Bug: https://crbug.com/gerrit/13706 Change-Id: I335eba68124055321c4149979bec36ac16ef81eb Tested-by: Ian Kasprzak <iankaz@google.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292262 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* launcher: abort if python3 reexec failedMike Frysinger2021-01-051-3/+4
| | | | | | | | | | | | | We don't support Python 2 anymore, so stop allowing it to fallback. If we try to run the latest version with Python 2, it just hits syntax errors which confuses people. Dump a clear error message that their system is too old and give up. Bug: https://crbug.com/gerrit/13795 Change-Id: I38c243cf09502f670cddad72c2d0148f736515e0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292443 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* launcher: support Python 3.5 for nowMike Frysinger2021-01-051-10/+23
| | | | | | | | | | | | | The codebase still supports Python 3.5, so allow use of that instead of requiring Python 3.6+. Supporting this mode well is a bit tricky as we want to first scan for newer versions before falling back to older ones. And we have to avoid infinite loops in the process. Bug: https://crbug.com/gerrit/13795 Change-Id: I47949a173899bfa9ab20d3fefa1a97bf002659f6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/292442 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: detach HEAD in internal worktree checkout.Remy Böhmer2020-12-261-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When checkout is done with Git worktrees then the HEAD in the bare-git repositories point to the initialized default (e.g. 'refs/heads/master'). This default branch does not exist locally and is not automatically created. When a user now creates a branch in any git repository named 'master' then it is no longer possible to get rid of this branch, neither is it possible to switch to another branch and switch back to this master branch. Git concludes the 'master' branch is already checked out (in the bare Git) and that results in a lockdown of this master branch. To repoduce this issue, run these commands in a repo tree checked out with --worktree: - git checkout master # assuming the remote repo has a master branch, # a local tracking branch master is created here - git checkout -b temp - git checkout master # This one now fails - git branch -d master # fails too The failure is caused by Git assuming the master branch is checked out by the bare git repository since HEAD is pointing towards it. To workaround this, we always detach HEAD in the bare-git when syncing. We don't need it to point to a ref in general, but we would like it to be valid so git tools "just work" if they're run in here. Signed-off-by: Remy Bohmer <oss@bohmer.net> Change-Id: I15c96604363c41f0d01c42f533174393097daeb5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290985 Reviewed-by: Mike Frysinger <vapier@google.com>
* Add parallelism to 'branches' commandChris McDonald2020-12-142-4/+43
| | | | | | | | | | | | | | | Spread the operation of querying which local branches exist across a pool of processes and build the name map of projects -> branches as these tasks finish rather than blocking on the entire query. The search operations are submitted in batches to reduce the overhead of interprocess communication. The `chunksize` argument used to control this batch size was selected by incrementing through powers of two until it stopped being faster. Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342 Tested-by: Chris Mcdonald <cjmcdonald@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-30 01:09:19 +0000