summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* tests: fix duplicate method from copy & paste errorMike Frysinger2021-03-041-2/+2
| | | | | | | | | Change-Id: Ib748c61b1e65aee6dff8b97a9753d14c470a827f Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/299002 Reviewed-by: Daniel Kutik <daniel.kutik@lavawerk.com> Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: fix variable typov2.13.2Mike Frysinger2021-03-031-1/+1
| | | | | | | | | Bug: https://crbug.com/gerrit/11293 Reported-by: Daniel Kutik <daniel.kutik@lavawerk.com> Change-Id: I37bac58aa1dc9ecc10e29253d14ff9e6fb42427c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298942 Reviewed-by: Ian Kasprzak <iankaz@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: relax include name rules for user-specified pathv2.13.1Mike Frysinger2021-03-023-15/+62
| | | | | | | | | | | | | Allow the user to specify relative or absolute or any other funky path that they want when using `repo init` or `repo sync`. Our goal is to restrict the paths in the remote manifest git repo we cloned from the network, not protect the user from themselves. Bug: https://crbug.com/gerrit/14156 Change-Id: I1ccfb2a6bd1dce2bd765e261bef0bbf0f8a9beb6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298823 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: greatly speed up processing overheadv2.13Mike Frysinger2021-03-011-42/+17
| | | | | | | | | | | | | | | | With the recent commit 0501b29e7ae072e0b10ea9ddd913ec6d5975f690 ("status: Use multiprocessing for `repo status -j<num>` instead of threading"), the limitation with project serialization no longer applies. It turns out that ad-hoc logic is expensive. In the CrOS checkout (~1000 projects w/8 jobs by default), it adds about ~7sec overhead to all invocations. With a fast nop run: time repo forall -j8 -c true This goes from ~11sec to ~4sec -- more than 50% speedup. Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: simplify arg passing to worker childrenMike Frysinger2021-03-011-20/+6
| | | | | | | | | | | The ProjectArgs function can be inlined which simplifies it quite a bit. We shouldn't need the custom exception handling here either. This also makes the next commit easier to review. Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: move nested func out to the classMike Frysinger2021-03-011-5/+7
| | | | | | | | | | | This is in preparation for simplifying the jobs support. The nested function is referenced in the options object which can't be pickled, so pull it out into a static method instead. Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* download: handle shared projects a bit betterMike Frysinger2021-03-011-2/+19
| | | | | | | | | | | | | | If a manifest checksout a project multiple times, repo download isn't able to accurately pick the right project. We were just picking the first result which could be a bit random for the user. If we hit that situation, check if the cwd is one of the projects, and if it isn't, we emit an error and tell the user it's an ambiguous request. Bug: https://crbug.com/gerrit/13070 Change-Id: Id1059b81330229126b48c7312569b37504808383 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: skip clone bundles when we've already initialized the object dirMike Frysinger2021-03-011-0/+6
| | | | | | | | | | | | | | The clone bundle logic assumes there is a one-to-one mapping between the projects/ and project-objects/ trees. When using shared projects (where we checkout different branches from the same project), this would lead us to fetching the same clone bundle multiple times. Automatically skip the clone bundle logic if the project-objects/ dir already exists. Bug: https://crbug.com/gerrit/10993 Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_refs: fix crash with binary . files in .git/refs/Mike Frysinger2021-02-281-4/+7
| | | | | | | | | | | | | | | | | | | | | | | On macOS, the Finder app likes to poop .DS_Store files in every path that the user browses. If the user pokes around the .git/ tree, it could generate a .DS_Store file in there too. When repo goes to read all the local refs, it tries to decode this binary file as UTF-8 and subsequently crashes. Since paths that begin with . are not valid refs, ignore them like we already do with paths that end in .lock. Also bump the check up to ignore dirs that match since that follows the git rules: they apply to any component in its path, not just the final path (name). We don't implement the full valid ref algorithm that git employs as it's a bit complicated, and we only really need to focus on what will practically show up locally. Bug: https://crbug.com/gerrit/14162 Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com>
* git_command: pass GIT_DIR on Windows with /Mike Frysinger2021-02-281-0/+3
| | | | | | | | | | | | | | | | | | | When using Git under Windows, it seems that Git doesn't always parse GIT_DIR correctly when it uses the Windows \ form, but does when it uses / only. For example, when using worktrees: $ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test $ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git fd00dbbd0c06 (detached HEAD) ..\..\..\..\..\src\src\third_party\lss\.git fd00dbbd0c06 (detached HEAD) ..\..\..\..\..\tools\test\.git fd00dbbd0c06 (detached HEAD) Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: validate project name & path and include name attributesMike Frysinger2021-02-282-75/+173
| | | | | | | | | | | These attribute values are used to construct local filesystem paths, so apply the existing filesystem checks to them. Bug: https://crbug.com/gerrit/14156 Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* manifest: refactor the filesystem checking logic for more reuseMike Frysinger2021-02-282-5/+33
| | | | | | | | | | | | | | | | | | | | | This function is currently written with copyfile & linkfile in mind. Generalize the logic & function arguments slightly so we can reuse in more places that make sense. This changes the validation logic slightly too in that we no longer allow "." for the dest attribute with copyfile & linkfile, nor for the src attribute with copyfile. We already rejected those later on when checking against the active filesystem, but now we reject them a little sooner when parsing. The empty path check isn't a new requirement exactly -- repo used to crash on it, so it was effectively blocked, but now we diagnosis it. Bug: https://crbug.com/gerrit/14156 Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* checkout: add --jobs supportMike Frysinger2021-02-271-10/+27
| | | | | | | | | | | Use multiprocessing to run in parallel. When operating on multiple projects, this can speed things up. Across 1000 repos, it goes from ~9sec to ~5sec with the default -j8. Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: switch local checkout to multiprocessingMike Frysinger2021-02-271-124/+52
| | | | | | | | | | | | | | | | This avoids GIL limitations with using threads for parallel processing. In a CrOS checkout with ~1000 repos, the nop case goes from ~6 sec down to ~4 sec with -j8. Not a big deal, but shows that this actually works to speed things up unlike the threading model. This reworks the checkout logic to return results for processing in the main thread instead of leaving every thread to do its own processing. Bug: https://crbug.com/gerrit/12389 Change-Id: I143e5e3f7158e83ea67e2d14e5552153a874248a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298063 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* abandon/start: add --jobs supportMike Frysinger2021-02-272-38/+83
| | | | | | | | | | | Use multiprocessing to run in parallel. When operating on multiple projects, this can greatly speed things up. Across 1000 repos, it goes from ~30sec to ~3sec with the default -j8. Change-Id: I0dc62d704c022dd02cac0bd67fe79224f4e34095 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297484 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* progress: include execution time summaryMike Frysinger2021-02-261-4/+23
| | | | | | | | | | We're already keeping tracking of the start time, so might as well use it to display overall execution time for steps. Change-Id: Ib4cf8b2b0dfcdf7b776a84295d59cc569971bdf5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298482 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: superproject - support for switching hosts and switching branches.Raman Tenneti2021-02-252-53/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | + superproject will be fetched into a directory with the name “<remote name>-superproject.git” instead of the current “superproject.git” folder. + Deleted _Clone method and added _Init method. + _Init method will do “git init --bare <remote>-superproject.git”. It will create the folder and set up a bare repository in <remote>-superproject.git folder. + _Fetch method, will pass <remote url>, <branch> arguments. Moved the --filter argument from “git clone” to “git fetch”. _Fetch method will execute the following command to fetch superproject. Added --no-tags argument. master: git fetch <remote url> --force --no-tags --filter blob:none branch: git fetch <remote url> --force --no-tags --filter blob:none \ <branch>:<branch> + Performance improvements for aosp-master ++ repo init performance improved from 35 seconds to 17 seconds. ++ repo init --use-superproject is around 5 to 7 secsonds slower. ++ repo sync --use-superproject is around 3 to 4 minutes faster. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m20.648s user 0m8.046s sys 0m3.271s + Without superproject $ time repo init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main real 0m13.078s user 0m9.783s sys 0m2.528s $ time repo_dev sync -c -j32 --use-superproject ... real 15m7.072s user 110m7.216s sys 20m17.559s + Without superproject $ time repo sync -c -j32 ... real 19m25.644s user 91m56.331s sys 20m59.170s Bug: [google internal] b/180492484 Bug: [google internal] b/179470886 Bug: [google internal] b/180124069 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ib04bd7f1e25ceb75532643e58ad0129300ba3299 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297702 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* grep: move nested func out to the classMike Frysinger2021-02-251-37/+35
| | | | | | | | | | | This is in preparation for adding jobs support. The nested function is referenced in the options object which can't be pickled, so pull it out into a static method instead. Change-Id: I280ed2bf26390a0203925517a0d17c13053becaa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297983 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: capture all git output by defaultMike Frysinger2021-02-252-6/+22
| | | | | | | | | | | | The default sync output should show a progress bar only for successful commands, and the error output for any commands that fail. Implement that policy here. Bug: https://crbug.com/gerrit/11293 Change-Id: I85716032201b6e2b45df876b07dd79cb2c1447a5 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297905 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress/sync: include active number of jobsMike Frysinger2021-02-252-12/+35
| | | | | | | | | | Provide a bit more info to users that things are actively running. Bug: https://crbug.com/gerrit/11293 Change-Id: Ie8eeaa8804d1ca71cf5c78ad850fa2d17d26208c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297904 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* progress: always enable always_print_percentageMike Frysinger2021-02-252-18/+11
| | | | | | | | | | | | | | | | | | | The idea for skipping some progress updates was to avoid spending too much time on the progress bar itself. Unfortunately, for large projects (100s if not 1000s) of repos, we get into the situation with large/slow checkouts that we skip showing updates when a repo finishes, but not enough repos finished to increase the percent. Since the progress bar should be relatively fast compared to the actual network & local dick operations, have it show an update whenever the caller requests it. A test with ~1000 repos shows that the progress bar in total adds <100ms. Bug: https://crbug.com/gerrit/11293 Change-Id: I708a0c4bd923c59c7691a5b48ae33eb6fca4cd14 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297903 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: support -b HEAD as a shortcut to "the default"Mike Frysinger2021-02-252-6/+10
| | | | | | | | | | | When people switch to non-default branches, they sometimes want to switch back to the default, but don't know the exact name for that branch. Add a -b HEAD shortcut for that. Change-Id: I090230da25f9f5a169608115d483f660f555624f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297843 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* project: fix http error retry logicMike Frysinger2021-02-251-4/+6
| | | | | | | | | | | | When sync moved to consume clone output, it merged stdout & stderr, but the retry logic in this function is based on stderr only. Move it over to checking stdout. Change-Id: I71bdc18ed25c978055952721e3a768289d7a3bd2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297902 Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* tests: fix init subcmd after url changeMike Frysinger2021-02-251-1/+1
| | | | | | | | | | My recent 401c6f072564966437a74dc2f33280a85d79dc84 ("init: make --manifest-url flag optional") commit broke the unittest. Change-Id: I19ad0e8c8cbb84ab5474ebc370e00acfe957e136 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298223 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* tests: refactor manifest testsMike Frysinger2021-02-251-90/+98
| | | | | | | | | | | The XmlManifestTests class is getting to be large and we're only adding more to it. Factor out the core logic into a new TestCase so we can reuse it to better group more tests. Change-Id: I5113444a4649a70ecfa8d83d3305959a953693f7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298222 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use superproject if manifest's config has superproject enabled.Raman Tenneti2021-02-251-1/+3
| | | | | | | | | | | | | | | | | | | If --use-superproject is passed as argument to "repo init", then --use-superproject need not be specified during "repo sync". Tested the code with the following commands. $ time repo_dev sync -c -j32 ... WARNING: --use-superproject is experimental and not for general use Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: Ibb33f3038a2515f74a6c4f7cb785d354b26ee680 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298102 Tested-by: Raman Tenneti <rtenneti@google.com> Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: Ian Kasprzak <iankaz@google.com>
* docs: add warnings about repos data modelMike Frysinger2021-02-251-2/+22
| | | | | | | | | | For people coming across these docs and thinking that repo's methods are good to replicate, add a note warning them against doing so. Change-Id: I443a783794313851a6e7ba1c39baebac988bff9a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298164 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* platform_utils: delete unused FileDescriptorStreams APIsMike Frysinger2021-02-241-158/+0
| | | | | | | | | | | | | | Now that we've converted the few users of this over to subprocess APIs, we don't need this anymore. It's been a bit hairy to maintain across different operating systems, so there's no desire to bring it back. Using multiprocessing Pool to batch things has been working better in general anyways. Change-Id: I10769e96f60ecf27a80d8cc2aa0d1b199085252e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297682 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: update event is_set APIMike Frysinger2021-02-231-7/+7
| | | | | | | | | Python 3 renamed this method from isSet to is_set. Change-Id: I8f9bb0b302d55873bed3cb20f2d994fa2d082157 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297742 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_command: switch process capturing over to subprocessMike Frysinger2021-02-231-33/+7
| | | | | | | | | | | | Now that these code paths are all synchronous, there's no need to run our own poll loop to read & pass thru/save output. Delete all of that and just let the subprocess module take care of it all. Change-Id: Ic27fe71b6f964905cf280ce2b183bb7ee46f4a0d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297422 Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* diff: add --jobs supportMike Frysinger2021-02-232-5/+41
| | | | | | | | | Use multiprocessing to run diff in parallel. Change-Id: I61e973d9c2cde039d5eebe8d0fe8bb63171ef447 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297483 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* forall: rewrite parallel logicMike Frysinger2021-02-221-66/+40
| | | | | | | | | | | | | | | | | This fixes intermingling of parallel jobs and simplifies the code by switching to subprocess.run. This also provides stable output in the order of projects by returning the output as a string that the main loop outputs. This drops support for interactive commands, but it's unclear if anyone was relying on that, and the default behavior (-j2) made that unreliable. If it turns out someone still wants this, we can look at readding it. Change-Id: I7555b4e7a15aad336667292614f730fb7a90bd26 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297482 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* forall: improve pool logicMike Frysinger2021-02-221-15/+10
| | | | | | | | | | Use a pool contextmanager to take care of the messy details like properly cleaning it up when aborting. Change-Id: I264ebb591c2e67c9a975b6dcc0f14b29cc66a874 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297243 Reviewed-by: Chris Mcdonald <cjmcdonald@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* status: improve parallel execution stabilityMike Frysinger2021-02-223-13/+22
| | | | | | | | | | | | | | | | | The status command runs a bunch of jobs in parallel, and each one is responsible for writing to stdout directly. When running many noisy jobs in parallel, output can get intermingled. Pass down a StringIO buffer for writing to so we can return the entire output as a string so the main job can handle displaying it. This fixes interleaved output as well as making the output stable: we always display results in the same project order now. By switching from map to imap, this ends up not really adding any overhead. Bug: https://crbug.com/gerrit/12231 Change-Id: Ic18b07c8074c046ff36e306eb8d392fb34fb6eca Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297242 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* command: unify --job option & default valuesMike Frysinger2021-02-225-25/+27
| | | | | | | | | | | | Extend the Command class to support adding the --jobs option to the parser if the command declares it supports running in parallel. Also pull the default value used for the number of local jobs into the command module so local commands can share it. Change-Id: I22b0f8d2cf69875013cec657b8e6c4385549ccac Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297024 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Chris Mcdonald <cjmcdonald@google.com>
* git_command: make execution synchronousMike Frysinger2021-02-201-6/+6
| | | | | | | | | | | | | Every use of GitCommand in the tree just calls Wait as soon as it's instantiated. Move the bulk of the logic into the init path to make the call synchronous to simplify. We'll cleanup the users of the Wait API to follup commits -- having this split makes it easier to track down regressions. Change-Id: I1e8c519efa912da723749ff7663558c04c1f491c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297244 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* use simpler super() magicMike Frysinger2021-02-194-17/+16
| | | | | | | | | | Python 3 has a simpler super() style so switch to it to make the code a little simpler and to stop pylint warnings. Change-Id: I1b3ccf57ae968d56a9a0bcfc1258fbd8bfa3afee Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297383 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* error: fix pickling of all exceptionsMike Frysinger2021-02-192-8/+61
| | | | | | | | | | | | | | Make sure all our custom exceptions can be pickled so that if they get thrown in a multiprocess subprocess, we don't crash & hang due to multiprocessing being unable to pickle+unpickle the exception. Details/examples can be seen in Python reports like: https://bugs.python.org/issue13751 Change-Id: Iddf14d3952ad4e2867cfc71891d6b6559130df4b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297382 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: make --manifest-url flag optionalMike Frysinger2021-02-182-8/+23
| | | | | | | | | | | Since the --manifest-url flag is always required when creating a new checkout, allow the url to be specified via a positional argument. This brings it a little closer to the `git clone` UI. Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* gitc_utils: rewrite to use multiprocessingMike Frysinger2021-02-181-21/+29
| | | | | | | | | | | This is the only code in the tree that uses GitCommand asynchronously. Rewrite it to use multiprocessing.Pool as it makes the code a little bit easier to understand and simpler. Change-Id: I3ed3b037f24aa1e9dfe8eec9ec21815cdda7678a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297143 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com>
* project: make diff tools synchronousMike Frysinger2021-02-181-52/+44
| | | | | | | | | | | | | | | | | | | These are the only users in the tree that process the output as it's produced. All others capture all the output first and then process the results. However, these functions still don't fully return until it's finished processing, and these funcs are in turn used in other synchronous code paths. So it's unclear whether anyone will notice that it's slightly slower or less interactive. Let's try it out and see if users report issues. This will allow us to simplify our custom GitCommand code and move it over to Python's subprocess.run, and will help fix interleaved output when running multiple commands in parallel (e.g. `repo diff -j8`). Change-Id: Ida16fafc47119d30a629a8783babeba890515de0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297144 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Jonathan Nieder <jrn@google.com>
* git_command: rework stdin handlingMike Frysinger2021-02-172-10/+8
| | | | | | | | | | | We only provide input to GitCommand in one place, so inline the logic to be more synchronous and similar to subprocess.run. This makes the code simpler and easier to understand. Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* trim redundant pass statementsMike Frysinger2021-02-163-3/+0
| | | | | | | | | Clean up a few linter warnings. Change-Id: I531d0263a202435d32d83d87ec24998f4051639c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297062 Reviewed-by: Jonathan Nieder <jrn@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_command: use subprocess.run for version infoMike Frysinger2021-02-161-5/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: If125ea7d776cdfa38a0440a2b03583de079c4839 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297023 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* sync: use subprocess.run to verify tagsMike Frysinger2021-02-161-13/+4
| | | | | | | | | The code is a bit simpler & easier to reason about. Change-Id: I149729c7d01434b08b58cc9715dcf0f0d11201c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297022 Reviewed-by: Michael Mortensen <mmortensen@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* init: added --use-superproject option to clone superproject.v2.12.2Raman Tenneti2021-02-116-136/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added --no-use-superproject to repo and init.py to disable use of manifest superprojects. Replaced the term "sha" with "commit id". Added _GetBranch method to Superproject object. Moved shared code between init and sync into SyncSuperproject function. This function either does git clone or git fetch. If git fetch fails it does git clone. Changed Superproject constructor to accept manifest, repodir and branch to avoid passing them to multiple functions as argument. Changed functions that were raising exceptions to return either True or False. Saved the --use-superproject option in config as repo.superproject. Updated internal-fs-layout.md document. Updated the tests to work with the new API changes in Superproject. Performance for the first time sync has improved from 20 minutes to around 15 minutes. Tested the code with the following commands. $ ./run_tests -v Tested the sync code by using repo_dev alias and pointing to this CL. $ repo init took around 20 seconds longer because of cloning of superproject. $ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject ... real 0m35.919s user 0m21.947s sys 0m8.977s First run $ time repo sync --use-superproject ... real 16m41.982s user 100m6.916s sys 19m18.753s No difference in repo sync time after the first run. Bug: [google internal] b/179090734 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Raman Tenneti <rtenneti@google.com>
* diffmanifests/sync: simplify repodir lookupMike Frysinger2021-02-112-3/+3
| | | | | | | | | | We have access to repodir on the command object itself, so we don't need to pull it indirectly out of the manifest object. Change-Id: I8688fb1c84979825efa966dc787e78c6f7ba3823 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296542 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* git_config: add SetBoolean helperMike Frysinger2021-02-114-14/+81
| | | | | | | | | A little sugar simplifies the code a bit. Change-Id: Ie2b8a965faa9f9ca05c7be479d03e8e073cd816d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296522 Reviewed-by: Raman Tenneti <rtenneti@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* superproject: Pass branch to git ls-tree.Raman Tenneti2021-02-081-4/+8
| | | | | | | | | | | | | | Tested the code with the following commands. $ ./run_tests -v Bug: [google internal] b/179702819 Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I7d2b609ac2f927c94701757aa1502ba236afe7c0 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296342 Reviewed-by: Mike Frysinger <vapier@google.com>
* sync: pass --bare option when doing git clone of superproject.v2.12.1Raman Tenneti2021-02-083-26/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | Changed "git pull" to "git fetch" as we are using --bare option. Used the following command to fetch: git fetch origin +refs/heads/*:refs/heads/* --prune Pass --branch argument to Superproject's UpdateProjectsRevisionId function. Returned False/None when directories don't exist instead of raise GitError exception from _Fetch and _LsTree functions. The caller of Fetch does Clone if Fetch fails. Tested the code with the following commands. $ ./run_tests -v Tested the init and sync code by copying all the repo changes into my Android AOSP checkout and running repo sync with --use-superproject option. Bug: https://crbug.com/gerrit/13709 Bug: https://crbug.com/gerrit/13707 Tested-by: Raman Tenneti <rtenneti@google.com> Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222 Reviewed-by: Mike Frysinger <vapier@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-31 23:05:44 +0000