summaryrefslogtreecommitdiffstats
path: root/git_command.py
Commit message (Collapse)AuthorAgeFilesLines
* Use hash for ControlPath instead of full variablesAnders Björklund2020-04-151-1/+35
| | | | | | | | | | | | | | | | | The generated socket path can be too long, if your FQDN is very long... Typical error message from ssh client: unix_listener: path "/tmp/ssh-fqduawon/master-USER@HOST:PORT.qfCZ51OAZgTzVLbg" too long for Unix domain socket Use a hashed version instead, to keep within the socket file path limit. This requires OpenSSH_6.7p1, or later. Change-Id: Ia4bb9ae8aac6c4ee31d5a458f917f3753f40001b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255632 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Anders Björklund <anders.bjorklund.2@volvocars.com>
* assume environment always accepts stringsMike Frysinger2020-02-191-13/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Different Python & OS versions have different environ behavior wrt accepted types & encoding. Since we're migrating to be Python 3 only, lets change our code to assume strings always work as that's what the newer Python 3 does. This will fail under Python 2 for some env vars, mostly on Windows, but the effort of maintaining shim layers that can handle these edge cases isn't worth it when we're dropping that code. We leave the logic in the `repo` launcher for now as it is simple, and we want it to be able to switch versions a bit longer than the rest of the tree. Here's the support table: | *NIX | Windows | Python 2 | ASCII string | str or bytes, not unicode | Python 3 | str or bytes | str only | Windows uses strings natively in its environment all the time. But it doesn't allow unicode strings under Python 2, so we have to encode. Python 2 on *NIX is funky in that it always lowers to ASCII, so we had to manually encode to avoid errors regardless of unicode or str. Python 3 on Windows & *NIX will accept strings. *NIX will also accept bytes but Windows will not. Bug: https://crbug.com/gerrit/12145 Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* project: move successful fetch output behind verboseMike Frysinger2020-02-181-2/+6
| | | | | | | | | | | | | Syncing projects works fine the majority of the time. So rather than dump all of that noisy output to stdout, lets capture it and only show when things fail or in verbose mode. This tidies up the default `repo sync` output. Bug: https://crbug.com/gerrit/11293 Change-Id: I8314dd92e1e6aadeb26e36a8c92610da419684e6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/255413 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* Fix blank line issues reported by flake8David Pursehouse2020-02-121-0/+14
| | | | | | | | | | | | | | | | | | | | - E301 expected 1 blank line - E302 expected 2 blank lines - E303 too many blank lines - E305 expected 2 blank lines after class or function definition - E306 expected 1 blank line before a nested definition Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E301,E302,E303,E305,E306 Manually fix issues in project.py caused by misuse of block comments. Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix indentation issues reported by flake8David Pursehouse2020-02-121-4/+4
| | | | | | | | | | | | | | | | | | | | | - E121 continuation line under-indented for hanging indent - E122 continuation line missing indentation or outdented - E125 continuation line with same indent as next logical line - E126 continuation line over-indented for hanging indent - E127 continuation line over-indented for visual indent - E128 continuation line under-indented for visual indent - E129 visually indented line with same indent as next logical line - E131 continuation line unaligned for hanging indent Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E121,E122,E125,E126,E127,E128,E129,E131 Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix various whitespace issues reported by pyflakesDavid Pursehouse2020-02-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | - E201 whitespace after '[' - E202 whitespace before '}' - E221 multiple spaces before operator - E222 multiple spaces after operator - E225 missing whitespace around operator - E226 missing whitespace around arithmetic operator - E231 missing whitespace after ',' - E261 at least two spaces before inline comment - E271 multiple spaces after keyword Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E201,E202,E221,E222,E225,E226,E231,E261,E271 Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix flake8 E251 unexpected spaces around keyword / parameter equalsDavid Pursehouse2020-02-121-13/+13
| | | | | | | | | | | Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place --select E251 Change-Id: I58009e1c8c91c39745d559ac919be331d4cd9e77 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254598 Tested-by: David Pursehouse <dpursehouse@collab.net> Reviewed-by: Mike Frysinger <vapier@google.com>
* repo: lower min version of git a bitMike Frysinger2020-02-121-2/+11
| | | | | | | | | | | | | | | | | | | | | | We were perhaps a bit too hasty to jump to git-2.10. Existing LTS releases of Ubuntu are quite old still: Trusty has 1.9 while Xenial has 2.5. While we plan on dropping support for those eventually as we migrate to Python 3.6, we don't need to be so strict just yet on the git versions. We also want to disconnect the version the repo launcher requires from the version the rest of the source tree requires. The repo launcher doesn't need as many features, and being flexible there allows us more freedom to upgrade & rollback as needed. So we'll allow git-1.7 again, but start warning on any users older than git-1.9. This aligns better with existing LTS releases, and gives users a chance to start upgrading before we cut them off. Change-Id: I140305dd8e42c9719c84e2aee0dc6a5c5b18da25 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254573 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* repo: raise min version of gitMike Frysinger2020-02-051-1/+2
| | | | | | | | | | | | The git-2.10 series was released in 2016. Since we're moving to require Python 3.6 which was also released in 2016, bumping up the git version seems reasonable. Also we don't really test any git versions close to as old as 1.7.2 which was released in 2010. Change-Id: Ib71b714de6cd0b7dd50d0b300b108a560ee27331 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253134 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* git_command: set GIT_HTTP_USER_AGENT on all requestsMike Frysinger2019-10-011-0/+13
| | | | | | | | | | | | | | | | | We've been setting the User-Agent header when making connections from repo itself, but not when running git (as the latter will set up User-Agent itself). Our Gerrit/Git admins say it'll be helpful if we pass through the repo version settings even when running git. We currently set GIT_HTTP_USER_AGENT and not GIT_USER_AGENT as it's unclear if the extended form works over all protocols. We can wait for a user request. Bug: https://crbug.com/gerrit/11144 Change-Id: I21d293f49534058dbc23225152451df26c5b7bef Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239233 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* git_command: refactor User-Agent settingsMike Frysinger2019-10-011-49/+79
| | | | | | | | | | | | | | | Convert the RepoUserAgent function into a UserAgent class. This makes it cleaner to hold internal state, and will make it easier to add a separate git User-Agent, although we don't do it here. We make the RepoSourceVersion independent of GitCommand so that it can be called by the class (later). Bug: https://crbug.com/gerrit/11144 Change-Id: Iab4e1f974b8733a36b243b2d03f5085a96effa19 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239232 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* move UserAgent to git_command for wider userMike Frysinger2019-10-011-0/+46
| | | | | | | | | | | We can't import the main module, so move the UserAgent helper out of it and into the git_command module so it can be used in more places. Bug: https://crbug.com/gerrit/11144 Change-Id: I8093c8a20bd1dc7d612d0e2a85180341817c0d86 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231057 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* rename local trace moduleMike Frysinger2019-08-271-1/+1
| | | | | | | | | | There is a standard Python "trace" module, so having a local trace.py prevents us being able to import that. Rename the module to avoid. Change-Id: I23e29ec95a2204bb168a641323d05e76968d9b57 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234832 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* Add support for partial clone.v1.13.4Xin Li2019-07-161-2/+4
| | | | | | | | | | | | | A new option, --partial-clone is added to 'repo init' which tells repo to utilize git's partial clone functionality, which reduces disk and bandwidth usage when downloading by omitting blob downloads initially. Different from restricting clone-depth, the user will have full access to change history, etc., as the objects are downloaded on demand. Change-Id: I60326744875eac16521a007bd7d5481112a98749 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* git_command: drop custom version helperMike Frysinger2019-07-111-12/+2
| | | | | | | | | | | | Since ParseGitVersion can call `git --version` automatically, we don't need this duplicate version() helper anymore. The only other user is the `repo version` code, so convert that to version_tuple().full. Bug: https://crbug.com/gerrit/11144 Change-Id: I9d77822fc39f4ba28884d9183359169cabf5f17d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/231055 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* set default file encoding to utf-8Mike Frysinger2019-06-131-0/+1
| | | | | | | | There's no reason to support any other encoding in these files. This only affects the files themselves and not streams they open. Bug: https://crbug.com/gerrit/10418 Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
* Set GIT_SSH_VARIANT when setting GIT_SSHJonathan Nieder2017-10-191-0/+1
| | | | | | | | | | | | | | | | | | | | Make it explicit that the ssh wrapper we use for control master support accepts OpenSSH-compatible command line arguments instead of asking Git to guess. The GIT_SSH_VARIANT setting was introduced in Git v2.13.0-rc0~3^2~2 (2017-02-01) as a more reliable detection method than relying on the ssh command name. Fortunately the default variant was 'ssh' (i.e., OpenSSH-compatible) so this wasn't initially required. Now Git wants to start using more OpenSSH features (-o SendEnv=GIT_PROTOCOL), and in order to do so its ssh variant detection will need to be tweaked. Set GIT_SSH_VARIANT explicitly so this helper can continue to work regardless of how Git modifies its autodetection. Reported-by: William Yan <wyan@google.com> Change-Id: I6bf2c53b4eb5303a429eae6cb68e0a5ccce89064
* Make "git command" and "forall" work on WindowsRenaud Paquay2017-05-291-21/+8
| | | | | | | | | | Python on Windows does not support non blocking file operations. To workaround this issue, we instead use Threads and a Queue to simulate non-blocking calls. This is happens only when running with the native Windows version of Python, meaning Linux and Cygwin are not affected by this change. Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
* Add rpc: to default protocol whitelistv1.12.33Jonathan Nieder2016-02-261-1/+1
| | | | Change-Id: I57e1c3d93c0ce56da9c487df65eb3d258e0260e8
* Set GIT_ALLOW_PROTOCOL to limit dangerous protocolsDan Willemsen2015-11-261-0/+3
| | | | | | | | | See git commit 33cfccbbf35a -- some protocols allow arbitrary command execution as part of the URL. Instead of blindly allowing those, whitelist the allowed URL protocols unless the user has already done so. Bug: Issue 210 Change-Id: I6bd8e721aa5e3dab53ef28cfdc8fde33eb74ef76
* git_command: only decode when neededAnthony King2015-06-031-1/+4
| | | | | | strings no longer need decoding, since unicode is str Change-Id: I9516d298fee7ddc058452394b7759327fe3aa7a8
* decode the buffer before appendingAnthony King2015-06-031-0/+2
| | | | | | | | | | output from a process is in bytes in python3. we need to decode it. in Python3, bytes don't have an encode attribute. use this to identify it. Change-Id: I152f2ec34614131027db680ead98b53f9b321ed5
* Always capture output for GitCommandJohn L. Villalovos2015-03-261-12/+52
| | | | | | | | | | | | | | Switch the GitCommand program to always capture the output for stdout and stderr. And by default print the output while running. The options capture_stdout and capture_stderr have effectively become options to supress the printing of stdout and stderr. Update the 'git fetch' to use '--progress' so that the progress messages will be displayed. git checks if the output location isatty() and if it is not a TTY it will by default not print the progress messages. Change-Id: Ifdae138e008f80a59195f9f43c911a1a5210ec60
* Decode git versionChirayu Desai2014-08-201-2/+2
| | | | | | | | | Used by 'repo --version' With Python 3, * Before: b'git version 2.1.0' * After: git version 2.1.0 Change-Id: I4321bb0f09e92cda1123c35910338b940e82a305
* Share git version parsing code with wrapper modulev1.12.12Conley Owens2014-01-301-11/+3
| | | | | | | 'repo' and 'git_command.py' had their own git version parsing code. This change shares that code between the modules. DRY is good. Change-Id: Ic896d2dc08353644bd4ced57e15a91284d97d54a
* Respect version hyphenationv1.12.11Conley Owens2014-01-301-1/+1
| | | | | | | | The last change regarding version parsing lost handling of version hyphenation, this restores that. In otherwords, 1.1.1-otherstuff is parsed as (1,1,1) instead of (1,1,0) Change-Id: I3753944e92095606653835ed2bd090b9301c7194
* Handle release candidates in git version parsingConley Owens2014-01-301-4/+8
| | | | | | | Right now repo chokes on git versions like "1.9.rc1". This change treats 'rc*' as a '0'. Change-Id: I612b7b431675ba7415bf70640a673e48dbb00a90
* Fix some python3 encoding issuesChirayu Desai2013-11-211-1/+1
| | | | | | * Add .decode('utf-8') where needed * Add 'b' to `open` where needed, and remove where unnecessary Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
* Change print statements to work in python3Sarah Owens2012-11-131-2/+3
| | | | | | This is part of a series of changes to introduce Python3 support. Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
* Fix pylint warning W0108: Lambda may not be necessaryDavid Pursehouse2012-11-071-2/+2
| | | | | | Remove unnecessary usage of lambda. Change-Id: I06d41933057d60d15d307ee800cca052a44754c6
* Even more coding style cleanupDavid Pursehouse2012-10-301-3/+3
| | | | | | | | | | | Fixing some more pylint warnings: W1401: Anomalous backslash in string W0623: Redefining name 'name' from outer scope W0702: No exception type(s) specified E0102: name: function already defined line n Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
* Use modern Python exception syntaxSarah Owens2012-10-231-1/+1
| | | | | | | | | | | "except Exception as e" instead of "except Exception, e" This is part of a transition to supporting Python 3. Python >= 2.6 support "as" syntax. Note: this removes Python 2.5 support. Change-Id: I309599f3981bba2b46111c43102bee38ff132803
* Rename "dir" variablesMickaël Salaün2012-10-101-4/+4
| | | | | | | The variable name "dir" conflicts with the name of a Python built-in function: http://docs.python.org/library/functions.html#dir Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
* Fix bug in version_tuple to handle strings with -rc#Chad Jones2012-07-311-1/+1
| | | | | | | Example of version string that caused a problem: git version 1.7.11-rc3 Change-Id: I8a68b6b37f7b2ded23a1f8ae0d12131050a8807b CC: sop@google.com
* Single quote http.proxy in GIT_CONFIG_PARAMETERSv1.9.4Shawn O. Pearce2012-06-131-1/+1
| | | | | | | | Git requires the values in this environment variable to be single quoted. repo must wrap the expression into '' before adding it to the environment. Change-Id: I20a1fb8772f9aa6e9fd5a0516c981c2ca020ef05
* Use GIT_CONFIG_PARAMETERS instead of -c for http.proxyv1.9.3Shawn O. Pearce2012-06-051-2/+6
| | | | | | | | | Ancient versions of Git don't understand the -c command line flag that we tried to use to pass http_proxy down into Git on Darwin. Use the environment variable instead, to more gracefully degrade with these old versions. Change-Id: Iffffa32088c1fd803895b990b3377ecfec6a1b14
* Pass http_proxy as -c http.proxy on Mac OS XShawn O. Pearce2012-05-251-0/+2
| | | | | | | | The system libcurl library seems to ignore http_proxy on Mac OS X systems. Copy the http_proxy environment variable (if set) as `git -c http.proxy` whenever running a Git command. Change-Id: I0ab29336897178f70b85092601f9fcc306dd17e1
* Stabilize repo communication with subprocesses.Ulrik Sjölin2011-11-291-19/+3
| | | | | | | | | Make repo use the standard way in python to work with pipes. Communication via pipes to sub processes is done by calling communicate(). This will make repo not hang every now and then. Change-Id: Ibe2c4ecbdbcbe72f0b725ca50d54088e5646fc5d
* Enhance HTTP supportShawn O. Pearce2011-09-191-16/+19
| | | | | | | | | | | | | | Setting REPO_CURL_VERBOSE=1 in the environment will register a debug level HTTPHandler on the urllib2 library, showing HTTP requests and responses on the stderr channel of repo. During any HTTP or HTTPS request created inside of the repo process, a custom User-Agent header is now defined: User-Agent: git-repo/1.7.5 (Linux) git/1.7.7 Python/2.6.5 Change-Id: Ia5026fb1e1500659bd2af27416d85e205048bf26 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Use os.environ.copy() instead of dict()Shawn O. Pearce2011-01-091-1/+1
| | | | | Signed-off-by: Shawn O. Pearce <sop@google.com> (cherry picked from commit 3218c13205694434edb2375ab8a8515554eed366)
* Encode the environment variables passed to gitShawn O. Pearce2011-01-091-4/+7
| | | | | | | | Windows allows the environment to have unicode values. This will cause Python to fail to execute the command. Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: kill git fetch process before SSH control master processv1.6.9.6Shawn O. Pearce2010-05-111-1/+28
| | | | | | | | | | | | | | | | If the SSH control master process is killed while an active git fetch is using its network socket, the underlying SSH client may not realize the connection was broken. This can lead to both the client and the server waiting indefinitely for network messages which will never be sent. Work around the problem by keeping track of any processes that use the tunnels we establish. If we are about to kill any of the SSH control masters that we started, ensure the clients using them are successfully killed first. Change-Id: Ida6c124dcb0c6a26bf7dd69cba2fbdc2ecd5b2fc Signed-off-by: Shawn O. Pearce <sop@google.com>
* Rename _ssh_sock() to fix code style issue.Nico Sallembien2010-04-271-2/+2
| | | | | Since _ssh_sock is imported out of the git_command module, the leading underscore should be removed from the function name.
* Refactor git version detection for reuseShawn O. Pearce2009-06-121-0/+24
| | | | | | | | This way we can use it to detect feature support in the underlying git, such as new options or commands that have been added in more recent versions. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Work around 'ControlPath too long' on Mac OS XShawn O. Pearce2009-04-211-1/+4
| | | | | | | | | | | | Mac OS X sets TMPDIR to a very long path within /var, so long that a socket created in that location is too big for a struct sockaddr_un on the platform, resulting in OpenSSH being unable to create or bind to a socket in that location. Instead we try to use the very short and very common /tmp, but fall back to the guessed default if /tmp does not exist. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Automatically use SSH control master support during syncShawn O. Pearce2009-04-181-0/+26
| | | | | | | | | By creating a background ssh "control master" process which lives for the duration of our sync cycle we can easily cut the time for a no-op sync of 132 projects from 60s to 18s. Bug: REPO-11 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Fix `repo --trace` to show ref and config loadsShawn O. Pearce2009-04-181-7/+3
| | | | | | | | | The value of the varible TRACE was copied during the import, which happens before the --trace option can be processed. So instead we now use a function to determine if the value is set, as the function can be safely copied early during import. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Initial Contributionv1.0The Android Open Source Project2008-10-211-0/+164
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-13 12:34:51 +0000