summaryrefslogtreecommitdiffstats
path: root/subcmds/forall.py
Commit message (Collapse)AuthorAgeFilesLines
* Expose upstream and dest-branch attributes through environmentSean McAllister2020-04-161-0/+4
| | | | | | | | | | | | | | | | | | | | | | Recent changes in ChromeOS Infra to ensure we're reading from snapshot manifests properly have exposed several bugs in our assumptions about manifest files. Mainly that the revision field for a project does _not_ have to refer to a ref, it can just be a commit hash. Several places assume that the revision field can be parsed as a ref to get the branch the project is on, which isn't true. To fix this we need to be able to look at the upstream and dest-branch attributes of the repo, so we expose them through the environment variables set in `repo forall`. Test: manual 'repo forall' run Bug: https://crbug.com/1032441 Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Sean McAllister <smcallis@google.com>
* forall.py: Close file after removing the streamKarsten Pfeiffer-Raddatz2020-04-141-1/+1
| | | | | | | | | | | | In order to remove the stream fileno() will be called on the filedescriptor. If the file is already closed fileno() will raise an error and forall will fail. Bug: https://crbug.com/gerrit/12563 Change-Id: Ib7b895fe881c844e3eb3672b011fdcdbdae63024 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/262838 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Karsten Pfeiffer-Raddatz <raddatz.karsten@gmail.com>
* assume environment always accepts stringsMike Frysinger2020-02-191-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Different Python & OS versions have different environ behavior wrt accepted types & encoding. Since we're migrating to be Python 3 only, lets change our code to assume strings always work as that's what the newer Python 3 does. This will fail under Python 2 for some env vars, mostly on Windows, but the effort of maintaining shim layers that can handle these edge cases isn't worth it when we're dropping that code. We leave the logic in the `repo` launcher for now as it is simple, and we want it to be able to switch versions a bit longer than the rest of the tree. Here's the support table: | *NIX | Windows | Python 2 | ASCII string | str or bytes, not unicode | Python 3 | str or bytes | str only | Windows uses strings natively in its environment all the time. But it doesn't allow unicode strings under Python 2, so we have to encode. Python 2 on *NIX is funky in that it always lowers to ASCII, so we had to manually encode to avoid errors regardless of unicode or str. Python 3 on Windows & *NIX will accept strings. *NIX will also accept bytes but Windows will not. Bug: https://crbug.com/gerrit/12145 Change-Id: I3cf8f95a06902754ea1f08ad4b28503f7063531b Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/248972 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: Michael Mortensen <mmortensen@google.com> Reviewed-by: Mike Frysinger <vapier@google.com>
* pyflakes: Fix remaining "E501 line too long" warningsDavid Pursehouse2020-02-131-1/+2
| | | | | | | | | | | We increased the max line length to 100 columns which got rid of the majority of these warnings, but there were still a few lines that exceeded 100 columns. Change-Id: Ib3372868ca2297f83073a14f91c8ae3df9d0d0e6 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254699 Tested-by: David Pursehouse <dpursehouse@collab.net> Reviewed-by: Mike Frysinger <vapier@google.com>
* Fix blank line issues reported by flake8David Pursehouse2020-02-121-0/+3
| | | | | | | | | | | | | | | | | | | | - E301 expected 1 blank line - E302 expected 2 blank lines - E303 too many blank lines - E305 expected 2 blank lines after class or function definition - E306 expected 1 blank line before a nested definition Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E301,E302,E303,E305,E306 Manually fix issues in project.py caused by misuse of block comments. Change-Id: Iee840fcaff48aae504ddac9c3e76d2acd484f6a9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254599 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix indentation issues reported by flake8David Pursehouse2020-02-121-20/+20
| | | | | | | | | | | | | | | | | | | | | - E121 continuation line under-indented for hanging indent - E122 continuation line missing indentation or outdented - E125 continuation line with same indent as next logical line - E126 continuation line over-indented for hanging indent - E127 continuation line over-indented for visual indent - E128 continuation line under-indented for visual indent - E129 visually indented line with same indent as next logical line - E131 continuation line unaligned for hanging indent Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E121,E122,E125,E126,E127,E128,E129,E131 Change-Id: Ifd95fb8e6a1a4d6e9de187b5787d64a6326dd249 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254605 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fix various whitespace issues reported by pyflakesDavid Pursehouse2020-02-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | - E201 whitespace after '[' - E202 whitespace before '}' - E221 multiple spaces before operator - E222 multiple spaces after operator - E225 missing whitespace around operator - E226 missing whitespace around arithmetic operator - E231 missing whitespace after ',' - E261 at least two spaces before inline comment - E271 multiple spaces after keyword Fixed automatically with autopep8: git ls-files | grep py$ | xargs autopep8 --in-place \ --select E201,E202,E221,E222,E225,E226,E231,E261,E271 Change-Id: I367113eb8c847eb460532c7c2f8643f33040308c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/254601 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: David Pursehouse <dpursehouse@collab.net>
* Fixing forall subcommand for Py3v2.0Jiri Tyr2020-02-101-1/+1
| | | | | | | | | | | | | | | | Execution of 'repo forall -p -c' doesn't work with Py3 and ends up with an error: Got an error, terminating the pool: TypeError: can only concatenate str (not "bytes") to str That's fixed by using the decode() method. Change-Id: Ice01aaa1822dde8d957b5bf096021dd5a2b7dd51 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/253659 Reviewed-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Jiri Tyr <jiri.tyr@gmail.com>
* forall: add an --ignore-missing optionv1.13.7.1Mike Frysinger2019-10-221-0/+7
| | | | | | | | | | | | | | | In CrOS, our infra has to deal with partial checkouts constantly (for a variety of reasons). To help reset back to a good state, we run git commands via `repo forall`, but don't care about the missing checkouts. Add a flag so we can disambiguate between missing repos and failing git subcommands. Bug: https://crbug.com/1013377 Bug: https://crbug.com/1013623 Change-Id: Ie3498c6d111276c60d2ecedbba21bfa778588d50 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/241935 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* forall: exit 1 if we skip any reposMike Frysinger2019-10-011-3/+3
| | | | | | | | | | If a repo doesn't exist (e.g. an incomplete sync), make sure we exit non-zero when they get skipped. Change-Id: Ifff711e374416b1e6b9b8da4fdc6f14b27ced450 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/239235 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* split out cli validation from executionv1.13.5Mike Frysinger2019-08-281-1/+2
| | | | | | | | | | | | | | | | | | | A common pattern in our subcommands is to verify the arguments & options before executing things. For some subcommands, that check stage is quite long which makes the execution function even bigger. Lets split that logic out of the execute phase so it's easier to manage these. This is most noticeable in the sync subcommand whose Execute func is quite large, and the option checking makes up ~15% of it. The manifest command's Execute can be simplified significantly as the optparse configuration always sets output_file to a string. Change-Id: I7097847ff040e831345e63de6b467ee17609990e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* Remove double forall from "repo help forall" outputSolomon Kinard2019-07-081-1/+1
| | | | | | | | | | | | %prog represents the full subcommand ("repo" + subcommand name), not a Windows-style environment variable for "repo". The current help output shows repo forall% forall ... Correct the variable usage so it shows "repo forall ..." instead. Change-Id: I1fea55572428cc922ddf24ace1168a3d8f82dad0
* set default file encoding to utf-8Mike Frysinger2019-06-131-0/+1
| | | | | | | | There's no reason to support any other encoding in these files. This only affects the files themselves and not streams they open. Bug: https://crbug.com/gerrit/10418 Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
* update markdown/help header formatMike Frysinger2018-10-101-4/+2
| | | | | | | Since gitiles recommends using # headers over ---/=== underlines, change the manifest-format.md over and all our help texts. Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
* Remove unused pylint suppressionsDavid Pursehouse2018-07-241-2/+0
| | | | | | | | | | | pylint is not used since bb5b1a0. The pyflakes cleanup mentioned in that commit has not been done, but given that this project is no longer being actively developed I don't think it's worth spending time doing it. Leaving the pylint suppressions causes confusion because it leads people to think that we are still using pylint. Change-Id: If7d9f280a0f408c780f15915ffdb80579ae21f69
* forall: Clarify expansion of REPO_ environment values with -cDavid Pursehouse2017-10-311-0/+7
| | | | | | | | | | | | | | | | | | | | | | | If a user executes: repo forall -c echo $REPO_PROJECT then $REPO_NAME is expanded by the user's shell first, and passed as $1 to the shell that executes echo. This will either result in no output, or output of whatever REPO_NAME is set to in the user's shell. Either way, this is an unexpected result. The correct way to do it is: repo forall -c 'echo $REPO_PROJECT' such that $REPO_NAME is passed in to the shell literally, and then expanded to the value set in the environment that was passed to the shell. Update the documentation to make this clearer. Change-Id: I713caee914172ad8d8f0fafacd27026502436f0d
* Make "git command" and "forall" work on WindowsRenaud Paquay2017-05-291-20/+9
| | | | | | | | | | Python on Windows does not support non blocking file operations. To workaround this issue, we instead use Threads and a Queue to simulate non-blocking calls. This is happens only when running with the native Windows version of Python, meaning Linux and Cygwin are not affected by this change. Change-Id: I4ce23827b096c5138f67a85c721f58a12279bb6f
* Add --inverse-regex option to forall subcommandTakeshi Kanemoto2016-04-051-3/+8
| | | | | | | | | | Make it possible to exclude projects using regex/wildcard. The syntax is similar to that of the -r option, e.g.: repo forall -i ^platform/ ^device/ -c 'echo $REPO_PROJECT' Change-Id: Id250de5665152228c044c79337d3ac15b5696484
* Better error display on forallAlexandre Garnier2016-02-181-2/+4
| | | | | | | It was only displaying 'Project list error: GitError()' without any useful info about the project nor the error Change-Id: Iad66cbaa03cad1053b5ae9ecc90d7772aa42ac13
* Support filtering by group on forall and list subcmdGraham Christensen2015-07-301-1/+4
| | | | | | | | | | | | | | | | Enable operating against groups of repositories. As it stands, it isn't compatible with `-r/--regex`. `repo forall -g groupname -c pwd` will run `pwd` for all projects in groupname. `repo forall -g thisgroup,-butnotthisone -c pwd` will run `pwd` for all projects in `thisgroup` but not `butnotthisone`. `repo list -g groupname -n` will list all the names of repos in `groupname`. Change-Id: Ia75c50ce52541d1c8cea2874b20a4db2e0e54960
* forall: use smart sync override manifest if it existsDavid Pursehouse2015-07-061-0/+7
| | | | | | | | | | | | If a workspace is synced with the -s or -t option, the included projects may be different to those in the original manifest. However, when using the forall command, the list of the projects from the original manifest is used. If the smart sync manifest file exists, use it to override the original manifest. Change-Id: Iaefcbe148d2158ac046f158d98bbd8b5a5378ce7
* forall: Don't try to get lrev of projects in mirror workspaceDavid Pursehouse2015-07-061-1/+5
| | | | | | | | | git rev-parse fails for projects that don't have an explicit revision specified, and don't have a branch of the same name as the default revision. This can be the case in a workspace synced with the smart sync (-s) or smart tag (-t) option. Change-Id: I19bfe9fe7396170379415d85f10f6440dc6ea08f
* forall: setenv, only encode val if encode existsAnthony King2015-06-041-1/+3
| | | | Change-Id: I655e3043d0118c4e929897d3a51e5e013e5758dc
* Catch exceptions in project list generatorColin Cross2015-05-131-4/+19
| | | | | | | | | | | | | | | | If the generator that produces per-project worker arguments raises an exception it triggers python bug http://bugs.python.org/issue8296. Rewrite the generator expression as a generator function, and catch Exceptions and KeyboardInterrupts to end the iteration. Also add a pool worker initializer to disable SIGINT to prevent KeyboardInterrupts inside multiprocessing.Pool in the worker threads causing the same problem. Fixes easy-to-reproduce hangs when hitting ctrl-c during repo forall -c echo Change-Id: Ie4a65b3e1e07a64ed6bb6ff20f3912c4326718ca
* forall: use a generator to map the PoolAnthony King2015-04-081-2/+2
| | | | | | | | Before, a list was generated, which is why there was a massive delay. Using a generator will allow processes to start straight away Change-Id: Ia325b0b340cc328c08c9bcc92a6709bbdaf6a664
* Add --jobs option to forall subcommandTakeshi Kanemoto2014-06-241-113/+177
| | | | | | | | | | | | | Enable '--jobs' ('-j') option in the forall subcommand. For -jn where n > 1, the '-p' option can no longer guarantee the continuity of console output between the project header and the output from the worker process. SIG_INT is sent to all worker processes upon keyboard interrupt (Ctrl+C). Bug: Issue 105 Change-Id: If09afa2ed639d481ede64f28b641dc80d0b89a5c
* Add total count and iteration count to forall environmentMitchel Humpherys2014-03-311-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | For long-running forall commands sometimes it's useful to know which iteration is currently running. Add REPO_I and REPO_COUNT environment variables to reflect the current iteration count as well as the total number of iterations so that the user can build simple status indicators. Example: $ repo forall -c 'echo $REPO_I / $REPO_COUNT; git gc' 1 / 579 Counting objects: 41, done. Delta compression using up to 8 threads. Compressing objects: 100% (19/19), done. Writing objects: 100% (41/41), done. Total 41 (delta 21), reused 41 (delta 21) 2 / 579 Counting objects: 53410, done. Delta compression using up to 8 threads. Compressing objects: 100% (10423/10423), done. Writing objects: 100% (53410/53410), done. Total 53410 (delta 42513), reused 53410 (delta 42513) 3 / 579 ... Change-Id: I9f28b0d8b7debe423eed3b4bc1198b23e40c0c50 Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
* Print project name for -p on mirror clientsJorge Gonzalez2013-06-031-1/+6
| | | | | | | It doesn't make sense to print the relpath, since there's nothing checked out there and the dir shouldn't even exist. Change-Id: Id43631a8e0895929d3a5ad4ca8c2dc9e3d233e70
* Add regex support for subcommand forallZhiguang Li2013-04-291-1/+13
| | | | | | | | Filter the project list based on regex or wildcard matching of strings, then we can handle subset of all projects. Change-Id: Ib6c23aec79e7d981f7b6a5eb0ae93c44effec467 Signed-off-by: Zhiguang Li <muzili@gmail.com>
* Add option to abort on error in forallVictor Boivie2012-11-161-2/+13
| | | | | | | | | | Add a new option (-e, --abort-on-errors) which will cause forall to abort without iterating through remaining projects if a command exits unsuccessfully. Bug: Issue 17 Change-Id: Ibea405e0d98b575ad3bda719d511f6982511c19c Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com>
* Change print statements to work in python3Sarah Owens2012-11-131-1/+2
| | | | | | This is part of a series of changes to introduce Python3 support. Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
* Add pylint configuration and instructionsDavid Pursehouse2012-10-241-2/+2
| | | | | | | | | | pylint configuration file (.pylintrc) is added, and submission instructions are updated to include pylint usage steps. Deprecated pylint suppression (`disable-msg`) is updated in a few modules to make it work properly with the latest version (0.26). Change-Id: I4ec2ef318e23557a374ecdbf40fe12645766830c
* More coding style cleanupDavid Pursehouse2012-10-221-1/+5
| | | | | | | | | | | | | | | | Fixing more issues found with pylint. Some that were supposed to have been fixed in the previous sweep (Ie0db839e) but were missed: C0321: More than one statement on a single line W0622: Redefining built-in 'name' And some more: W0631: Using possibly undefined loop variable 'name' W0223: Method 'name' is abstract in class 'name' but is not overridden W0231: __init__ method from base class 'name' is not called Change-Id: Ie119183708609d6279e973057a385fde864230c3
* Coding style cleanupDavid Pursehouse2012-10-091-5/+2
| | | | | | | | | | | | | | | Fix the following issues reported by pylint: C0321: More than one statement on a single line W0622: Redefining built-in 'name' W0612: Unused variable 'name' W0613: Unused argument 'name' W0102: Dangerous default value 'value' as argument W0105: String statement has no effect Also fixed a few cases of inconsistent indentation. Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
* Add project annotation handling to repoJames W. Mills2012-04-231-0/+7
| | | | | | | | | | | | | | Allow the optional addition of "annotation" nodes nested under projects. Each annotation node must have "name" and "value" attributes. These name/value pairs will be exported into the environment during any forall command, prefixed with "REPO__" In addition, an optional "keep" attribute with case insensitive "true" or "false" values can be included to determine whether the annotation will be exported with 'repo manifest' Change-Id: Icd7540afaae02c958f769ce3d25661aa721a9de8 Signed-off-by: James W. Mills <jameswmills@gmail.com>
* forall: Silently skip missing projectsShawn O. Pearce2011-01-091-0/+6
| | | | | | | | | | If a project is missing locally, it might be OK to skip over it and continue running the same command in other projects. Bug: REPO-43 Change-Id: I64f97eb315f379ab2c51fc53d24ed340b3d09250 Signed-off-by: Shawn O. Pearce <sop@google.com> (cherry picked from commit d4cd69bdef28c5a9287c85c48a18ce621eba689d)
* Encode the environment variables passed to gitShawn O. Pearce2011-01-091-2/+2
| | | | | | | | Windows allows the environment to have unicode values. This will cause Python to fail to execute the command. Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Change project.revision to revisionExpr and revisionIdShawn O. Pearce2009-05-291-4/+2
| | | | | | | | | The revisionExpr field now holds an expression from the manifest, such as "refs/heads/master", while revisionId holds the current commit-ish SHA-1 of the revisionExpr. Currently that is only filled in if the manifest points directly to a SHA-1. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Change -p command to use stdout instead of stderr.Wink Saville2009-04-211-1/+1
|
* Add -p to `repo forall` to improve output formattingShawn O. Pearce2009-04-181-3/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When trying to read log output from many projects at once it can be difficult to make sense of which messages came from where. For many professional developers it is common to want to view the last week's worth of your work, so you can write a weekly summary of your activity for your status report. This is easier with the new -p option: repo forall -pc git log --reverse --since=1.week.ago --author=sop produces a report of all commits written by me in the last week, formatted in a paged output display, with headers inserted in front of each project's output. Where this can be even more useful is with git log's pickaxe, e.g. now we can use: repo forall -pc git log -Sbar v1.0..v1.1 to locate all additions or removals of the symbol 'bar' since v1.0, up to and including v1.1. Before displaying the matching commits in a project, a project header is shown, giving the user some context information for the matching results. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Fix formatting of 'repo help sync'Shawn O. Pearce2009-04-131-0/+1
| | | | | | The formatting for the enviroment variable section was incorrect. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Set forall environment variables to empty string if NoneShawn O. Pearce2009-03-171-7/+12
| | | | | | | | If the value obtained is None we now set the variable to '' instead, in an attempt to make execve() happier about our 3rd argument, the env dictionary. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Allow repo forall -c on a mirror by using GIT_DIR as pwdShawn O. Pearce2009-03-031-4/+12
| | | | | | | We can permit a forall on a mirror, but only if we put the command into the git repository. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Export additional environment variables to repo forall:Jeff Bailey2009-03-021-0/+17
| | | | | | | | | | | | | | | REPO_PATH is the path relative the the root of the client. REPO_REMOTE is the name of the remote system from the manifest. REPO_LREV is the name of the revision from the manifest, but translated to something the local repository knows. REPO_RREV is the name of the revision from the manifest. This allows us to do commands like: repo forall -c 'echo "(cd $REPO_PATH && git checkout `git rev-parse HEAD`)"'
* Initial Contributionv1.0The Android Open Source Project2008-10-211-0/+82
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-13 12:24:36 +0000