summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
* sync: merge project updates with status barMike Frysinger2019-11-121-11/+4
| | | | | | | | | | | | | | The current sync output displays "Fetching project" and "Checking out project" messages and progress bar updates independently leading to a lot of spam. Lets merge these periodic outputs with the status bar to get a little bit tighter output in the normal case. This doesn't solve all our problems, but gets us closer. Bug: https://crbug.com/gerrit/11293 Change-Id: Icd627830af4dd934a9355b7ace754b56dc96cfef Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244934 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* sync: report list of failing git treesVadim Bendebury2019-11-121-2/+11
| | | | | | | | | | | | | | | | | | When repo sync fails because some git trees are not in clean state and as such can not be rebased automatically, it is a pain to figure out which trees are the culprits. With this patch the list of offending trees is printed when repo sync reports checkout errors. TEST=ran 'repo sync' and observed the proper list of directories show up after the final error message Bug: https://crbug.com/gerrit/11293 Change-Id: Icdf1a03e9014ecb184f331f513cc9a2efc7d11ed Signed-off-by: Vadim Bendebury <vbendeb@google.com> Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244053 Reviewed-by: Mike Frysinger <vapier@google.com>
* use open context managers in more placesMike Frysinger2019-11-121-20/+5
| | | | | | | | | | Use open() as a context manager to simplify the close logic and make the code easier to read & understand. This is also more Pythonic. Change-Id: I579d03cca86f99b2c6c6a1f557f6e5704e2515a7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/244734 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* sync: create dedicated manifest project update funcMike Frysinger2019-09-191-24/+28
| | | | | | | | | | | Cut out some more standalone code from Execute to make this func a bit more manageable. The manifest project update is pretty simple and standalone, but still takes up a good chunk of what's left. Change-Id: Idc2442d9def495eccd0a49cda203c44aef16f129 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/236614 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* sync: create dedicated smart sync funcMike Frysinger2019-09-111-96/+100
| | | | | | | | | | | | The smart sync logic takes up about 45% of the overall Execute func and is about 100 lines of code. The only effect it has on the rest of the code is to set the manifest_name variable. Since this func is already quite huge, split the smart sync logic out. Change-Id: Id861849b0011ab47387d74e92c2ac15afcc938ba Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234835 Tested-by: Mike Frysinger <vapier@google.com> Reviewed-by: David Pursehouse <dpursehouse@collab.net>
* sync: fix deprecated command line option -fv1.13.5.1Stefan Müller-Klieser2019-08-301-0/+1
| | | | | | | | | | | | | | | | | In commit d9e5cf0e ("sync: invert --force-broken with --fail-fast") the force-broken option has been deprecated. Accidentally the option has been changed from Boolean to Value. This breaks all users of repo with: main.py: error: -f option requires an argument This is easy to avoid by keeping the type. Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de> Change-Id: Ia8b589cf41ac756d10c61e17ec8d76ba8f7031f9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/235043 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Mike Frysinger <vapier@google.com>
* split out cli validation from executionv1.13.5Mike Frysinger2019-08-281-20/+14
| | | | | | | | | | | | | | | | | | | A common pattern in our subcommands is to verify the arguments & options before executing things. For some subcommands, that check stage is quite long which makes the execution function even bigger. Lets split that logic out of the execute phase so it's easier to manage these. This is most noticeable in the sync subcommand whose Execute func is quite large, and the option checking makes up ~15% of it. The manifest command's Execute can be simplified significantly as the optparse configuration always sets output_file to a string. Change-Id: I7097847ff040e831345e63de6b467ee17609990e Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234834 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* sync: invert --force-broken with --fail-fastMike Frysinger2019-08-271-12/+14
| | | | | | | | | | | | | | | | | People seem to not expect the sync process to halt immediately if an error is encountered. It's also basically guaranteed to leave their tree in an incomplete state. Lets invert the default behavior so we attempt to sync (both fetch & checkout) all projects. If an error is hit, we still exit(1) and show it at the end. If people want the sync to abort quickly, they can use the new option --fail-fast. Bug: https://crbug.com/gerrit/11293 Change-Id: I49dd6c4dc8fd5cce8aa905ee169ff3cbe230eb3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/234812 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* rebase/sync: use exit(1) for errors instead of exit(-1)Mike Frysinger2019-08-081-4/+4
| | | | | | | | | | | Callers don't actually see -1 (they'll usually see 255, but the exact answer here is complicated). Just switch to 1 as that's the standard value tools use to indicate an error. Change-Id: Ib712db1924bc3e5f7920bafd7bb5fb61f3bda44f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233553 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* sync: fix handling of -f and local checkoutsMike Frysinger2019-08-081-3/+1
| | | | | | | | | | | | | The partial clone rework (commit 745be2ede1e67421275afc00c04d996d9d6908ee "Add support for partial clone") changed the behavior when a single repo hit a failure: it would always call sys.exit() immediately. This isn't even necessary as we already pass down an error event object which the workers set and the parent checks. Just delete the exit entirely. Change-Id: Id72d8642aefa2bde24e1a438dbe102c3e3cabf48 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/233552 Reviewed-by: David Pursehouse <dpursehouse@collab.net> Tested-by: Mike Frysinger <vapier@google.com>
* Add support for partial clone.v1.13.4Xin Li2019-07-161-20/+156
| | | | | | | | | | | | | A new option, --partial-clone is added to 'repo init' which tells repo to utilize git's partial clone functionality, which reduces disk and bandwidth usage when downloading by omitting blob downloads initially. Different from restricting clone-depth, the user will have full access to change history, etc., as the objects are downloaded on demand. Change-Id: I60326744875eac16521a007bd7d5481112a98749 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/229532 Reviewed-by: Mike Frysinger <vapier@google.com> Tested-by: Xin Li <delphij@google.com>
* sync: use integer division with job countsMike Frysinger2019-06-141-2/+2
| | | | | | | | Neither of the fields here expect floats so make sure we use integer division when calculating things. Bug: https://crbug.com/gerrit/10418 Change-Id: Ibda068b16a7bba7ff3efba442c4bbff4415caa6e
* set default file encoding to utf-8Mike Frysinger2019-06-131-0/+1
| | | | | | | | There's no reason to support any other encoding in these files. This only affects the files themselves and not streams they open. Bug: https://crbug.com/gerrit/10418 Change-Id: I053cb40cd3666ce5c8a0689b9dd938f24ca765bf
* sync: deleted unused repos in reversed order (children before parent)Kuang-che Wu2019-04-061-1/+2
| | | | | Bug: chromium:950002 Test: repo sync chromeos using release manifest file Change-Id: I613df6a1973eb36acd806a703e72f5172554bcc7
* sync: Add option '--force-remove-dirty'Oleksii Okolielov2019-03-111-3/+17
| | | | | | | | | | Forcefully remove dirty projects if option '--force-remove-dirty' is given. The '--force-remove-dirty' option can be used to remove previously used projects with uncommitted changes. WARNING: This may cause data to be lost since uncommitted changes may be removed with projects that no longer exist in the manifest. Change-Id: I844a6e943ded522fdc7b1b942c0a1269768054bc
* Add support for long pathsRenaud Paquay2018-10-221-13/+13
| | | | | | | | | | | | | | | | | | | | * Add more file i/o wrappers in platform_utils to allow using long paths (length > MAX_PATH) on Windows. * Paths using the long path syntax ("\\?\" prefix) should never escape the platform_utils API surface area, so that this specific syntax is not visible to the rest of the repo code base. * Forward many calls from os.xxx to platform_utils.xxx in various place to ensure long paths support, specifically when repo decides to delete obsolete directories. * There are more places that need to be converted to support long paths, this commit is an initial effort to unblock a few common use cases. * Also, fix remove function to handle directory symlinks Change-Id: If82ccc408e516e96ff7260be25f8fd2fe3f9571a
* update markdown/help header formatMike Frysinger2018-10-101-4/+2
| | | | | | | Since gitiles recommends using # headers over ---/=== underlines, change the manifest-format.md over and all our help texts. Change-Id: I96391d41fba769e9f26870d497cf7cf01c8d8ab3
* Replace all os.remove callsRenaud Paquay2017-08-311-5/+5
| | | | | | | | os.remove raises an exception when deleting read-only files on Windows. Replace all calls with calls to platform_utils.remove, which deals with deals with that issue. Change-Id: I4dc9e0c9a36b4238880520c69f5075eca40f3e66
* Implement islink, readlink and realpath using Win32 apiRenaud Paquay2017-08-311-1/+1
| | | | Change-Id: I18452cbb32d24db73601ad10485dbe6bb278731c
* Merge changes from topic "windows-support"David Pursehouse2017-08-301-2/+2
|\ | | | | | | | | | | | | | | * changes: Port os.rename calls to work on Windows Workaround shutil.rmtree limitation on Windows Add support for creating symbolic links on Windows Make "git command" and "forall" work on Windows
| * Workaround shutil.rmtree limitation on WindowsRenaud Paquay2017-05-291-2/+2
| | | | | | | | | | | | | | | | | | | | By default, shutil.rmtree raises an exception when deleting readonly files on Windows. Replace all shutil.rmtree with platform_utils.rmtree, which adds an error handler to make files read-write when they can't be deleted. Change-Id: I9cfea9a7b3703fb16a82cf69331540c2c179ed53
* | Merge "Always print percentage when syncing quietly"David Pursehouse2017-08-021-1/+2
|\ \
| * | Always print percentage when syncing quietlyTim Schumacher2017-07-151-1/+2
| | | | | | | | | | | | Change-Id: I574396e63520781067ed1e991c41caf7640e5731
* | | Fix "list comprehension redefines 'x'" warnings from pyflakesDavid Pursehouse2017-07-101-2/+2
|/ / | | | | | | | | | | | | | | | | $ git ls-files | grep py$ | xargs pyflakes subcmds/stage.py:101: list comprehension redefines 'p' from line 63 subcmds/sync.py:784: list comprehension redefines 'p' from line 664 subcmds/upload.py:467: list comprehension redefines 'avail' from line 454 Change-Id: Ia65d1a72ed185ab3357e1a91ed4450c719e75a7c
* | Merge "Add a newline after "Fetching projects" progress output"David Pursehouse2017-06-281-1/+2
|\ \
| * | Add a newline after "Fetching projects" progress outputTim Schumacher2017-06-131-1/+2
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Output before change: Fetching project platform/packages/providers/UserDictionaryProvider Fetching projects: 66% (773/1171) Fetching project platform/external/regex-re2 Fetching project device/generic/mini-emulator-x86_64 Output after change: Fetching project platform/packages/providers/UserDictionaryProvider Fetching projects: 66% (773/1171) Fetching project platform/external/regex-re2 Fetching project device/generic/mini-emulator-x86_64 Change-Id: I4da84da58316c69294e4da2792f83885dc942701
* / sync: Continue job if some fetchs failed but force-broken is setNicolas Cornu2017-06-161-1/+1
|/ | | | | | | With --force-broken it continue to fetch other projects but nothing is added in directory because it abort some lines later. Change-Id: I32c4a4619b3028893dc4f98e8d4e5bc5c09adb27
* sync: Add support to dump a JSON event log of all sync events.David Riley2017-05-291-7/+24
| | | | Change-Id: Id4852968ac1b2bf0093007cf2e5ca951ddab8b3b
* init: add --submodules to sync manifest submodulesMartin Kelly2017-05-231-2/+3
| | | | | | | | | | | | | | | | repo sync can sync submodules via the --fetch-submodules option. However, if the manifest repo has submodules, those will not be synced. Having submodules in the manifest repo -- while not commonly done -- can be useful for inheriting a manifest from another project using <include> and layering changes on top of it. In this way, you can avoid having to deal with merge conflicts between your own manifests and the other project's manifests (for example, if you're managing an Android fork). Add a --submodule option to init that automatically syncs the submodules in the manifest repo whenever the manifest repo changes. Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8 Signed-off-by: Martin Kelly <mkelly@xevo.com>
* sync.py: report the remote URL on fatal git remote errorsMarc Herbert2017-04-041-1/+3
| | | | | | | | | | | | | | | | | repo can be configured to download from any number of remote git repos. However when one fails repo doesn't report which one. Example: Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git # Apply patch in ./.repo/repo/ # Simulate a git remote error: sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml repo sync --quiet --force-sync docs # error message now shows the remote URL Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
* sync: Fix semaphore release bug that causes thread 'leaks'Andrew Wheeler2016-10-111-10/+12
| | | | | | | | | | | | | | | | | | | When repo syncs a manifest that utilizes multiple branches in the same project, then the sync will use an extra thread for each "duplicate". For example, if the manifest includes the project "foo" and "bar" twice, then "repo sync -jN" will fetch with N+2 threads. This is caused by _FetchHelper() releasing the thread semaphore object each time it's called, even though _FetchProjectList() may call this function multiple times within the scope of a single thread. Fix by moving the thread semaphore release to _FetchProjectList(), which is only called once per thread instance. Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
* Support broken symlinks when cleaning obsolete pathsv1.12.37Dan Willemsen2016-09-271-1/+7
| | | | | | | | | | | When there's a symlink to a directory, os.walk still lists the symlink in dirs, even if it isn't configured to follow symlinks. This will fail the listdirs check if the symlink is broken (either before or during the cleanup). So instead, check for directory symlinks and remove them using os.remove. Bug: Issue 231 Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
* On project cleanup, don't remove nested projectsDan Willemsen2016-09-201-14/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | When there are nested projects in a manifest, like on AOSP right now: <project path="build" name="platform/build" /> <project path="build/blueprint" name="platform/build/blueprint" /> <project path="build/kati" name="platform/build/kati" /> <project path="build/soong" name="platform/build/soong" /> And the top "build" project is removed (or renamed to remove the nesting), repo just wipes away everything under build/ and re-creates the projects that are still there. But it only checks to see if the build/ project is dirty, so if there are dirty files in a nested project, they'll just be blown away, and a fresh worktree checked out. Instead, behave similarly to how `git clean -dxf` behaves and preserve any subdirectories that have git repositories in them. This isn't as strict as git -- it does not check to see if the '.git' entry is a readable gitdir, just whether an entry named '.git' exists. If it encounters any errors removing files, we'll print them all out to stderr and tell the user that we were unable to clean up the obsolete project, that they should clean it up manually, then sync again. Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
* When syncing a project with a shared object store, disable automatic pruning.Gabe Black2016-09-141-4/+7
| | | | | | | | | | | | | | | | | The shared object stores confuse git and make it throw away objects which are still in use. We'll avoid that problem by disabling automatic pruning on those projects, but there's nothing preventing a user from changing the config back or pruning a repository manually. BUG=chromium:375945 TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo with this change. Verified that the kernel projects and no others were identified as having shared object stores, and that repo successfully disabled automatic pruning in their configs. Re-enabled pruning and ran repo sync just on one of the kernel directories. Verified that pruning was re-disabled as a result. Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
* sync: Update help text for --smart-sync to be more specificDavid Pursehouse2016-04-131-1/+1
| | | | | | | The --smart-sync option should return the manifest for *the latest* known good build. Change-Id: I2f3216b5b9e1af2ea5f9c3bf1c025813a3b77581
* Sync: Fix error exit code when both -n and -f are usedHu Xiuyun2015-11-261-1/+2
| | | | | | | | | | | | | When repo sync is used with -f (--force-error) and a project fails to sync, the sync will continue but then exit with an error status. However if -n (--network-only) is also used, the exit code is 0, even when a project failed. Modify the logic to make sure the sync exits with the correct status. Bug: Issue 214 Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
* Sync: Add option to prune refs during syncDavid Pursehouse2015-10-271-1/+7
| | | | | | | | By passing --prune to the sync command, the --prune option is given to the `git fetch`, causing refs that no longer exist on the remote to be removed. Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
* Sync: Add HTTP Cookie File header on temporary cookie fileDavid Pursehouse2015-10-021-0/+1
| | | | | | | | | | | | | | | | | | | | The .gitcookies file generated by googlesource.com does not have the header: # (Netscape) HTTP Cookie File which causes python's MozillaCookieJar.load to fail with the error: "does not look like a Netscape format cookies file" Prepend the expected header onto the generated cookie file. We don't bother to check if the header already exists on the file; repeating it does not cause any problem. Bug: Issue 207 Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
* Sync: Don't fail when git cookies can't be loadedDavid Pursehouse2015-10-021-1/+4
| | | | | | | | If the git cookies file fails to load, use a default cookie jar instead. Bug: Issue 207 Change-Id: I7cb326c204f2784ab4dbd13801b3186667af5b78
* GITC: Always update the gitc manifest from the repo manifestv1.12.31Dan Willemsen2015-09-091-12/+20
| | | | | | | | | | | | | | This way any changes made to the main manifest are reflected in the gitc manifest. It's also necessary to use both manifests to sync since the information required to update the gitc manifest is actually in the repo manifest. This also fixes a few issues that came up when testing. notdefault groups weren't being saved to the gitc manifest in a method that matched 'sync'. The merge branch wasn't always being set to the correct value either. Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
* Revert "GITC: Always update the gitc manifest from the repo manifest"Dan Willemsen2015-09-091-15/+11
| | | | | | This reverts commit 250303b437855c2b50d052a05a08ed517423af8b. Change-Id: I1fd8af20f802553151aacb953c913f3305ca6057
* GITC: Always update the gitc manifest from the repo manifestDan Willemsen2015-09-091-11/+15
| | | | | | | | | | | | | | This way any changes made to the main manifest are reflected in the gitc manifest. It's also necessary to use both manifests to sync since the information required to update the gitc manifest is actually in the repo manifest. This also fixes a few issues that came up when testing. notdefault groups weren't being saved to the gitc manifest in a method that matched 'sync'. The merge branch wasn't always being set to the correct value either. Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
* GITC: Fix repo sync.v1.12.29Simran Basi2015-09-031-1/+1
| | | | | | | Fixing http://b/23785024 by calling os.getcwd() because variable cwd no longer exists. Change-Id: I21ff7d059e072f9f60726db76b67587a92c878ad
* GITC: Pull GITC Manifest Dir from the config.Simran Basi2015-08-311-13/+0
| | | | | | | Updates the repo launcher and gitc_utils to pull the manifest directory location out of the gitc config file. Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
* GITC: Add repo start support.Simran Basi2015-08-281-15/+29
| | | | | | | | | | | | | | | | | | | | | Add repo start support for GITC checkouts. If the user is in the GITC FS view, they can now run repo start to check out the sources and create a new working branch. When "repo start" is called on a GITC project, the revision tag is set to an empty string and saved in a new tag: old-revision. This tells the GITC filesystem to display the local copy of the sources when being viewed. The local copy is created by pulling the project sources and the new branch is created based off the original project revision. Updated main.py to setup each command's gitc_manifest when appropriate. Updated repo sync's logic to sync opened projects and updating the GITC manifest file for the rest. Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
* Smartsync: Don't fail if there isn't a cookiefilev1.12.27Dan Willemsen2015-08-201-13/+16
| | | | Change-Id: I434a259f43ca9808e88051ac8ba865c519a24702
* Sync: Refactor netrc parsingDavid Pursehouse2015-08-201-10/+9
| | | | | | | | | | | | | Don't emit a message when the netrc file doesn't exist or couldn't be opened. Instead of trying to unpack the result of info.authenticators() and catching the resulting TypeError when it's None, first store it to a local and only unpack it if it has a value. Also remove an unused import. Change-Id: I5c404d91e48c261c1ab850c3e5f040c4f4c235cb
* Merge "Support smart-sync through persistent-http[s]"Dan Willemsen2015-08-191-1/+104
|\
| * Support smart-sync through persistent-http[s]Dan Willemsen2015-08-191-1/+104
| | | | | | | | | | | | | | Use the same cookies and proxy that git traffic goes through for persistent-http[s] to support authentication for smart-sync. Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
* | GITC: Add repo sync support.Simran Basi2015-08-181-0/+29
|/ | | | | | | | | | | | | | Add repo sync support for GITC checkouts. If the user is in the GITC client directory they can still pull the sources as normal if they pass in the --force-gitc argument. Otherwise the user should call repo sync in the GITC view to update the user's remote view. (This works because .repo in the GITC view will link to .repo in the client config directory.) Part of the support for this change is the refactoring of GITC related code into gitc_utils.py. Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-06-04 00:47:39 +0000