summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
...
| * | Always print percentage when syncing quietlyTim Schumacher2017-07-151-1/+2
| | | | | | | | | | | | Change-Id: I574396e63520781067ed1e991c41caf7640e5731
* | | Fix "list comprehension redefines 'x'" warnings from pyflakesDavid Pursehouse2017-07-101-2/+2
|/ / | | | | | | | | | | | | | | | | $ git ls-files | grep py$ | xargs pyflakes subcmds/stage.py:101: list comprehension redefines 'p' from line 63 subcmds/sync.py:784: list comprehension redefines 'p' from line 664 subcmds/upload.py:467: list comprehension redefines 'avail' from line 454 Change-Id: Ia65d1a72ed185ab3357e1a91ed4450c719e75a7c
* | Merge "Add a newline after "Fetching projects" progress output"David Pursehouse2017-06-281-1/+2
|\ \
| * | Add a newline after "Fetching projects" progress outputTim Schumacher2017-06-131-1/+2
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Output before change: Fetching project platform/packages/providers/UserDictionaryProvider Fetching projects: 66% (773/1171) Fetching project platform/external/regex-re2 Fetching project device/generic/mini-emulator-x86_64 Output after change: Fetching project platform/packages/providers/UserDictionaryProvider Fetching projects: 66% (773/1171) Fetching project platform/external/regex-re2 Fetching project device/generic/mini-emulator-x86_64 Change-Id: I4da84da58316c69294e4da2792f83885dc942701
* / sync: Continue job if some fetchs failed but force-broken is setNicolas Cornu2017-06-161-1/+1
|/ | | | | | | With --force-broken it continue to fetch other projects but nothing is added in directory because it abort some lines later. Change-Id: I32c4a4619b3028893dc4f98e8d4e5bc5c09adb27
* sync: Add support to dump a JSON event log of all sync events.David Riley2017-05-291-7/+24
| | | | Change-Id: Id4852968ac1b2bf0093007cf2e5ca951ddab8b3b
* init: add --submodules to sync manifest submodulesMartin Kelly2017-05-231-2/+3
| | | | | | | | | | | | | | | | repo sync can sync submodules via the --fetch-submodules option. However, if the manifest repo has submodules, those will not be synced. Having submodules in the manifest repo -- while not commonly done -- can be useful for inheriting a manifest from another project using <include> and layering changes on top of it. In this way, you can avoid having to deal with merge conflicts between your own manifests and the other project's manifests (for example, if you're managing an Android fork). Add a --submodule option to init that automatically syncs the submodules in the manifest repo whenever the manifest repo changes. Change-Id: I45d34f04517774c1462d7f233f482d1d81a332a8 Signed-off-by: Martin Kelly <mkelly@xevo.com>
* sync.py: report the remote URL on fatal git remote errorsMarc Herbert2017-04-041-1/+3
| | | | | | | | | | | | | | | | | repo can be configured to download from any number of remote git repos. However when one fails repo doesn't report which one. Example: Fatal: remote error: Daily ls-remote rate limit exceeded for IP xx.xx.xx.xx TEST=repo init -q -u https://chromium.googlesource.com/chromiumos/manifest.git # Apply patch in ./.repo/repo/ # Simulate a git remote error: sed -i -e 's#chromiumos/docs#chromiumos/XXdocs#' .repo/manifests/full.xml repo sync --quiet --force-sync docs # error message now shows the remote URL Optional test tip: reduce the time.sleep(random(...)) in ./.repo/repo/project.py Change-Id: I4509383b6a43a8e66064778e8ed612d8a735c8b6
* sync: Fix semaphore release bug that causes thread 'leaks'Andrew Wheeler2016-10-111-10/+12
| | | | | | | | | | | | | | | | | | | When repo syncs a manifest that utilizes multiple branches in the same project, then the sync will use an extra thread for each "duplicate". For example, if the manifest includes the project "foo" and "bar" twice, then "repo sync -jN" will fetch with N+2 threads. This is caused by _FetchHelper() releasing the thread semaphore object each time it's called, even though _FetchProjectList() may call this function multiple times within the scope of a single thread. Fix by moving the thread semaphore release to _FetchProjectList(), which is only called once per thread instance. Change-Id: I1da78b145e09524d40457db5ca5c37d315432bd8
* Support broken symlinks when cleaning obsolete pathsv1.12.37Dan Willemsen2016-09-271-1/+7
| | | | | | | | | | | When there's a symlink to a directory, os.walk still lists the symlink in dirs, even if it isn't configured to follow symlinks. This will fail the listdirs check if the symlink is broken (either before or during the cleanup). So instead, check for directory symlinks and remove them using os.remove. Bug: Issue 231 Change-Id: I0ec45a26be566613a4a39bf694a3d9c6328481c2
* On project cleanup, don't remove nested projectsDan Willemsen2016-09-201-14/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | When there are nested projects in a manifest, like on AOSP right now: <project path="build" name="platform/build" /> <project path="build/blueprint" name="platform/build/blueprint" /> <project path="build/kati" name="platform/build/kati" /> <project path="build/soong" name="platform/build/soong" /> And the top "build" project is removed (or renamed to remove the nesting), repo just wipes away everything under build/ and re-creates the projects that are still there. But it only checks to see if the build/ project is dirty, so if there are dirty files in a nested project, they'll just be blown away, and a fresh worktree checked out. Instead, behave similarly to how `git clean -dxf` behaves and preserve any subdirectories that have git repositories in them. This isn't as strict as git -- it does not check to see if the '.git' entry is a readable gitdir, just whether an entry named '.git' exists. If it encounters any errors removing files, we'll print them all out to stderr and tell the user that we were unable to clean up the obsolete project, that they should clean it up manually, then sync again. Change-Id: I2f6a7dd205a8e0b7590ca5369e9b0ba21d5a6f77
* When syncing a project with a shared object store, disable automatic pruning.Gabe Black2016-09-141-4/+7
| | | | | | | | | | | | | | | | | The shared object stores confuse git and make it throw away objects which are still in use. We'll avoid that problem by disabling automatic pruning on those projects, but there's nothing preventing a user from changing the config back or pruning a repository manually. BUG=chromium:375945 TEST=Ran repo sync on fresh ChromeOS checkout, starting with a branch of repo with this change. Verified that the kernel projects and no others were identified as having shared object stores, and that repo successfully disabled automatic pruning in their configs. Re-enabled pruning and ran repo sync just on one of the kernel directories. Verified that pruning was re-disabled as a result. Change-Id: I728ed5b06f0087aeb5a23ba8f5410a7cd10af5b0
* sync: Update help text for --smart-sync to be more specificDavid Pursehouse2016-04-131-1/+1
| | | | | | | The --smart-sync option should return the manifest for *the latest* known good build. Change-Id: I2f3216b5b9e1af2ea5f9c3bf1c025813a3b77581
* Sync: Fix error exit code when both -n and -f are usedHu Xiuyun2015-11-261-1/+2
| | | | | | | | | | | | | When repo sync is used with -f (--force-error) and a project fails to sync, the sync will continue but then exit with an error status. However if -n (--network-only) is also used, the exit code is 0, even when a project failed. Modify the logic to make sure the sync exits with the correct status. Bug: Issue 214 Change-Id: I0b5d97a34642c5aa3743750ef14a42c9d5743c1d
* Sync: Add option to prune refs during syncDavid Pursehouse2015-10-271-1/+7
| | | | | | | | By passing --prune to the sync command, the --prune option is given to the `git fetch`, causing refs that no longer exist on the remote to be removed. Change-Id: I3cedacce14276d96ac2d5aabf2d07fd05e92bc02
* Sync: Add HTTP Cookie File header on temporary cookie fileDavid Pursehouse2015-10-021-0/+1
| | | | | | | | | | | | | | | | | | | | The .gitcookies file generated by googlesource.com does not have the header: # (Netscape) HTTP Cookie File which causes python's MozillaCookieJar.load to fail with the error: "does not look like a Netscape format cookies file" Prepend the expected header onto the generated cookie file. We don't bother to check if the header already exists on the file; repeating it does not cause any problem. Bug: Issue 207 Change-Id: I7d39720a1d36a6aae00f70691156514ebc04e579
* Sync: Don't fail when git cookies can't be loadedDavid Pursehouse2015-10-021-1/+4
| | | | | | | | If the git cookies file fails to load, use a default cookie jar instead. Bug: Issue 207 Change-Id: I7cb326c204f2784ab4dbd13801b3186667af5b78
* GITC: Always update the gitc manifest from the repo manifestv1.12.31Dan Willemsen2015-09-091-12/+20
| | | | | | | | | | | | | | This way any changes made to the main manifest are reflected in the gitc manifest. It's also necessary to use both manifests to sync since the information required to update the gitc manifest is actually in the repo manifest. This also fixes a few issues that came up when testing. notdefault groups weren't being saved to the gitc manifest in a method that matched 'sync'. The merge branch wasn't always being set to the correct value either. Change-Id: I435235cb5622a048ffad0059affd32ecf71f1f5b
* Revert "GITC: Always update the gitc manifest from the repo manifest"Dan Willemsen2015-09-091-15/+11
| | | | | | This reverts commit 250303b437855c2b50d052a05a08ed517423af8b. Change-Id: I1fd8af20f802553151aacb953c913f3305ca6057
* GITC: Always update the gitc manifest from the repo manifestDan Willemsen2015-09-091-11/+15
| | | | | | | | | | | | | | This way any changes made to the main manifest are reflected in the gitc manifest. It's also necessary to use both manifests to sync since the information required to update the gitc manifest is actually in the repo manifest. This also fixes a few issues that came up when testing. notdefault groups weren't being saved to the gitc manifest in a method that matched 'sync'. The merge branch wasn't always being set to the correct value either. Change-Id: I5dbc850dd73a9fbd10ab2470ae4c40e46ff894de
* GITC: Fix repo sync.v1.12.29Simran Basi2015-09-031-1/+1
| | | | | | | Fixing http://b/23785024 by calling os.getcwd() because variable cwd no longer exists. Change-Id: I21ff7d059e072f9f60726db76b67587a92c878ad
* GITC: Pull GITC Manifest Dir from the config.Simran Basi2015-08-311-13/+0
| | | | | | | Updates the repo launcher and gitc_utils to pull the manifest directory location out of the gitc config file. Change-Id: Id08381b8a7d61962093d5cddcb3ff6afbb13004b
* GITC: Add repo start support.Simran Basi2015-08-281-15/+29
| | | | | | | | | | | | | | | | | | | | | Add repo start support for GITC checkouts. If the user is in the GITC FS view, they can now run repo start to check out the sources and create a new working branch. When "repo start" is called on a GITC project, the revision tag is set to an empty string and saved in a new tag: old-revision. This tells the GITC filesystem to display the local copy of the sources when being viewed. The local copy is created by pulling the project sources and the new branch is created based off the original project revision. Updated main.py to setup each command's gitc_manifest when appropriate. Updated repo sync's logic to sync opened projects and updating the GITC manifest file for the rest. Change-Id: I7e4809d1c4fc43c69b26f2f1bebe45aab0cae628
* Smartsync: Don't fail if there isn't a cookiefilev1.12.27Dan Willemsen2015-08-201-13/+16
| | | | Change-Id: I434a259f43ca9808e88051ac8ba865c519a24702
* Sync: Refactor netrc parsingDavid Pursehouse2015-08-201-10/+9
| | | | | | | | | | | | | Don't emit a message when the netrc file doesn't exist or couldn't be opened. Instead of trying to unpack the result of info.authenticators() and catching the resulting TypeError when it's None, first store it to a local and only unpack it if it has a value. Also remove an unused import. Change-Id: I5c404d91e48c261c1ab850c3e5f040c4f4c235cb
* Merge "Support smart-sync through persistent-http[s]"Dan Willemsen2015-08-191-1/+104
|\
| * Support smart-sync through persistent-http[s]Dan Willemsen2015-08-191-1/+104
| | | | | | | | | | | | | | Use the same cookies and proxy that git traffic goes through for persistent-http[s] to support authentication for smart-sync. Change-Id: I20f4a281c259053a5a4fdbc48b1bca48e781c692
* | GITC: Add repo sync support.Simran Basi2015-08-181-0/+29
|/ | | | | | | | | | | | | | Add repo sync support for GITC checkouts. If the user is in the GITC client directory they can still pull the sources as normal if they pass in the --force-gitc argument. Otherwise the user should call repo sync in the GITC view to update the user's remote view. (This works because .repo in the GITC view will link to .repo in the client config directory.) Part of the support for this change is the refactoring of GITC related code into gitc_utils.py. Change-Id: I2636aaa50b450b6f091309db8dd0e8f4dbdad579
* Merge "Emit project info in case of sync exception."Conley Owens2015-07-311-1/+3
|\
| * Emit project info in case of sync exception.Dan Sandler2015-07-311-1/+3
| | | | | | | | | | | | | | | | Previously repo would only print the failing project path if Sync_NetworkHalf returned false/empty, but if it threw an exception the print() was never called. Change-Id: I58c41de43930df5e34b21561c205e062a72e290f
* | Add option to correct gitdir when syncingv1.12.25Kevin Degi2015-07-291-1/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | In some cases, a user may wish to continue with a sync even though it would require overwriting an existing git directory. This behavior is not safe as a default because it could result in the loss of some user data, but as an optional flag it allows the user more flexibility. To support this, add a --force-sync flag to the sync command that will attempt to overwrite the existing git dir if it is specified and the existing git dir points to the wrong obj dir. Change-Id: Ieddda8ad54e264a1eb4a9d54881dd6ebc8a03833
* | sync: Remove smart sync override manifest when not in smart sync modeDavid Pursehouse2015-07-061-5/+13
| | | | | | | | | | | | | | | | | | | | | | When syncing with the -s or -t option, a smart_sync_override.xml file is created. This file is left in the file system when syncing again without the -s or -t option. Remove the smart sync override manifest, if it exists, when not using the -s or -t option. Change-Id: I697a0f6405205ba5f84a4d470becf7cd23c07b4b
* | sync: Improve error message when writing smart sync manifest failsDavid Pursehouse2015-07-061-2/+3
| | | | | | | | | | | | | | | | | | The error message only states that writing the manifest failed. Include the exception message, so it's easier to track down the reason that the write failed. Change-Id: I06e942c48a19521ba45292199519dd0a8bdb1de7
* | Add option on sync to avoid fetching from remotes for existing sha1David Pursehouse2015-04-301-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In 2fb6466f795eb30c1dfa598501f5b5d2981e6a5f an optimisation was added to avoid fetching from remotes if the project is fixed to a revision and the revision is already available locally. This causes problems for users who expect all objects to be fetched by default. Change the logic so that the optimized behaviour is only enabled if an option is explicitly given to repo sync. Change-Id: I3b2794ddd8e0071b1787e166463cd8347ca9e24f
* | Add missing documentation of --current-branch option on sync commandDavid Pursehouse2015-01-291-0/+3
| | | | | | | | Change-Id: I72d6e3d51241148c1df97bbad26338debb1fcb4e
* | Allow selection of a target when using smart sync.Jeff Davidson2014-10-021-1/+4
| | | | | | | | Change-Id: I02a24471b9b62dbba3773f22a289825bc566acd9
* | Use JSON instead of pickleAnthony King2014-05-071-26/+21
|/ | | | | | | Use JSON as it is shown to be much faster than pickle. Also clean up the loading and saving functions. Change-Id: I45b3dee7b4d59a1c0e0d38d4a83b543ac5839390
* Clean up duplicate logic in subcmds/sync.py.David James2014-02-141-49/+34
| | | | | | | The fetch logic is now shared between the jobs == 1 and jobs > 1 cases. This refactoring also fixes a bug where opts.force_broken was not honored when jobs > 1. Change-Id: Ic886f3c3c00f3d8fc73a65366328fed3c44dc3be
* Add wrapper moduleConley Owens2014-01-301-2/+2
| | | | | | | | This takes the wrapper importing code from main.py and moves it into its own module so that other modules may import it without causing circular imports with main.py. Change-Id: I9402950573933ed6f14ce0bfb600f74f32727705
* Properly iterate through valuesv1.12.9Conley Owens2014-01-291-1/+1
| | | | | | | | | | | | | | | | | | | the value of Manifest.projects has changed from being the dictionary to the values of the dictionary. Here we handle this change correctly on a PostRepoUpgrade. From a `git diff v1.12.7 -- manifest_xml.py`: + @property def projects(self): self._Load() - return self._projects + return self._paths.values() self._paths does contain the projects according to this line of manifest_xml.py: 484 self._paths[project.relpath] = project Change-Id: I141f8d5468ee10dfb08f99ba434004a307fed810
* Merge "repo: Support multiple branches for the same project."Conley Owens2014-01-101-12/+42
|\
| * repo: Support multiple branches for the same project.David James2013-10-141-12/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is often useful to be able to include the same project more than once, but with different branches and placed in different paths in the workspace. Add this feature. This CL adds the concept of an object directory. The object directory stores objects that can be shared amongst several working trees. For newly synced repositories, we set up the git repo now to share its objects with an object repo. Each worktree for a given repo shares objects, but has an independent set of references and branches. This ensures that repo only has to update the objects once; however the references for each worktree are updated separately. Storing the references separately is needed to ensure that commits to a branch on one worktree will not change the HEAD commits of the others. One nice side effect of sharing objects between different worktrees is that you can easily cherry-pick changes between the two worktrees without needing to fetch them. Bug: Issue 141 Change-Id: I5e2f4e1a7abb56f9d3f310fa6fd0c17019330ecd
* | Add --archive option to init to sync using git archiveJulien Campergue2013-12-101-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This significantly reduces sync time and used brandwidth as only a tar of each project's revision is checked out, but git is not accessible from projects anymore. This is relevant when git is not needed in projects but sync speed/brandwidth may be important like on CI servers when building several versions from scratch regularly for example. Archive is not supported over http/https. Change-Id: I48c3c7de2cd5a1faec33e295fcdafbc7807d0e4d Signed-off-by: Julien Campergue <julien.campergue@parrot.com>
* | Fix some python3 encoding issuesChirayu Desai2013-11-211-1/+1
|/ | | | | | * Add .decode('utf-8') where needed * Add 'b' to `open` where needed, and remove where unnecessary Change-Id: I0f03ecf9ed1a78e3b2f15f9469deb9aaab698657
* Sync: Improved error message when manifest server RPC call failsDavid Pursehouse2013-09-251-1/+2
| | | | | | | | | | | When the RPC call fails, the error message returned by the server is printed, but it is not obvious that this is caused by RPC call failure. Prefix the error message with a descriptive message that explains what went wrong. Change-Id: I4b77af22aacc2e9843c4df9d06bf54e41d9692ff
* Sync: Print name of manifest server used for smart sync/smart tagDavid Pursehouse2013-09-251-0/+2
| | | | | | | | | | When syncing using smart sync or smart tag mode, print the url of the manifest server that is being used. This is useful in organisations that have multiple manifest servers used in different manifest branches. Change-Id: Ib5bc2de5af6f4a942d0ef735c65cbc0721059a61
* Merge "sync: assign manifest_name earlier"David Pursehouse2013-06-111-1/+2
|\
| * sync: assign manifest_name earlierChirayu Desai2013-06-111-1/+2
| | | | | | | | | | | | | | | | | | | | | | * manifest_name was never set if opt.smart_sync or opt.smart_tag is used. * Set it earlier, so that the code handles it correctly when it is None. * An UnboundLocalError is raised if running `repo sync` without any options: local variable 'manifest_name' referenced before assignment * This fixes the above regression caused by commit 53a6c5d93a8ba708208826dab64c55fe97f06d0b Change-Id: I57086670f3589beea8461ce0344f6ec47ab85b7b
* | Fix urllib.parse (urlparse) handlingChirayu Desai2013-06-111-2/+2
|/ | | | | | | | | | | | | | | Revert "Fix "'module' object is not callable" error", and fix it properly. * The urlparse module is renamed to urllib.parse in Python 3. * This commit fixes the code to use "urllib.parse.urlparse" instead of creating a new module urlib and setting urlib.parse to urlparse.urlparse. * Fixes an AttributeError: 'function' object has no attribute 'uses_relative' This reverts commit cd51f17c643370e6199216462c1be36f04d57291. Change-Id: I48490b20ecd19cf5a6edd835506ea5a467d556ac
* Merge "Fix "'module' object is not callable" error"Conley Owens2013-06-101-1/+1
|\
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-13 09:09:49 +0000