summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
...
* | Represent git-submodule as nested projectsChe-Liang Chiou2012-10-231-1/+20
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We need a representation of git-submodule in repo; otherwise repo will not sync submodules, and leave workspace in a broken state. Of course this will not be a problem if all projects are owned by the owner of the manifest file, who may simply choose not to use git-submodule in all projects. However, this is not possible in practice because manifest file owner is unlikely to own all upstream projects. As git submodules are simply git repositories, it is natural to treat them as plain repo projects that live inside a repo project. That is, we could use recursively declared projects to denote the is-submodule relation of git repositories. The behavior of repo remains the same to projects that do not have a sub-project within. As for parent projects, repo fetches them and their sub-projects as normal projects, and then checks out subprojects at the commit specified in parent's commit object. The sub-project is fetched at a path relative to parent project's working directory; so the path specified in manifest file should match that of .gitmodules file. If a submodule is not registered in repo manifest, repo will derive its properties from itself and its parent project, which might not always be correct. In such cases, the subproject is called a derived subproject. To a user, a sub-project is merely a git-submodule; so all tips of working with a git-submodule apply here, too. For example, you should not run `repo sync` in a parent repository if its submodule is dirty. Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
* Rename "dir" variablesMickaël Salaün2012-10-101-4/+4
| | | | | | | The variable name "dir" conflicts with the name of a Python built-in function: http://docs.python.org/library/functions.html#dir Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
* Coding style cleanupDavid Pursehouse2012-10-091-7/+6
| | | | | | | | | | | | | | | Fix the following issues reported by pylint: C0321: More than one statement on a single line W0622: Redefining built-in 'name' W0612: Unused variable 'name' W0613: Unused argument 'name' W0102: Dangerous default value 'value' as argument W0105: String statement has no effect Also fixed a few cases of inconsistent indentation. Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
* Remove unreachable codeDavid Pursehouse2012-10-051-8/+0
| | | | | | | Change 9bb1816b removed part of a block of code, but left the remaining part unreachable. Remove it. Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
* sync: Support manual authentication to the manifest serverDavid Pursehouse2012-09-211-19/+52
| | | | | | | | | | | | | | | | Add two new command line options, -u/--manifest-server-username and -p/--manifest-server-password, which can be used to specify a username and password to authenticate to the manifest server when using the -s/--smart-sync or -t/--smart-tag option. If -u and -p are not specified when using the -s or -t option, use authentication credentials from the .netrc file (if there are any). Authentication credentials from -u/-p or .netrc are not used if the manifest server specified in the manifest file already includes credentials. Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
* sync: Support authentication to manifest server with .netrcDavid Pursehouse2012-09-111-1/+29
| | | | | | | | | | | | | | | | | | | | When using the --smart-sync or --smart-tag option, and the specified manifest server is hosted on a server that requires authentication, repo sync fails with the error: HTTP 401 Unauthorized. Add support for getting the credentials from the .netrc file. If a .netrc file exists in the user's home directory, and it contains credentials for the hostname of the manifest server specified in the manifest, use the credentials to authenticate with the manifest server using the URL syntax extension for Basic Authentication: http://user:password@host:port/path Credentials from the .netrc file are only used if the manifest server URL specified in the manifest does not already include credentials. Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
* sync: Correct imports of `R_HEADS` and `HEAD`David Pursehouse2012-09-071-3/+1
| | | | | | | | | | | | | | `R_HEADS` is imported twice, from both the git_refs and project modules. It is actually defined in git_refs, and in project it is imported from there, so the import of `R_HEADS` from project in the sync module is redundant. Remove it. `HEAD` is imported from project, but like `R_HEADS` it is actually defined in git_refs. Import it from git_refs instead. Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
* sync: catch exceptions when connecting to the manifest serverDavid Pursehouse2012-09-061-3/+7
| | | | | | | | | | | | | | | | | When connecting to the manifest server, exceptions can occur but are not caught, resulting in the repo sync exiting with a python traceback. Add handling of the following exceptions: - IOError, which can be raised for example if the manifest server URL is malformed. - xmlrpclib.ProtocolError, which can be raised if the connection to the manifest server fails with HTTP error. - xmlrpclib.Fault, which can be raised if the RPC call fails for some other reason. Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
* Make "repo sync -j<count>" stop properly on Ctrl-C.David 'Digit' Turner2012-09-051-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | The threaded 'repo sync' implementation would very often freeze the process when interrupted by the user with Ctrl-C. The only solution being to kill -9 the process explicitly from another terminal. The reason for this is best explained here: http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html This patch makes all helper sync threads 'daemon', which allows the process to terminate immediately on Ctrl-C. Note that this will forcefully kill all threads in case of interruption; this is generally a bad thing, but: 1/ This is equivalent to calling kill -9 in another terminal, which is the _only_ thing that can currently stop the process. 2/ There doesn't seem to be a way to tell the worker threads to gently stop when they are in a blocking operation anyway (even in the non-threaded case). + Do the same for "repo status -j<count>". Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
* sync: Honor --no-clone-bundle with -j1Shawn O. Pearce2012-08-021-2/+4
| | | | Change-Id: I7c12902e386121a374d525be673092360c67c53d
* Add manifest groupsv1.8.2Colin Cross2012-04-131-2/+3
| | | | | | | | | | | | | | | | | Allows specifying a list of groups with a -g argument to repo init. The groups act on a group= attribute specified on projects in the manifest. All projects are implicitly labelled with "default" unless they are explicitly labelled "-default". Prefixing a group with "-" removes matching projects from the list of projects to sync. If any non-inverted manifest groups are specified, the default label is ignored. Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5 Reviewed-on: https://gerrit-review.googlesource.com/34570 Reviewed-by: Shawn Pearce <sop@google.com> Tested-by: Shawn Pearce <sop@google.com>
* sync: --no-clone-bundle disables the clone bundle supportv1.8.0Shawn O. Pearce2012-03-141-2/+13
| | | | Change-Id: Ia9ed7da8451b273c1be620c3dd0dcad777b29096
* sync: Add manifest_name parameterChris Wolfe2012-01-261-0/+12
| | | | | | | | | | | | This parameter changes the manifest used by 'repo sync' for only this execution. It should be useful for developers wishing to get the repo temporarily into a known state, without clobbering their existing manifest. Tested by shifting Chrome OS between minilayout and full, and between several release-builder-generated manifests. Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
* Add a sync flag that fetches only current branchAnatol Pomazau2011-11-031-3/+9
| | | | | | | | | | | There is also shortcuts in case if the "current branch" is a persistent revision such as tag or sha1. We check if the persistent revision is present locally and if it does - do no fetch anything from the server. This greately reduces sync time and size of the on-disk repo Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
* Fix Python 2.4 supportShawn O. Pearce2011-10-111-8/+4
| | | | | Change-Id: I89521ae52fa564f0d849cc51e71fee65b3c47bab Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Update default -j flag from manifestShawn O. Pearce2011-09-261-1/+2
| | | | | | | | | If the manifest is updated and the default sync-j attribute was modified, honor it during this sync session if the user has not supplied a -j flag on the command line. Change-Id: I127ee5c779e2bbbb40b30bddc10ec1fa704b3bf3 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Allow -j to have a default in manifestv1.7.6Shawn O. Pearce2011-09-221-1/+4
| | | | | | | | | | | | | This permits manifest authors to suggest a number of parallel fetch operations against a remote server. For example, Gerrit Code Review servers support queuing of requests and processes them in first-in, first-out order. Running concurrent fetches can utilize multiple CPUs on the Gerrit server, but will also decrease overall operation latency by having the request put into the queue ready to execute as soon as a CPU is free. Change-Id: I3d3904acb6f63516bae4b071c510ad57a2afab18 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Limit -j to file descriptorsShawn O. Pearce2011-09-221-0/+12
| | | | | | | | | | | Each worker thread requires at least 3 file descriptors to run the forked 'git fetch' child to operate against the local repository. Mac OS X has the RLIMIT_NOFILE set to 256 by default, which means a sync -j128 often fails when the workers run out of pipes within the Python parent process. Change-Id: I2cdb14621b899424b079daf7969bc8c16b85b903 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Smart tag supportVictor Boivie2011-07-201-15/+23
| | | | | | | | | This is an evolution of 'smart-sync' that adds a new option, -t, that allows you to specify a tag/label to use instead of the "latest good build" on the current manifest branch which -s does. Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com> Change-Id: I8c20fd91104a6aafa0271d4d33f6c4850aade17e
* Fix parallel sync on python < 2.6.Daniel Sandler2011-04-041-2/+2
| | | | | | | | Event.isSet was renamed to is_set in 2.6, but we should use the earlier syntax to avoid breaking compatibility with older Python installations. Change-Id: I41888ed38df278191d7496c1a6eed15e881733f4
* sync: Fix syntax error on Python 2.4v1.7.4.2Shawn O. Pearce2011-03-221-22/+23
| | | | | Change-Id: I371d032d5a1ddde137721cbe2b24bfa38f20aaaa Signed-off-by: Shawn O. Pearce <sop@google.com>
* Make 'repo sync -jN' exit with an error code in the case of sync errors.Doug Anderson2011-03-171-14/+70
| | | | | | | | | | | | The bug that this is fixing is described here: http://code.google.com/p/chromium-os/issues/detail?id=6813 This fix allows the helper threads to signal the main thread that they saw an error. When the main thread sees the error, it will let all existing threads finish, then exit with an error. Change-Id: If3019bc6b0b3ab9304d49ed2eea53e9d57f3095a
* sync: Run `git gc --auto` after fetchShawn O. Pearce2011-01-091-0/+2
| | | | | | | | | | | | | Users may wind up with a lot of loose object content in projects they don't frequently make changes in, but that are modified by others. Since we bypass many git code paths that would have otherwise called out to `git gc --auto`, its possible for these projects to have their loose object database grow out of control. To help prevent that, we now invoke it ourselves during the network half of sync. Signed-off-by: Shawn O. Pearce <sop@google.com> (cherry picked from commit 1875ddd47c0bf38e5cc52e1e5875caabce2d8742)
* Encode the environment variables passed to gitShawn O. Pearce2011-01-091-4/+4
| | | | | | | | Windows allows the environment to have unicode values. This will cause Python to fail to execute the command. Change-Id: I37d922c3d7ced0d5b4883f0220346ac42defc5e9 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Added feature to print a <notice> from manifest at the end of a sync.v1.7.1Doug Anderson2010-11-011-0/+5
| | | | | | | | | | | | | | | | | | | | | | | This feature is used to convey information on a when a branch has ceased development or if it is an experimental branch with a few gotchas, etc. You add it to your manifest XML by doing something like this: <manifest> <notice> NOTE TO DEVELOPERS: If you checkin code, you have to pinky-swear that it contains no bugs. Anyone who breaks their promise will have tomatoes thrown at them in the team meeting. Be sure to bring an extra set of clothes. </notice> <remote ... /> ... </manifest> Carriage returns and indentation are relevant for the text in this tag. This feature was requested by Anush Elangovan on the ChromiumOS team.
* sync: Use --force-broken to continue other projectsv1.7Andrei Warkentin2010-10-291-3/+15
| | | | | | | | | | This adds a new flag -f/--force-broken that will allow the rest of the sync process to continue instead of bailing when a particular project fails to sync. Change-Id: I23680f2ee7927410f7ed930b1d469424c9aa246e Signed-off-by: Andrei Warkentin <andreiw@motorola.com> Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync --quiet: be more quietShawn O. Pearce2010-10-291-8/+16
| | | | | Change-Id: I5e8363c7b32e4546d1236cfc5a32e01c3e5ea8e6 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: support --jobs to fetch projects simultaneouslyRoy Lee2010-05-271-7/+46
| | | | | | | | | | | | | | This patch does two things for being compatibile with those Python which are built without threading support: 1. As the Python document and Shawn suggested, import dummy_threading when the threading is not available. 2. Reserve the single threaded code and make it default. In cases the --jobs does not work properly with dummy_threading, we still have a safe fallback. Change-Id: I40909ef8e9b5c22f315c0a1da9be38eed8b0a2dc
* Aliasing sync -s to 'smartsync'v1.6.9.7Nico Sallembien2010-05-171-4/+5
| | | | | | | This alias will let people use this command without having to remember the option. Change-Id: I3256d9e8e884c5be9e77f70e9cfb73e0f0c544c6
* Override manifest file only after it is fully written to disk.v1.6.9.4Nico Sallembien2010-04-261-1/+1
| | | | | | We called "Override()" before closing the file passed in argument. Change-Id: I15adb99deb14297ef72fcb1b0945eb246f172fb0
* Strip refs/heads in the branch sent to the manifest server.Nico Sallembien2010-04-261-0/+3
| | | | | | | The manifest server doesn't want to have refs/heads passed to it, so we need to strip that when the branch contains it. Change-Id: I044f8a9629220e886fd5e02e3c1ac4b4bb6020ba
* sync: Safely skip already deleted projectsv1.6.9.3Anthony2010-04-141-25/+29
| | | | | | | Do not error if a project is missing on the filesystem, is deleted from manifest.xml, but still exists in project.list. Change-Id: I1d13e435473c83091e27e4df571504ef493282dd
* Add a 'smart sync' option to repo syncv1.6.9.2Nico Sallembien2010-04-131-0/+52
| | | | | | | | | | | | | This option allows the user to specify a manifest server to use when syncing. This manifest server will provide a manifest pegging each project to a known green build. This allows developers to work on a known good tree that is known to build and pass tests, preventing failed builds to hamper productivity. The manifest used is not "sticky" so as to allow subsequent 'repo sync' calls to sync to the tip of the tree. Change-Id: Id0a24ece20f5a88034ad364b416a1dd2e394226d
* Fixing project renaming bug.v1.6.8.8Nico Sallembien2009-12-101-9/+9
| | | | | | | | This bug happens when a project gets added to the manifest, and then is renamed. Users who happened to have run "repo sync" after the project was added but before the rename happened will try to read the data from the old project, as the manifest was only updated after all projects were updated successfully.
* sync: Keep the project.list file sortedShawn O. Pearce2009-06-041-0/+1
| | | | | | Its easier to locate an entry visually if the file is sorted. Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Tolerate blank lines in project.listv1.6.8.1Shawn O. Pearce2009-06-041-1/+5
| | | | | | | | If a line is blank in project.list, its not a relevant project path, so skip over it. Existing project.list files may have blank lines if sync was run with no projects at all, and the file was created empty. Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Don't process project.list in a mirrorShawn O. Pearce2009-06-041-0/+4
| | | | | | | | | We have no working tree, so we cannot update the project.list state file, nor should we try to delete a directory if a project is removed from the manifest. Clients would still need the repository for historical records. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Delete empty parent subdirs after deleting obsolete paths.v1.6.8Jaikumar Ganesh2009-06-021-0/+8
| | | | | | After sync, we delete obsolete project paths. Iterate and delete parent subdirs which are empty. Tested on projects within subdirectories.
* Update project paths after sync.Jaikumar Ganesh2009-06-021-0/+48
| | | | | | After a repo sync, some of the project paths might need to be removed. This changes maintains a list of project paths from the previous sync operation and compares.
* Change project.revision to revisionExpr and revisionIdShawn O. Pearce2009-05-291-5/+2
| | | | | | | | | The revisionExpr field now holds an expression from the manifest, such as "refs/heads/master", while revisionId holds the current commit-ish SHA-1 of the revisionExpr. Currently that is only filled in if the manifest points directly to a SHA-1. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Document the SSH ControlMaster behavior of repo syncShawn O. Pearce2009-04-211-0/+29
| | | | Signed-off-by: Shawn O. Pearce <sop@google.com>
* Clarify options that control the repo executable versionShawn O. Pearce2009-04-181-2/+3
| | | | Signed-off-by: Shawn O. Pearce <sop@google.com>
* Only fetch repo once-per-day under normal 'repo sync' usageShawn O. Pearce2009-04-181-3/+11
| | | | | | | | | | | Its unlikely that a new version of repo will be delivered in any given day, so we now check only once every 24 hours to see if repo has been updated. This reduces the sync cost, as we no longer need to contact the repo distribution servers every time we do a sync. repo selfupdate can still be used to force a check. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Change repo sync to be more friendly when updating the treev1.6.6Shawn O. Pearce2009-04-161-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | We now try to sync all projects that can be done safely first, before we start rebasing user commits over the upstream. This has the nice effect of making the local tree as close to the upstream as possible before the user has to start resolving merge conflicts, as that extra information in other projects may aid in the conflict resolution. Informational output is buffered and delayed until calculation for all projects has been done, so that the user gets one concise list of notice messages, rather than it interrupting the progress meter. Fast-forward output is now prefixed with the project header, so the user can see which project that update is taking place in, and make some relation of the diffstat back to the project name. Rebase output is now prefixed with the project header, so that if the rebase fails, the user can see which project we were operating on and can try to address the failure themselves. Since rebase sits on a detached HEAD, we now look for an in-progress rebase during sync, so we can alert the user that the given project is in a state we cannot handle. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Add 'repo selfupdate' to upgrade only repoShawn O. Pearce2009-04-131-15/+24
| | | | | | | Users may want to upgrade only repo to the latest release, but leave their working tree state alone and avoid 'repo sync'. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Add a project progress meter to 'repo sync'Shawn O. Pearce2009-04-101-1/+8
| | | | | | | | This way users can see how much is left during fetch. Its especially useful when most syncs are no-ops but there are hundreds of repositories to poll. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Add 'repo sync -l' to only do local operationsShawn O. Pearce2009-04-101-25/+33
| | | | | | | | | | | | | This permits usage of 'repo sync' while offline, as we bypass the network based portions of the code and do only the local sync. An example use case might be: repo sync -n ; # while we have network ... some time later ... repo sync -l ; # while without network, come up to date Signed-off-by: Shawn O. Pearce <sop@google.com>
* Add 'repo sync -d' to detach projects from their current topicShawn O. Pearce2009-04-101-1/+15
| | | | | | | | | | The -d flag moves the project back to a detached HEAD state, matching what is listed in the manifest. This can be useful to set a client to something stable (or at least well-known), such as before a sequence of 'repo download' commands are used to get some changes for testing. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Add 'repo sync -n' to only do the network transferShawn O. Pearce2009-04-101-0/+7
| | | | | | | | This makes it easier to update all repositories, without actually impacting the working directory, or learning about how to use `repo forall -c 'git fetch $REPO_REMOTE' `. Signed-off-by: Shawn O. Pearce <sop@google.com>
* Hide the internal sync --repo-upgraded flag from usersShawn O. Pearce2009-04-101-1/+2
| | | | | | | | This is only meant to be passed through while repo upgrades itself during a sync. It should never be something a user invokes on their own. Signed-off-by: Shawn O. Pearce <sop@google.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-06-07 05:35:32 +0000