summaryrefslogtreecommitdiffstats
path: root/subcmds/sync.py
Commit message (Collapse)AuthorAgeFilesLines
...
| * Fix "'module' object is not callable" errorDavid Pursehouse2013-06-081-1/+1
| | | | | | | | | | | | | | | | In a couple of files the urlparse method was not being set up correctly for python < 3 and this resulted in an error being thrown when trying to call it. Change-Id: I4d2040ac77101e4e228ee225862f365ae3d96cec
* | Degrade: Fix smart sync/smart tagVictor Boivie2013-06-081-3/+4
|/ | | | | | | | | | | This was broken in b2bd91c, which updated the manifest after it had been overridden, which made it fall back to the original file (and not the one from the manifest server). This builds on 0766900 and overrides the manifest by the one downloaded from the manifest server completely. Change-Id: Ic3972390a68919b614616631d99c9e7a63c0e0db
* Move Python version checking to a separate moduleDavid Pursehouse2013-05-231-11/+6
| | | | | | | | | | | | | | | | | Add a new module with methods for checking the Python version. Instead of handling Python3 imports with try...except blocks, first check the python version and then import the relevant modules. This makes the code a bit cleaner and will result in less diff when/if we remove support for Python < 3 later. Use the same mechanism to handle `input` vs. `raw_input` and add suppression of pylint warnings caused by redefinition of the built-in method `input`. Change-Id: Ia403e525b88d77640a741ac50382146e7d635924 Also-by: Chirayu Desai <cdesai@cyanogenmod.org> Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
* Some fixes for supporting python3Chirayu Desai2013-04-181-8/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix imports. * Use python3 syntax. * Wrap map() calls with list(). * Use list() only wherever needed. (Thanks Conley!) * Fix dictionary iteration methods (s/iteritems/items/). * Make use of sorted() in appropriate places * Use iterators directly in the loop. * Don't use .keys() wherever it isn't needed. * Use sys.maxsize instead of sys.maxint TODO: * Make repo work fully with python3. :) Some of this was done by the '2to3' tool [1], by applying the needed fixes in a way that doesn't break compatibility with python2. Links: [1]: http://docs.python.org/2/library/2to3.html Change-Id: Ibdf3bf9a530d716db905733cb9bfef83a48820f7 Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
* sync: be more verboseChirayu Desai2013-04-151-0/+5
| | | | | | | * Print project name if the "quiet" option is not used. Change-Id: I99863bb50f66e4dcbaf2d170bdd05971f2a4e19a Signed-off-by: Chirayu Desai <cdesai@cyanogenmod.org>
* Reload the correct manifest during sync.Tim Kilbourn2013-03-081-2/+9
| | | | | | | Fix for issue #134 https://code.google.com/p/git-repo/issues/detail?id=134 Change-Id: I94c2dea5dd63917e3f9c90cbd628921d7d61b12a
* Fix: Missing spaces in printed messagesDavid Pursehouse2013-03-051-2/+2
| | | | | | | | | | | | | | | | | | | | | Several messages are printed with the `print` method and the message is split across two lines, i.e.: print('This is a message split' 'across two source code lines') Which causes the message to be printed as: This is a message splitacross two source code lines Add a space at the end of the first line before the line break: print('This is a message split ' 'across two source code lines' Also correct a minor spelling mistake. Change-Id: Ib98d93fcfb98d78f48025fcc428b6661380cff79
* Add --no-tags option to prevent fetching of tagsMitchel Humpherys2013-02-271-3/+9
| | | | | | | Add an option to pass `--no-tags' to `git fetch'. Change-Id: I4158cc369773e08e55a167091c38ca304a197587 Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
* Allow sync to run even when the manifest is broken.Torne (Richard Coles)2012-12-051-2/+5
| | | | | | | | | If the current manifest is broken then "repo sync" fails because it can't retrieve the default value for --jobs. Use 1 in this case, in order that you can "repo sync" to get a fixed manifest (assuming someone fixed it upstream). Change-Id: I4262abb59311f1e851ca2a663438a7e9f796b9f6
* Represent git-submodule as nested projects, take 2Che-Liang Chiou2012-11-191-2/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Previous submission of this change broke Android buildbot due to incorrect regular expression for parsing git-config output. During investigation, we also found that Android, which pulls Chromium, has a workaround for Chromium's submodules; its manifest includes Chromium's submodules. This new change, in addition to fixing the regex, also take this type of workarounds into consideration; it adds a new attribute that makes repo not fetch submodules unless submodules have a project element defined in the manifest, or this attribute is overridden by a parent project element or by the default element.) We need a representation of git-submodule in repo; otherwise repo will not sync submodules, and leave workspace in a broken state. Of course this will not be a problem if all projects are owned by the owner of the manifest file, who may simply choose not to use git-submodule in all projects. However, this is not possible in practice because manifest file owner is unlikely to own all upstream projects. As git submodules are simply git repositories, it is natural to treat them as plain repo projects that live inside a repo project. That is, we could use recursively declared projects to denote the is-submodule relation of git repositories. The behavior of repo remains the same to projects that do not have a sub-project within. As for parent projects, repo fetches them and their sub-projects as normal projects, and then checks out subprojects at the commit specified in parent's commit object. The sub-project is fetched at a path relative to parent project's working directory; so the path specified in manifest file should match that of .gitmodules file. If a submodule is not registered in repo manifest, repo will derive its properties from itself and its parent project, which might not always be correct. In such cases, the subproject is called a derived subproject. To a user, a sub-project is merely a git-submodule; so all tips of working with a git-submodule apply here, too. For example, you should not run `repo sync` in a parent repository if its submodule is dirty. Change-Id: I4b8344c1b9ccad2f58ad304573133e5d52e1faef
* Tidy up code formatting a bit moreDavid Pursehouse2012-11-141-6/+6
| | | | | | | | | | | | Enable the following Pylint warnings: C0322: Operator not preceded by a space C0323: Operator not followed by a space C0324: Comma not followed by a space And make the necessary fixes. Change-Id: I74d74283ad5138cbaf28d492b18614eb355ff9fe
* Fix inconsistent indentationDavid Pursehouse2012-11-141-84/+84
| | | | | | | | | | The repo coding style is to indent at 2 characters, but there are many places where this is not followed. Enable pylint warning "W0311: Bad indentation" and make sure all indentation is at multiples of 2 characters. Change-Id: I68f0f64470789ce2429ab11104d15d380a63e6a8
* Change print statements to work in python3Sarah Owens2012-11-131-48/+54
| | | | | | This is part of a series of changes to introduce Python3 support. Change-Id: I373be5de7141aa127d7debdbce1df39148dbec32
* Always show --manifest-server-* optionsConley Owens2012-11-061-6/+6
| | | | | | | | | The --manifest-server-* flags broke the smartsync subcmd since the corresponding variables weren't getting set. This change ensures that they will always be set, regardless of whether we are using sync -s or smartsync. Change-Id: I1b642038787f2114fa812ecbc15c64e431bbb829
* sync: Only parallelize gc for git >= 1.7.2Dave Borowitz2012-10-311-2/+3
| | | | | | | | | This minimum version is required for the -c argument to set config on the command line. Without this option, git by default uses as many threads per invocation as there are CPUs, so we cannot safely parallelize without hosing a system. Change-Id: I8fd313dd84917658162b5134b2d9aa34a96f2772
* Even more coding style cleanupDavid Pursehouse2012-10-301-1/+1
| | | | | | | | | | | Fixing some more pylint warnings: W1401: Anomalous backslash in string W0623: Redefining name 'name' from outer scope W0702: No exception type(s) specified E0102: name: function already defined line n Change-Id: I5afcdb4771ce210390a79981937806e30900a93c
* Revert "Represent git-submodule as nested projects"v1.11.1Shawn O. Pearce2012-10-261-20/+1
| | | | | | | | | | | This reverts commit 69998b0c6ff724bf620480140ccce648fec7d6a9. Broke Android's non-gitmodule use case. Conflicts: project.py subcmds/sync.py Change-Id: I68ceeb63d8ee3b939f85a64736bdc81dfa352aed
* repo selfupdate: Fix _PostRepoUpgrade takes 2 argumentsShawn O. Pearce2012-10-261-3/+3
| | | | Change-Id: I1cf9e0674ea366ddce96c949e0bc085e3452b25a
* Update PGP keys during _PostRepoUpgrade in `sync`Conley Owens2012-10-251-2/+6
| | | | | | | | Previously, if a key was added, a client wouldn't add the key during the sync step. This would cause issues if a new key were added and a subsequent release were signed by that key. Change-Id: I4fac317573cd9d0e8da62aa42e00faf08bfeb26c
* Merge "sync: Run gc --auto in parallel"Shawn Pearce2012-10-251-2/+53
|\
| * sync: Run gc --auto in parallelDave Borowitz2012-10-251-2/+53
| | | | | | | | | | | | | | | | | | We can't just let this run wild with a high (or even low) -j, since that would hose a system. Instead, limit the total number of threads across all git gc subprocesses to the number of CPUs reported by the multiprocessing module (available in Python 2.6 and above). Change-Id: Icca0161a1e6116ffa5f7cfc6f5faecda510a7fb9
* | Merge "sync: Keep a moving average of last fetch times"Shawn Pearce2012-10-251-5/+17
|\|
| * sync: Keep a moving average of last fetch timesDave Borowitz2012-10-241-5/+17
| | | | | | | | | | | | | | | | | | | | Try to more accurately estimate which projects take the longest to sync by keeping an exponentially weighted moving average (a=0.5) of fetch times, rather than just recording the last observation. This should discount individual outliers (e.g. an unusually large project update) and hopefully allow truly slow repos to bubble to the top. Change-Id: I72b2508cb1266e8a19cf15b616d8a7fc08098cb3
* | Merge "sync: Order projects according to last fetch time"Shawn Pearce2012-10-251-1/+60
|\|
| * sync: Order projects according to last fetch timeDave Borowitz2012-10-241-1/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some projects may consistently take longer to fetch than others, for example a more active project may have many more Gerrit changes than a less active project, which take longer to transfer. Use a simple heuristic based on the last fetch time to fetch slower projects first, so we do not tend to spend the end of the sync fetching a small number of outliers. This algorithm is probably not optimal, and due to inter-run latency variance and Python thread scheduling, we may not even have good estimates of a project sync time. Change-Id: I9a463f214b3ed742e4d807c42925b62cb8b1745b
* | Fix pylint warnings introduced by the submodule patchChe-Liang Chiou2012-10-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | "69998b0 Represent git-submodule as nested projects" has introduced a few pylint warnings. W0612:1439,8:Project._GetSubmodules.get_submodules: Unused variable 'sub_gitdir' W0613:1424,36:Project._GetSubmodules.get_submodules: Unused argument 'path' W0612:1450,25:Project._GetSubmodules.parse_gitmodules: Unused variable 'e' W0622:516,8:Sync.Execute: Redefining built-in 'all' Change-Id: I84378e2832ed1b5ab023e394d53b22dcea799ba4
* | Merge "Use modern Python exception syntax"Conley Owens2012-10-251-3/+3
|\ \
| * | Use modern Python exception syntaxSarah Owens2012-10-231-3/+3
| |/ | | | | | | | | | | | | | | | | | | | | "except Exception as e" instead of "except Exception, e" This is part of a transition to supporting Python 3. Python >= 2.6 support "as" syntax. Note: this removes Python 2.5 support. Change-Id: I309599f3981bba2b46111c43102bee38ff132803
* / Represent git-submodule as nested projectsChe-Liang Chiou2012-10-231-1/+20
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We need a representation of git-submodule in repo; otherwise repo will not sync submodules, and leave workspace in a broken state. Of course this will not be a problem if all projects are owned by the owner of the manifest file, who may simply choose not to use git-submodule in all projects. However, this is not possible in practice because manifest file owner is unlikely to own all upstream projects. As git submodules are simply git repositories, it is natural to treat them as plain repo projects that live inside a repo project. That is, we could use recursively declared projects to denote the is-submodule relation of git repositories. The behavior of repo remains the same to projects that do not have a sub-project within. As for parent projects, repo fetches them and their sub-projects as normal projects, and then checks out subprojects at the commit specified in parent's commit object. The sub-project is fetched at a path relative to parent project's working directory; so the path specified in manifest file should match that of .gitmodules file. If a submodule is not registered in repo manifest, repo will derive its properties from itself and its parent project, which might not always be correct. In such cases, the subproject is called a derived subproject. To a user, a sub-project is merely a git-submodule; so all tips of working with a git-submodule apply here, too. For example, you should not run `repo sync` in a parent repository if its submodule is dirty. Change-Id: I541e9e2ac1a70304272dbe09724572aa1004eb5c
* Rename "dir" variablesMickaël Salaün2012-10-101-4/+4
| | | | | | | The variable name "dir" conflicts with the name of a Python built-in function: http://docs.python.org/library/functions.html#dir Change-Id: I850f3ec8df7563dc85e21f2876fe5e6550ca2d8f
* Coding style cleanupDavid Pursehouse2012-10-091-7/+6
| | | | | | | | | | | | | | | Fix the following issues reported by pylint: C0321: More than one statement on a single line W0622: Redefining built-in 'name' W0612: Unused variable 'name' W0613: Unused argument 'name' W0102: Dangerous default value 'value' as argument W0105: String statement has no effect Also fixed a few cases of inconsistent indentation. Change-Id: Ie0db839e7c57d576cff12d8c055fe87030d00744
* Remove unreachable codeDavid Pursehouse2012-10-051-8/+0
| | | | | | | Change 9bb1816b removed part of a block of code, but left the remaining part unreachable. Remove it. Change-Id: Icdc6061d00e6027df32dee9a3bad3999fe7cdcbc
* sync: Support manual authentication to the manifest serverDavid Pursehouse2012-09-211-19/+52
| | | | | | | | | | | | | | | | Add two new command line options, -u/--manifest-server-username and -p/--manifest-server-password, which can be used to specify a username and password to authenticate to the manifest server when using the -s/--smart-sync or -t/--smart-tag option. If -u and -p are not specified when using the -s or -t option, use authentication credentials from the .netrc file (if there are any). Authentication credentials from -u/-p or .netrc are not used if the manifest server specified in the manifest file already includes credentials. Change-Id: I6cf9540d28f6cef64c5694e8928cfe367a71d28d
* sync: Support authentication to manifest server with .netrcDavid Pursehouse2012-09-111-1/+29
| | | | | | | | | | | | | | | | | | | | When using the --smart-sync or --smart-tag option, and the specified manifest server is hosted on a server that requires authentication, repo sync fails with the error: HTTP 401 Unauthorized. Add support for getting the credentials from the .netrc file. If a .netrc file exists in the user's home directory, and it contains credentials for the hostname of the manifest server specified in the manifest, use the credentials to authenticate with the manifest server using the URL syntax extension for Basic Authentication: http://user:password@host:port/path Credentials from the .netrc file are only used if the manifest server URL specified in the manifest does not already include credentials. Change-Id: I06e6586e8849d0cd12fa9746789e8d45d5b1f848
* sync: Correct imports of `R_HEADS` and `HEAD`David Pursehouse2012-09-071-3/+1
| | | | | | | | | | | | | | `R_HEADS` is imported twice, from both the git_refs and project modules. It is actually defined in git_refs, and in project it is imported from there, so the import of `R_HEADS` from project in the sync module is redundant. Remove it. `HEAD` is imported from project, but like `R_HEADS` it is actually defined in git_refs. Import it from git_refs instead. Change-Id: I8e2b0217d0d9f9f4ee5ef5b8cd0b026174ac52f4
* sync: catch exceptions when connecting to the manifest serverDavid Pursehouse2012-09-061-3/+7
| | | | | | | | | | | | | | | | | When connecting to the manifest server, exceptions can occur but are not caught, resulting in the repo sync exiting with a python traceback. Add handling of the following exceptions: - IOError, which can be raised for example if the manifest server URL is malformed. - xmlrpclib.ProtocolError, which can be raised if the connection to the manifest server fails with HTTP error. - xmlrpclib.Fault, which can be raised if the RPC call fails for some other reason. Change-Id: I3a4830aef0941debadd515aac776a3932e28a943
* Make "repo sync -j<count>" stop properly on Ctrl-C.David 'Digit' Turner2012-09-051-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | The threaded 'repo sync' implementation would very often freeze the process when interrupted by the user with Ctrl-C. The only solution being to kill -9 the process explicitly from another terminal. The reason for this is best explained here: http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html This patch makes all helper sync threads 'daemon', which allows the process to terminate immediately on Ctrl-C. Note that this will forcefully kill all threads in case of interruption; this is generally a bad thing, but: 1/ This is equivalent to calling kill -9 in another terminal, which is the _only_ thing that can currently stop the process. 2/ There doesn't seem to be a way to tell the worker threads to gently stop when they are in a blocking operation anyway (even in the non-threaded case). + Do the same for "repo status -j<count>". Change-Id: Ieaf45b0eacee36f35427f8edafd87415c2aa7be4
* sync: Honor --no-clone-bundle with -j1Shawn O. Pearce2012-08-021-2/+4
| | | | Change-Id: I7c12902e386121a374d525be673092360c67c53d
* Add manifest groupsv1.8.2Colin Cross2012-04-131-2/+3
| | | | | | | | | | | | | | | | | Allows specifying a list of groups with a -g argument to repo init. The groups act on a group= attribute specified on projects in the manifest. All projects are implicitly labelled with "default" unless they are explicitly labelled "-default". Prefixing a group with "-" removes matching projects from the list of projects to sync. If any non-inverted manifest groups are specified, the default label is ignored. Change-Id: I3a0dd7a93a8a1756205de1d03eee8c00906af0e5 Reviewed-on: https://gerrit-review.googlesource.com/34570 Reviewed-by: Shawn Pearce <sop@google.com> Tested-by: Shawn Pearce <sop@google.com>
* sync: --no-clone-bundle disables the clone bundle supportv1.8.0Shawn O. Pearce2012-03-141-2/+13
| | | | Change-Id: Ia9ed7da8451b273c1be620c3dd0dcad777b29096
* sync: Add manifest_name parameterChris Wolfe2012-01-261-0/+12
| | | | | | | | | | | | This parameter changes the manifest used by 'repo sync' for only this execution. It should be useful for developers wishing to get the repo temporarily into a known state, without clobbering their existing manifest. Tested by shifting Chrome OS between minilayout and full, and between several release-builder-generated manifests. Change-Id: I14194b665195b0e78f368d9ec8b8a83227af2627
* Add a sync flag that fetches only current branchAnatol Pomazau2011-11-031-3/+9
| | | | | | | | | | | There is also shortcuts in case if the "current branch" is a persistent revision such as tag or sha1. We check if the persistent revision is present locally and if it does - do no fetch anything from the server. This greately reduces sync time and size of the on-disk repo Change-Id: I23c6d95185474ed6e1a03c836a47f489953b99be
* Fix Python 2.4 supportShawn O. Pearce2011-10-111-8/+4
| | | | | Change-Id: I89521ae52fa564f0d849cc51e71fee65b3c47bab Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Update default -j flag from manifestShawn O. Pearce2011-09-261-1/+2
| | | | | | | | | If the manifest is updated and the default sync-j attribute was modified, honor it during this sync session if the user has not supplied a -j flag on the command line. Change-Id: I127ee5c779e2bbbb40b30bddc10ec1fa704b3bf3 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Allow -j to have a default in manifestv1.7.6Shawn O. Pearce2011-09-221-1/+4
| | | | | | | | | | | | | This permits manifest authors to suggest a number of parallel fetch operations against a remote server. For example, Gerrit Code Review servers support queuing of requests and processes them in first-in, first-out order. Running concurrent fetches can utilize multiple CPUs on the Gerrit server, but will also decrease overall operation latency by having the request put into the queue ready to execute as soon as a CPU is free. Change-Id: I3d3904acb6f63516bae4b071c510ad57a2afab18 Signed-off-by: Shawn O. Pearce <sop@google.com>
* sync: Limit -j to file descriptorsShawn O. Pearce2011-09-221-0/+12
| | | | | | | | | | | Each worker thread requires at least 3 file descriptors to run the forked 'git fetch' child to operate against the local repository. Mac OS X has the RLIMIT_NOFILE set to 256 by default, which means a sync -j128 often fails when the workers run out of pipes within the Python parent process. Change-Id: I2cdb14621b899424b079daf7969bc8c16b85b903 Signed-off-by: Shawn O. Pearce <sop@google.com>
* Smart tag supportVictor Boivie2011-07-201-15/+23
| | | | | | | | | This is an evolution of 'smart-sync' that adds a new option, -t, that allows you to specify a tag/label to use instead of the "latest good build" on the current manifest branch which -s does. Signed-off-by: Victor Boivie <victor.boivie@sonyericsson.com> Change-Id: I8c20fd91104a6aafa0271d4d33f6c4850aade17e
* Fix parallel sync on python < 2.6.Daniel Sandler2011-04-041-2/+2
| | | | | | | | Event.isSet was renamed to is_set in 2.6, but we should use the earlier syntax to avoid breaking compatibility with older Python installations. Change-Id: I41888ed38df278191d7496c1a6eed15e881733f4
* sync: Fix syntax error on Python 2.4v1.7.4.2Shawn O. Pearce2011-03-221-22/+23
| | | | | Change-Id: I371d032d5a1ddde137721cbe2b24bfa38f20aaaa Signed-off-by: Shawn O. Pearce <sop@google.com>
* Make 'repo sync -jN' exit with an error code in the case of sync errors.Doug Anderson2011-03-171-14/+70
| | | | | | | | | | | | The bug that this is fixing is described here: http://code.google.com/p/chromium-os/issues/detail?id=6813 This fix allows the helper threads to signal the main thread that they saw an error. When the main thread sees the error, it will let all existing threads finish, then exit with an error. Change-Id: If3019bc6b0b3ab9304d49ed2eea53e9d57f3095a
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-05-13 12:34:59 +0000