From 5510a784fffd4fee89bc068d2689e865dfa0b9c3 Mon Sep 17 00:00:00 2001 From: Bruce Ashfield Date: Fri, 8 Sep 2023 19:58:53 +0000 Subject: oe-go-mod-autogen: add README and extended --help information Signed-off-by: Bruce Ashfield --- scripts/README-oe-go-mod-autogen.txt | 81 ++++++++++++++++++++++++++++ scripts/oe-go-mod-autogen.py | 100 +++++++++++++++++++++++++++++++++-- 2 files changed, 178 insertions(+), 3 deletions(-) create mode 100644 scripts/README-oe-go-mod-autogen.txt diff --git a/scripts/README-oe-go-mod-autogen.txt b/scripts/README-oe-go-mod-autogen.txt new file mode 100644 index 00000000..e5c11d90 --- /dev/null +++ b/scripts/README-oe-go-mod-autogen.txt @@ -0,0 +1,81 @@ +Description: + + oe-go-mod-autogen.py is a helper script for go mod recipes. It is based + on intiial efforts to use only the git fetcher versus golangs module + infrastructure. + + Example: + + cmd: /meta-virtualization/scripts/oe-go-mod-autogen.py \ + --repo https://github.com/docker/compose --rev v2.20.3 + output: src_uri.inc, relocation.inc, modules.txt + + Copy these three generated files to replace the original ones, + then we only need update PV and SRCREV, and docker-compose is upgraded. + + See --help for more explanations and examples. + + Below are some technical details. + + * get module's repo from module name + + This script checks the following two URLs to determine the module's repo. + 1. https://?=go-get=1 + 2. https://pkg.go.dev/ + + The module_name_tweaked is derived from module_name, with the last components + removed one by one. Let me use two examples to explain this. + + For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as + module_name_tweaked for searching. And we can correctly get the repo URL, so + the search stops. + + For module_name github.com/k3s-io/etcd/api/v3, the following ones are used + as module_name_tweaked: + github.com/k3s-io/etcd/api/v3 + github.com/k3s-io/etcd/api + github.com/k3s-io/etcd + And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search + stops. + + * determine the srcdir:destdir mapping in 'vendor' creation + + To correctly form the 'vendor' directory, the mapping is critical. + This script makes use of tag matching and path matching to determine + the subpath in the repo for the module. + + * avoid subpath being overriden by parent path + + We need to avoid subpath being overriden by parent path. This is needed + for both SRC_URI ordering in src_uri.inc and the sites mapping ordering + in relocation.inc. This script simply uses the length as the ordering key, + simply for the reason that if a path is a subpath of another path, it must + be longer. + + * the .git suffix is removed to sync with each other + + Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script + remove the '.git' suffix from repo URL so that the repo URLs are in sync + with each. + + * basic directory hierarchy and caching mechanism + + /repos: hold the repos downloaded and checked + /wget-contents: hold the contents to determine the module's repo + /wget-contents/.repo_url.cache: the repo value cache + This is to avoid unnecessary URL fetching and repo cloning. + + * the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script + + The script must get the correct repo_url, fullsrc_rev and subpath for + each required module in go.mod to correctly generate the src_uri.inc and + relocation.inc files. If this process fails for any required module, this + script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE + to True in this script. The purpose is to encourage people to report + problems to meta-virt so that we can improve this script according to + these feedbacks. But this variable can set to False, then the script + only records the failed modules in self.modules_unhandled with reasons + added, people can modify the generated src_uri.inc and relocation.inc + to manually handle these unhandled modules if they are urgent to + add/upgrade some go mod recipes. + diff --git a/scripts/oe-go-mod-autogen.py b/scripts/oe-go-mod-autogen.py index aa155f0f..2970847e 100755 --- a/scripts/oe-go-mod-autogen.py +++ b/scripts/oe-go-mod-autogen.py @@ -644,17 +644,111 @@ def main(): epilog=textwrap.dedent('''\ Overview: + ========= go-mod-oe is a tool for processing go dependencies to generate dependencies suitable for OE fetcher consumption. In particular, it creates a build structure suitable for - '-mod="vendor"' go builds. All dependencies are in the vendor/ - directory, so no golang specific fetching or network access happens - during the build. + '-mod="vendor"' go builds. Once complete all go mod dependencies + are in the vendor/ directory, so no golang specific fetching or + network access happens during the build. The files src_uri.inc, relocation.inc and modules.txt are generated and suitable for recipe inclusion. + + A recipe build can then use these files to leverage the git fetcher + and related functionality (mirrors, sstate, etc). + + Note 1: --rev does not have to be a tag, if you want to track the tip of + a branch specify the latest git has on that branch, and it will + be used. + + Note 2: This script does not generate an entire recipe, the way the + the outputs are used can be modified as required. + + Note 3: if a go.mod has a bad revision, or needs to be manually updated + to fetch fixes: go.mod in the main repository (see the repos/ + directory). If go.mod is edited, modules.txt also has to be + updated to match the revision information. + + How to use in a recipe: + ======================= + + There are examples in meta-virtualization of recipes that use this + script and stragegy for builds: docker-compose, nerdcli, k3s + + 1) The recipe should set the master repository SRCREV details, and then include + the src_uri.inc file: + + SRCREV_nerdcli = "e084a2df4a8861eb5f0b0d32df0643ef24b81093" + SRC_URI = "git://github.com/containerd/nerdctl.git;name=nerdcli;branch=master;protocol=https" + + include src_uri.inc + + This results in the SRC_URI being fully populated with the main + repository and all dependencies. + + 2) The recipe should either copy, or include the relocation.inc file. It sets + a variable "sites" that is a list of source locations (where the src_uri.inc + fetches) and destination in a vendor directory, it also has a do_compile:prepend() + that contains a loop which relocates the fetches into a vendor.copy directory. + + It is expected to be processed as follows, before compilation starts: + + # sets the "sites" variable and copies files + include relocation.inc + + The do_compile:prepend, contains the following loop: + + cd ${S}/src/import + # this moves all the fetches into the proper vendor structure + # expected for build + for s in ${sites}; do + site_dest=$(echo $s | cut -d: -f1) + site_source=$(echo $s | cut -d: -f2) + force_flag=$(echo $s | cut -d: -f3) + mkdir -p vendor.copy/$site_dest + if [ -n "$force_flag" ]; then + echo "[INFO] $site_dest: force copying .go files" + rm -rf vendor.copy/$site_dest + rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest + else + [ -n "$(ls -A vendor.copy/$site_dest/*.go 2> /dev/null)" ] && { echo "[INFO] vendor.fetch/$site_source -> $site_dest: go copy skipped (files present)" ; true ; } || { echo "[INFO] $site_dest: copying .go files" ; rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest ; } + fi + done + + The main compile() function, should set the appropriate GO variables, + copy modules.txt and build the appripriate target: + + # our copied .go files are to be used for the build + ln -sf vendor.copy vendor + + 3) The modules.txt file should be copied into the recipe directory, included + on the SRC_URI and copied into place after the relocation has been + processed. + + # patches and config + SRC_URI += "file://0001-Makefile-allow-external-specification-of-build-setti.patch \\ + file://modules.txt \ + " + + ..... + + cp ${WORKDIR}/modules.txt vendor/ + + Example: Updating the K3S recipe + ================================ + + % cd meta-virtualization/recipe-containers/k3s/ + # produces src_uri.inc, relocation.inc and modules.txt in the current directory + % ../../scripts/oe-go-mod-autogen.py --repo https://github.com/rancher/k3s.git --rev v1.27.5+k3s1 + + % cp modules.txt k3s/ + + ... add and commit files. + + ''')) parser.add_argument("--repo", help = "Repo for the recipe.", required=True) parser.add_argument("--rev", help = "Revision for the recipe.", required=True) -- cgit v1.2.3-54-g00ecf