Summary

Recently during the work on the Funtoo Telemetry Project Phase 1, I have been creating CatPkgs using the YAML driven autogenetors, specifically the github-1.py generator.

After discussing with drobbins over Discord, it sounds like a long term goal is to convert many freestyle autogens that leverage autogen.py to YAML driven autogens.

With that in mind, I would like to request a new feature to the schema of GitHub YAML autogens for the Funtoo Telemetry Project:

•  Support for GitHub Releases multi-architecture binary tarballs in the funtoo-metatools autogen YAML schema

Scope

This funtoo-metatools feature would be scoped to very specific types of packages in kit-fixups that meet these requirements:

• *-bin CatPks only
• Software hosted on GitHub Repositories
• Software automatically released via GitHub Releases
• GitHub releases that have multi-architecture tarballs assets (Golang, Rust, etc)

A good high quality example of this type of package: Prometheus GitHub Releases at: https://github.com/prometheus/prometheus/releases

Implementation Ideas

Taking all this into consideration here are some ideas around implementation including pseudo code. I will be using upstream Prometheus GitHub Releases as our example package that could use this feature today.

autogen.yml Schema

In the new app-metrics/promethus-bin autogen.yml , the YAML tarball dictionary key looks like this:

prometheus-bin:
  generator: github-1
  packages:
    - prometheus-bin:
        tarball: prometheus-{version}.linux-amd64.tar.gz 

As you can see the architecture of the package is hardcoded, so that the generator will download the right GitHub Release asset tarball. This architecture name is set by the upstream software build system generator the asset tarball, in this scenario it is Golang.

Because not all packages attach these types of assets, this would be for specific types of packages. I would add a new schema dictionary key that is essentially a boolean under within the package list item's YAML schema that can toggle this feature on or off, so like this:

  packages:
    - prometheus-bin:
        multi_arch: true
        tarball: prometheus-{version}.{arch}.tar.gz

Then to make things flexible add another a sister dictionary key the goes along with multi-arch that is a list of upstream Linux architectures, as defined by the assets, so like:

prometheus-bin:
  generator: github-1
  packages:
    - prometheus-bin:
        multi_arch: true
        arches:
          - linux-amd64
          - linux-arm64
          - linux-ppc64
        tarball: prometheus-{version}.{arch}.tar.gz

Having this in your autogen.yml schema would then conditionally enable some functions in the actually funtoo-metatools github module, which will automatically scrape the GitHub Releases JSON dictionary's assets key for these tarball URLS, basically grabbing the ones you want.

Question: Why not just scrape all linux-* assets from the GitHub Release JSON assets lists?

Answer: Because not all Linux architectures automatically released and included that GitHub Release may be supported on Funtoo. Also, different languages may have different architecture names based on the languages cross compiler, so being explicit about this is probably more robust. Furthermore, package maintainers may want to test different architecture slowly and roll them out. Being able to explicitly turn on different architectures gives greater flexible, sacrificing some overhead.

That being said, funtoo-metatools could have a helper function that dumps out all the GitHub Releases assets tarball names, for convenience when developing to identify which architectures are available on a Released asset.

Ebuild Template

So the other component of adding the architecture specific tarballs to the ebuild template SRC_URI and enabling the Gentoo/Funtoo specific keywords for the package. All this should be handled easily with some reusable Jinja template code.

That is fine to use Jinja but one small wrinkle will be the Gentoo/Funtoo architecture keywords will not map 1-1 to the upstream GitHub release tarball architecture.

So, a small mapping will need to be present in funtoo-metatools and available to the github-1 generator functions. Example of a mapping:

funtoo_github_arches = {
  "linux-amd64": ['amd64'],
  "linux-arm64": ['arm64'],
  "linux-ppc64": ['ppc64'],
}
>>> funtoo_github_arches
{'linux-amd64': ['amd64'], 'linux-arm64': ['arm64'], 'linux-ppc64': ['ppc64']}
>>> type(funtoo_github_arches)
<class 'dict'>
 

With that mapping within the generator functions, the Funtoo set of keywords can be auto-assembled with some additional functions and logic plus added to the pkginfo object as an attribute with a value of a space delimited string of arches.

Then in the ebuild template you would simply just reference the arches just like we do github_users or other attributes:

KEYWORDS="{{ arches }}"