Uploaded image for project: 'Funtoo Linux'
  1. Funtoo Linux
  2. FL-6574

lethargic build times - pillow

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • None
    • None

      During 1.4 transition, emerge would remain stalled on building catpkg. This process would remain unfinished for hours. Observation was an extremely depressed load average being made on CPU. (6-core AMD Bulldozer w/ persistent load of
      avgs 0.5 0.35 0.35 observed).

      Review of log of catpkg found that upstream ([source:|https://github.com/python-pillow/Pillow/blob/master/setup.py] detects and limits ninja build. Unlike make, my understanding is that ninja is very sensitive to build order and load. In my case, log file indicated that ninja build was limited to four threads(line 393 and others).

      In researching the lethargy, a portage related problem became apparent. catpkg build process is further throttled by ebuild/eclass.

      catpkg ebuild is EAPI=6 with python 3.7. Ebuild calls the function distutils-r1_python_compile which then further throttles build based on EAPI and python version.
      From file:

      core-kit/eclass/distutils-r1.eclass
              if python_is_python3 && [[ ${EPYTHON} != python3.4 ]]; then
                      if [[ ${EAPI} != [56] || ${EPYTHON} != python3.[56] ]]; then
                              local jobs=$(makeopts_jobs "${MAKEOPTS}" INF)
                              if [[ ${jobs} == INF ]]; then
                                      local nproc=$(get_nproc)
                                      jobs=$(( nproc + 1 ))
                              fi
                              build_args+=( -j "${jobs}" )
                      fi
              fi
      

      This code from the distutils-r1.eclass, in conjunction with the catpkg ebuild, unnecessarily throttles the build process that upstream has regulated. The MAKEOPT setting for core usage, -j, is not passed to the package.

      Change is to move ebuild EAPI to 7 to avoid implications of distutils-r1 function shown above. Also, clean of unnecessary patch files was undertaken. The lethargy of build is somewhat addressed but more needs to be studied to ascertain instances where upstream using ninja regulates CPU usage.

      PR to follow.

            Unassigned Unassigned
            digifuzzy digifuzzy [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: