This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) can be turned into namespace packages by removing the empty __init__.py file.
Then several distribution package can share the same namespace.

Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

Per convention in the Sage library, namespace packages are recognized by the presence of all.py or all__*.py files (#33033).

In this ticket, we apply it to turn sagemath-objects and sagemath-categories (from #29865) into namespace packages,
in which __init__.py are removed from the MANIFEST.in files.

Because the actual files are not removed from the source tree, the normal build of the Sage distribution is not affected.

To test these two distributions:

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-categories && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Follow-up tickets:

• Remove __init__.py files for subpackages designated to be namespace packages #33011 Remove the __init__.py files for packages designated to be namespace packages
• Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
• sage_setup: Modify clean_stale_files to support out-of-tree namespace packages #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

• Meta-ticket: Modularize sagelib into separate distributions (pip-installable packages) sagemath-... for Sage 10.x #29705: Meta-ticket: Modularize sagelib into separate distutils packages
• Initial discussion of namespace packages for Sage in Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems #28175 and Use native namespace package sage-numerical-backends-coin#4
• src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory #21654: src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory

Depends on #33803

CC: @isuruf @embray @jdemeyer @dimpase @dcoudert @videlec @vbraun @tobiasdiez @kiwifb

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: f9df1ae

Reviewer: Kwankyu Lee, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28925