Meta-ticket: Modularize sagelib into separate distributions (pip-installable packages) sagemath-... for Sage 10.x

@videlec

We modularize sagelib into separate distributions (pip-installable packages), so that these can be built and run with a much smaller set of dependencies. The packages are made available individually on PyPI in source and wheel form. The modularization is done without changing the overall structure of the source tree by means of PEP 420 implicit namespace packages.

SageMath modularization project: The five by five (sage-devel, 2023-06)
I. The goals – II. The tools – III. The hooks – IV. The rules – V. The blocs
Video from Sage Days 120
Section Packaging the Sage Library in the Sage Developer's Guide (added in Sage 9.5)

Mega-PR #35095 (not for merging; changes are being cherry-picked from here for integration in Sage) adds the new distributions. Instead of defining a distribution package for every possible community or subfield of mathematics that Sage supports, the modularization project introduces three types of distribution packages:

Distributions named after a basic mathematical structure: The packages may also cover a wide range of generalizations/applications of the structure after which they are named. Users who work in a specialized research area will, of course, recognize what structures they need. The down-to-earth naming also creates discoverability by a broader audience. Not many more distribution packages than these 7 are needed:

sagemath-combinat: provides "everything combinatorial", except for graphs.
sagemath-graphs: also provides posets, combinatorial designs, abstract simplicial complexes, quivers, etc.
sagemath-groups
sagemath-modules: also has matrices, tensors, homology, coding theory, etc.
sagemath-polyhedra: also provides fans, hyperplane arrangements, polyhedral complexes, linear and mixed-integer optimization, lattice point sets, and toric varieties
sagemath-schemes
sagemath-symbolics: symbolic expressions and symbolic calculus

Distributions named after a third-party non-Python dependency: This makes technical sense because the dependencies will be localized to this distribution package, but it also helps give attribution and visibility to these libraries and projects that Sage depends on.

Standard packages	Optional Packages
sagemath-flint	sagemath-bliss
sagemath-gap	sagemath-brial
sagemath-glpk	sagemath-coxeter3
sagemath-giac	sagemath-mcqd
sagemath-homfly	sagemath-meataxe
sagemath-lcalc	sagemath-sirocco
sagemath-libbraiding	sagemath-tdlib
sagemath-libecm
sagemath-linbox
sagemath-ntl
sagemath-pari
sagemath-singular

Distributions named after a technical functionality:

sagemath-objects: Sage extends Python's object system by dynamic mix-in classes that are driven by categories and axioms. It is loosely modeled on concepts of category theory and inspired by Scratchpad/Axiom/FriCAS, Magma, and MuPAD. This distribution package makes Sage objects, the element/parent framework, basic categories and functors, the coercion system and the related metaclasses available.
sagemath-categories: This distribution package contains the full set of categories defined by Sage, as well as basic mathematical objects such as integers and rational numbers, a basic implementation of polynomials, and affine spaces. None of this brings in additional dependencies.
sagemath-environment
sagemath-plot: Plotting facilities, depending on matplotlib.
sagemath-repl: The top-level interactive environment with the preparser that defines the surface language of Sage. This distribution also includes the doctesting facilities, as the doctests are written in the surface language.
sagemath-standard: Everything as provided by a standard installation of the Sage distribution. This is reduced to an empty meta-package.
sagemath-standard-no-symbolics

Older posts and presentations:

Initial proposal (sage-devel, 2020-05)
Presentation video: https://researchseminars.org/talk/SageDays110/11/ (2020-10)
Schematics (2020-10)
Current status of pip-installability (Sage 9.4 release tour, 2021-08)
Refactoring tasks in the Sage library to support modularization (sage-devel, 2021-09)
How to modularize for fun and profit (sage-devel, 2021-10)
Master plan of sage modularization (sage-devel, 2021-11)
Presentation video: https://researchseminars.org/talk/SageDays112358/17/ (2022-06)

Initial steps: (2020)

In particular, create a package sagemath-objects that makes Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available. This package would, for example, not provide or know about integers, polynomials, or matrices (thus avoiding all the dependencies on the various libraries implementing these). Ideally it would only have a build dependency on Cython and no runtime dependencies.

(Doctesting sagemath-objects would still need more of Sage, but some it could also be done by mocking instead of with the real classes.)

a) These facilities could see a wider use in the mathematical Python community, and in this way we could hope to benefit from a larger developer base. For example, we would hope that sagemath-objects could be ported for use with PyPy instead of CPython.

b) Building and deploying a user package would be easier if it could depend on a small package such as sagemath-objects only instead of the whole multigigabyteness of Sage.

This has been completed in Sage 9.6 (Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865).
Remove the mechanism of OptionalExtensions from sagelib.

Currently, a user would install, for example, the optional package tdlib and then rebuild sagelib so that the OptionalExtension sage.graphs.graph_decompositions.tdlib is built and installed.

With this ticket, the user would instead install a new optional package sagemath-tdlib (which has tdlib as a dependency); this installs the Extension sage.graphs.graph_decompositions.tdlib (as a namespace package).

This has been completed in Sage 9.2 (Replace use of module_list and OptionalExtension by extending find_python_sources #29701).

Not within the scope of this ticket:

We will not change the overall structure of the source tree (SAGE_ROOT/src/sage). We will achieve this by using the mechanism of native namespace packages introduced in Python 3.3 (PEP 420, see Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925).
It is not a prerequisite to fight against all circular imports. In particular, a parent class and its corresponding element class would certainly not be separated into separate distribution packages.

Initially noted constraints and challenges: (2020)

Inconsistent handling of symlinks by pip/setuptools. We cannot simply create a distribution of a subset of the files in src by creating a flock of symlinks (see Install sage-env-config as part of sage_conf #29850) and expect everything to work; notably, pip fails because of longstanding issues with installs from local directories (see discussion in Install sage_conf using flit instead of setuptools #29854).

Solution (implemented in Sage 9.2, Install all Python packages via pip wheel (or setup.py bdist_wheel), store wheels in $SAGE_LOCAL/var/lib/sage/wheels #29500): Do not use pip for installations from local source directories. Use setup.py bdist_wheel, followed by pip install of the wheel. pip 21.1's --use-feature=in-tree-build makes this workaround unnecessary (Use pip --use-feature=in-tree-build, replace use of sdh_setup_bdist_wheel by sdh_pip_install #32046, Sage 9.5).
The category framework can not really be separated from the Parent/Element/CategoryObject (e.g. coercion is involved in comparisons and binary operations).

Solution: Therefore they are in the same distribution sagemath-objects.
The category framework cannot live without the integers (e.g. the output of .cardinality() expects an integer).

Near term solution: Modularization of sagelib: Break out a separate package sagemath-polyhedra #32432 makes sagemath-categories larger, to include integers and some other basic rings.

Possible future solution: The coercion system already has the notion of a global coercion model - which is determined at runtime, not compile time (sage.structure.element.get_coercion_model). When methods in sagemath-objects need to create an integer, they should obtain the appropriate class of integers in the same way. An existing function provides this interface already (but has a static implementation):
```
sage: py_scalar_parent(int)
Integer Ring
```
sage.structure.coerce.py_scalar_parent, py_scalar_to_element: Make dynamic #29875 will make this implementation dynamic.
Imports in sage.categories:
```
bimodules imports QQ, RR
classical_crystals imports SR
finite_coxeter_groups imports QQbar
```
Such imports can be removed using the techniques described in section Module-level runtime dependencies in the Sage developer's guide.