New development workflow

[blythed]

[blythed]

This will make it easier for open-source contributors to contribute, without forcing them to get their hands too dirty.

[fnikolai]

We also need to consider this race condition when we make a new release.

[jieguangzhou]

I think we only need one main branch main. All feature fixes can be merged into the main branch. It should also include unit tests and integration tests to ensure the main branch is free of fundamental errors (CI passes). Otherwise, individual PR errors could lead to difficulties in CI maintenance later on.

Reasons:

• For new developers, only contributions merged into main (the default branch) will appear in the contributors' list. This is a GitHub mechanism. If we merge into dev, it will reduce the visibility of their contributions.
• When releasing, we only need to checkout a release branch from a specific commit in the main branch, or we can directly proceed with CI/CD for release.
• When encountering incompatible features (such as a predict refactor), since we are in the early stages, we can directly release a y version (x.y.z). As our project becomes more stable, we can fix bugs in z, release minor versions in y, and major versions in x.
• If the released version needs a bugfix and is compatible with main, fix it on main and cherry-pick it to the new z version. If incompatible, fix it directly in the z version.
• For some bug fixes, we currently fix and accumulate them until an appropriate time, and suggest releasing a z version directly to speed up iteration.
• The CI time is within 10 minutes, which generally isn't too long for developers. I think it's within an acceptable range, and there's no need to split CI across multiple branches.

In short, keeping just one main branch is sufficient.

[blythed]

Ok, so one of my major concerns is how difficult it is to get the CI passing.

• docker environment
• integration tests
• mypy checking
• (potential more checks we should really be doing, like checking the notebook examples)

@jieguangzhou do you have suggestions how to make it easier for contributors?

[jieguangzhou]

Ok, so one of my major concerns is how difficult it is to get the CI passing.

• docker environment
• integration tests
• mypy checking
• (potential more checks we should really be doing, like checking the notebook examples)

@jieguangzhou do you have suggestions how to make it easier for contributors?

Some suggestions @blythed

• docker environment

• Create a base image to use the base environment installation, except for the superduperdb package and requirements
• Unless the requirements change, the image is not rebuilt, just only copy directory again.

• integration tests

There is no special suggestion, but I still think that every PR needs to be integrated tested, otherwise there will be big trouble in the future.

mypy checking

This can provide error guidance, or the contributor documentation can guide users to use pre-commit.

More tests

Added to nightly check, add slack bot warning

[fnikolai]

@blythed The CI is nothing more than a "trigger" to Makefile actions.

Since everything is already there, we could create two additional actions that will run parts of the CI within your local machine:

make precommit-check that includes :

1. Linters (ruff, mypy, etc)
2. Unit testing

make precommit-check-all that includes :

1. Linters (ruff, mypy, etc)
2. Unit testing
3. Integration testing