Proposal: Transition to Single Table BigQuery Coverage

Overview

Currently, the coverage merger collects data from per-namespace BigQuery tables (syzbot_coverage.<namespace>). We want to transition to using single tables instead, to simplify management and enable coverage for all namespaces.
We propose using two shared tables:

1. syzbot_coverage.public: For public kernel namespaces.
2. <...>
   These tables will have an additional namespace column to distinguish the data.

Cost vs. Maintenance Trade-off

• Data Retrieval Cost: Querying shared tables will be slightly more expensive because we will be scanning larger partitions (which now contain data for multiple namespaces).
• Maintenance Benefit: This change significantly reduces maintenance overhead. We will manage only 2 tables instead of $N$ tables (where $N$ grows with every new namespace). Onboarding new namespaces will no longer require creating new BigQuery tables or configuring new GCS-to-BQ data transfers.
• Mitigation: To minimize the cost increase, we are making namespace the primary clustering key in the new tables. This allows BigQuery to prune blocks and scan only the data relevant to the target namespace.

Overall, the reduction in maintenance cost and ease of onboarding new namespaces outweighs the minor increase in BigQuery query costs.

Proposed Technical Changes

To support a gradual, namespace-by-namespace transition, we propose the following changes:

1. Schema Update (pkg/coveragedb/bq-schema.json): Add a required namespace column.
2. syz-ci Updates:
   • Update syz-ci to include the namespace in the raw coverage JSONL files uploaded to GCS.
3. Merger & Tool Updates (pkg/covermerger, tools/syz-covermerger, tools/syz-bq.sh):
   • Update InitNsRecords to accept an optional tableName.
   • If tableName is provided, query that table and filter by namespace.
   • If empty (fallback/transition state), query the old per-namespace table?
   • Update syz-bq.sh to accept -e <table_name> and conditionally query either the table directly (new) or INFORMATION_SCHEMA.PARTITIONS (old).
   • Get rid of syz-bq.sh and rewrite it in golang?
4. Dashboard Updates (dashboard/app):
   • Add UseSingleTable flag to CoverageConfig.
   • Add logic to route queries to new tables based on namespace access level if UseSingleTable is enabled.

Clustering Key Optimization

For the new tables, we propose changing the clustering keys:

• Current clustering: file_path, kernel_commit, hit_count
• Proposed clustering: namespace, file_path, kernel_commit

Proposed Rollout Plan

To ensure zero data loss and minimal disruption, we propose the following phased rollout:

Phase 1: Preparation (Backward Compatible)

1. Alter Old Tables: Add namespace column (optional) to all existing per-namespace tables.
2. Add the namespace discovery API: It is a prerequisite for the syz-ci changes.
   • dashboard, pkg: add client_info API and namespace schema field #7605
3. Deploy syz-ci: Deploy updated syz-ci. It will start uploading new format (with namespace), which old tables will accept.
   • syz-ci: query manager namespace from dashboard #7618
4. Deploy Dashboard (Inactive): Deploy dashboard code changes with UseSingleTable: false to verify old path stability.
5. Create New Tables: Create public and XXX tables with namespace as REQUIRED, clustered by namespace, file_path, kernel_commit.### Phase 2: Migration/Switchover1. Pause Transfers: Temporarily pause BigQuery Data Transfers.2. Migrate History: Copy historical data from old tables to new tables, backfilling the namespace column (e.g., COALESCE(namespace, 'upstream')).
6. Update & Resume Transfers: Point Data Transfers to new tables and resume them.
7. Enable Dashboard: Set UseSingleTable: true in dashboard config for transitioned namespaces and deploy.

Phase 3: Cleanup

• After 1-2 quarters of stability, delete old per-namespace tables and remove legacy fallback code.

Rollback Plan

If issues arise after switchover, we can rollback by:

1. Reverting dashboard config (UseSingleTable: false).
2. Pointing Data Transfers back to old tables.