Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate.

I spent some tokens on analyzing the issue in the hope that it will help you pinning down the issue. If you wish, I can also dive into fixing them, but that would require some synchronisation of efforts.

Root cause analysis — ranked by severity × probability

1. CRITICAL / HIGH PROBABILITY — Uninitialised repos are never re-enqueued on subsequent requests

Where: scheduler.py, specifically the schedule() method's decision tree.

The bug: When a repo already exists in the DB (repository is not None) but has never been successfully checked (last_access is None, hash is None), the only branch that would re-enqueue it is:

elif repository.hash != latest:

But repository.hash is None (never set) and latest is a real hash string. In Python, None != "abc123..." is True, so this does re-enqueue. That part is OK by accident.

However, if the first check failed (SSH error 255, timeout, empty output, or JSONDecodeError), the DB row already exists with hash=None, the task is completed and removed from the in-memory queue via done() at scheduler.py, but the DB is never updated. On the next request, repository.hash (None) != latest evaluates to True, so it should re-enqueue. This path does work — but only if someone actually visits /info/ or /status/ again.

The real problem: if nobody visits those pages again after the first failure, the repo stays uninitialised forever. There is no retry mechanism, no background sweep, no watchdog. This is the most likely root cause for repos stuck "after weeks" — the initial check failed silently, and nobody re-triggered it.

Fix direction: Add a lightweight periodic sweep (e.g. a background thread or timer) that queries the DB for rows where last_access IS NULL and re-enqueues them. This is the single highest-impact change.

2. HIGH / HIGH PROBABILITY — TaskQueue is a process-wide singleton with class-level mutable state shared across gunicorn workers via fork()

Where: task.py

_instance = None
__urls: set[str] = set()
__urls_lock: Lock = Lock()

These are class variables. TaskQueue uses a singleton pattern via __new__. With gunicorn --workers=4, gunicorn forks after create_app() is called. At fork time:

• The _instance, __urls set, and __urls_lock are copied into each worker.
• Each worker gets its own independent TaskQueue with its own __urls set.
• Each worker also gets its own Scheduler with its own Runner threads.

This means:

• A task enqueued in Worker A is invisible to Workers B, C, D.
• If Worker A creates a DB row and enqueues a task, but the next request from the same user lands on Worker B, Worker B sees the DB row (repository is not None) but the task is not in Worker B's queue, so it falls through to the hash != latest check. This can cause duplicate concurrent checks of the same repo across workers.
• If a Runner thread dies (see item 3), only that worker loses capacity — no other worker compensates.

But more critically for the bug: after a gunicorn worker is recycled (which gunicorn does periodically via max_requests or on crash), the in-memory queue is lost. Any tasks that were in-flight or queued are silently dropped. The DB row exists but was never updated, so last_access remains None.

Fix direction: Either run with --workers=1 --threads=N (trading parallelism), use --preload and ensure the scheduler is started post-fork, or move to a durable queue (Redis, database table) shared across workers.

3. HIGH / MEDIUM PROBABILITY — Unhandled exceptions in Runner.run() silently kill worker threads

Where: scheduler.py

The try/except in Runner.run() only catches:

• subprocess.TimeoutExpired (line ~112)
• JSONDecodeError (line ~139)

Any other exception — e.g. AttributeError if Repository.find(self.url) returns None in task.py, a database connection error, an OSError from SSH, or an encoding error — will propagate up, exit the while self.__running loop, and kill the thread permanently. There is no respawn logic.

With NB_RUNNER=6 threads per worker process, losing threads over weeks of uptime is very plausible. Once all 6 are dead, no more tasks are ever processed in that worker, even though they're enqueued. The queue grows silently. No monitoring or health check exists.

Fix direction: Wrap the entire loop body in a broad except Exception with logging, so a single bad task can't kill a thread. Optionally add a health-check endpoint that reports the number of alive runner threads.

4. MEDIUM / HIGH PROBABILITY — SSH failure (exit code 255) leaves repo permanently uninitialised with no retry

Where: scheduler.py

When SSH returns 255, the code logs a warning and explicitly skips the DB update. The task is then marked done() and removed from the queue. The repo stays with hash=None and last_access=None.

If the SSH failure is transient (network blip, worker restart, connection limit), this is fine — the next request re-enqueues. But if it's persistent (misconfigured key, firewall rule, DNS issue), every attempt will fail with 255 and be silently discarded, with no escalation, no retry backoff, and no alert beyond a log warning.

Over weeks, if the API worker is temporarily unreachable (maintenance, restart), every repo checked during that window gets silently dropped and must wait for another user request to retry.

Fix direction: Re-enqueue tasks that fail with SSH 255 after a delay, with a retry counter. Or at minimum, don't call done() so the task stays in the queue (though this risks infinite retry storms without backoff).

5. MEDIUM / MEDIUM PROBABILITY — Empty output from linting silently leaves repo uninitialised

Where: scheduler.py

When output is empty, the code logs a warning but still passes the empty string to task.update_db(output). json_loads("") raises JSONDecodeError, which is caught at line 139 and logged — but the DB is not updated. The task is marked done. Same outcome as SSH 255: repo stays stuck.

Fix direction: Skip update_db() when output is empty, and consider re-enqueuing.

6. MEDIUM / LOW-MEDIUM PROBABILITY — Case-sensitivity mismatch between create() and find()

Where: models.py vs models.py

Repository.create() stores the URL as-is (case-preserving). Repository.find() uses db.func.lower() for case-insensitive lookup. But is_initialised() at models.py uses filter_by(url=url), which is case-sensitive.

If a user registers git.example.com/Org/Repo but later visits /info/git.example.com/org/repo, then:

• find() returns the row (case-insensitive match) → repository is not None
• schedule() sees hash is None, re-enqueues with the lowercase URL
• update_db() calls find(lowercase_url) → finds the row → updates it with the lowercase URL
• But is_initialised(original_case_url) uses filter_by(url=...) (case-sensitive) → returns False because the stored URL was changed to lowercase by the update

This can cause a repo to flip between "initialised" and "uninitialised" depending on the case of the URL in the request, and trigger redundant re-checks.

Fix direction: Make is_initialised() and is_compliant() use the same case-insensitive lookup as find().

7. LOW / MEDIUM PROBABILITY — is_registered() reads FORMS_FILE from disk on every request with no caching

Where: models.py

Every call to is_registered() opens and parses the entire JSON file. This is called from status() (for badges), info(), schedule() (via create()), etc. If the file is large or the filesystem is slow (network mount in Docker), this adds latency. More importantly, if the file is being written to by the forms app at the same moment (no locking on the read side), the read could get a partial/corrupt JSON and raise an unhandled exception that propagates up to the request handler — or worse, into a Runner thread if called from create() inside schedule().

Fix direction: Cache the file contents with a short TTL, or use FileLock on reads (matching the write side in json_store.py).

Summary table

Most likely root cause for the observed symptom ("newly registered projects stuck after weeks"): a combination of # 1 and # 3/# 4. The initial check fails (SSH issue, timeout, empty output, or unhandled exception), the runner thread either dies or discards the result, and no mechanism ever retries the check unless a user manually visits /info/ or /status/ again. Over weeks, runner threads may gradually die from unhandled exceptions (# 3), reducing throughput to zero in affected workers.

The badge requests should be treated differently since scrapers also trigger them and they account for a big amount of traffic.

In a discussion with @tobiasd we agreed that a higher-than-usual age threshold will suffice.

Also the caching of the data should be adjusted to equal our internal timeout

1. reset
2. query all

License file image.png.license (for image.png) is ignored as empty.
License information inside image.xml is ignored.

My hypothesis is that reuse-tool cannot read contents of Git LFS files and silently considers them empty, but it can read file sizes and doesn't filter them out.

(Originally reported from @80486DX2-66 on codeberg.org as issue 1343)

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.