Slack sync: tighter safety-backfill + stale-source warner - #817
Open
henry-dowling wants to merge 1 commit into
Open
Slack sync: tighter safety-backfill + stale-source warner#817henry-dowling wants to merge 1 commit into
henry-dowling wants to merge 1 commit into
Conversation
The Slack Events-API webhook is the live ingestion path; a plain 6-hour backfill was the only fallback. When the webhook stopped delivering, a channel could sit up to 6h stale with no signal — and a second Slack source that had been failing every run went unnoticed for a month. - Drop the Slack safety-backfill interval from 21600s (6h) to 1800s so worst-case staleness when the webhook is down is minutes, not hours. - Add source_service.stale_sources() + a hourly warn_stale_sources beat task that logs any enabled source that is failing, never synced, or last succeeded more than 3 intervals ago, so a dead token / wedged reconciler surfaces instead of rotting silently. Test: pytest backend/tests/test_sources.py (89 passed; one unrelated pre-existing posthog-cleanup failure). ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Our Slack integration went stale and I traced it in prod (Neon
stash-prod+ Render logs). Freshness has two paths: the live Events-API webhook (/api/v1/integrations/slack/events→ingest_slack_event) and a periodic safety backfill (index_slack). Today the webhook delivered zero events (Slack isn't POSTing us — a separate config fix in the Slack app console), so the only thing keeping data fresh was a 6-hour backfill. A message posted at 17:47 UTC didn't land until the next backfill window — it was invisible for hours.Separately, a duplicate Slack source (owner sam@) had been failing on every run since June 12 (
RuntimeError: no allowed channels configured) with no signal anywhere.This PR doesn't fix the webhook (that's Slack-app config), but it makes the fallback tight and makes silent staleness visible.
Changes
DEFAULT_SYNC_INTERVAL_S["slack"]21600s (6h) → 1800s (30m). When the webhook is down, worst-case staleness is minutes, not a quarter-day. Cost is ~1conversations.historycall per allowlisted channel per 30m — comfortably within Slack limits.source_service.stale_sources()+ hourlywarn_stale_sourcesbeat task. Logs a WARNING for anysync_enabledsource that is currently failing (sync_errorset), never completed (last_synced_at IS NULL), or last succeeded more than 3 intervals ago. The month-dead Slack source would have surfaced on day one.Scope / limitations (deliberate)
Test
pytest backend/tests/test_sources.py→ 89 passed. Newtest_stale_sources_flags_failing_and_overduecovers failing/overdue/fresh. One failure (test_provider_cleanup_removes_source_and_retained_rows[posthog-posthog_project]) is pre-existing and unrelated — it fails identically on the base commit with my changes stashed.ruff check/ruff format --checkclean.Backend-only; no GUI changes.
🤖 Generated with Claude Code