Skip to content

feat: add versioned artifact DAG reconciliation for #1679 - #2360

Draft
Even1115 wants to merge 1 commit into
Tencent:mainfrom
Even1115:codex/issue-1679-artifact-dag
Draft

feat: add versioned artifact DAG reconciliation for #1679#2360
Even1115 wants to merge 1 commit into
Tencent:mainfrom
Even1115:codex/issue-1679-artifact-dag

Conversation

@Even1115

Copy link
Copy Markdown

Summary

  • add a versioned, content-addressed processing artifact store with canonical keys, payload validation, checksums, fail-open recovery, singleflight, and Redis-backed leases
  • cache deterministic DocReader, embedding, chat, and VLM processing stages while keeping live, streaming, query, tool, and multimodal operations out of cache
  • introduce stable chunk and derived-artifact identities, attempt fencing, and desired-state reconciliation so reprocessing publishes new output before exact stale cleanup
  • track Neo4j contributions per stable chunk and processing attempt for precise graph reconciliation
  • add PostgreSQL/SQLite migrations plus MySQL and ParadeDB initialization schemas

Why

Knowledge reprocessing could repeat expensive provider calls, generate unstable identifiers, expose delete-first availability windows, and allow stale attempts to overwrite or remove newer output. This change models reusable intermediate results as versioned artifacts and makes publication generation-aware and non-destructive.

Impact

  • unchanged processing inputs can reuse verified artifacts across attempts
  • chunk and derived-output IDs remain stable across insertion and reordering
  • provider failures preserve the last published output
  • successful empty output is distinguished from failure and can reconcile stale state exactly
  • stale attempts are fenced before provider work and publication

Validation

Passed before publishing this draft:

  • artifact key, codec, identity, lease, runtime, fence, and reconciliation unit tests
  • repository processing-artifact and concurrent attempt-fence tests
  • service stable-ID, image prompt, table, and reconciliation tests
  • SQLite migration tests
  • container and Neo4j repository compile checks
  • git diff --check

Still pending before marking ready for review:

  • final full regression and race-test pass after the local machine restart
  • live Neo4j integration verification
  • live PostgreSQL/MySQL migration verification

Related to #1679.

@Even1115
Even1115 force-pushed the codex/issue-1679-artifact-dag branch from 37871db to 33d0a5f Compare July 28, 2026 01:19
@Even1115
Even1115 force-pushed the codex/issue-1679-artifact-dag branch from 33d0a5f to a11cd1a Compare July 28, 2026 01:35
@Even1115 Even1115 changed the title feat: add versioned artifact DAG reconciliation feat: add versioned artifact DAG reconciliation for #1679 Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant