Various issues and slow performance with Cognify

[LStromann]

=Hi all,

I've been really excited about Cognee - it seems like the perfect knowledge graph architecture. I'm trying to use it to set up a 'coworker agent' slackbot with memory pulled from Notion, Slack, and Github, but unfortunately hit several roadblocks in the cognify step. I'm using Temporal mode since I want to prioritize more recent sources, but saw similar issues in base cognify.

My stack is: postgres, pgvector, and neo4j, and currently just running locally on a macbook with M4 chip. Tried both haiku-4-5 + openai embedding large, and then switched to gemini-embedding-001 / gemini-flash-latest for faster performance.

I'm trying to load about 3000 files (32MB) into a single dataset in Cognee.

1. I saw various blocking errors during Cognify, including:

   • trying to embed too large of a string, e.g. a really long URL in my source docs
   • maxxing out token output
   • postgres and neo4j deadlocks
   • rate limiting - seems like I'm able to control rate limiting of the main LLM call, but not the embedding LLM? Also - I'm only able to control the req/min, but not tokens/min
   • The SummarizedContent schema allows at most one item, but the model produced two tool-use blocks

2. It seems like if a file fails to Cognify, it blocks all other files. So loading all my files at once, and then Cognifying at the end didn't work because Cognify would crash, and there wasn't an easy way to tell which file(s) had caused the crash. I switched to an approach of adding+cognifying 1 file at a time, skipping the file if Cognify crashed, but that was very slow.

3. It's pretty hard to understand and keep track of all of the errors + logs. It'd be really nice if there was a 'per source file' view where I could see overall: how many files succeeded/failed, and click into logs per file, and retry cognifying per file.

4. Cognify seemed unworkably slow for us. It seemed like it'd take 40+ hours to process our initial data.

Thanks and still really excited for Cognee's future!

This discussion was automatically pulled from Discord.

[LStromann]

Hey Hazel, thanks for the detailed feedback really appreciate you sharing all this context.

[LStromann]

Regarding 2. and 3. There is an option to continue if a file processing failed:

RAISE_INCREMENTAL_LOADING_ERRORS=False

It is on by default, but can be set to continue if an error happens

In the end it returns the status of all the files processed which succeeded and which failed and why

[LStromann]

4. A lot of the speed of cognify depends on the LLM models used, try using a faster LLM model to get a performance and speed boost (the default gpt-5-mini while giving good results is a bit slow)

[LStromann]

Regarding 1. Can you open new help issues with more details regrading each of the issues

It will help us keep track and resolve them