GPTCache doesn't seem to work with LlamaIndex implementation

[ashwinr1980]

I have a built a question answering app over my organization's data using LlamaIndex and OpenAI. I'm trying to use GPTCache with that, but it doesn't look like it's being used. Here' a snippet of code for an exact match test

    cache.init()
    cache.set_openai_key()

    QA_PROMPT_TMPL = (
        "Context:\n"
        "---------------------\n"
        "{context_str}"
        "\n---------------------\n"
        "Question:\n"
        "{query_str}"
        "\n---------------------\n"
        "Instructions:\n"
        "Please provide an answer to the question that is truthful and directly related to the information presented in the context above."
        "If the answer is not in the context, please respond with \"I don't know\".\n"
    )

    QA_PROMPT = QuestionAnswerPrompt(QA_PROMPT_TMPL)

    start_time = time.time()
    # index is a GPTSimpleVectorIndex object provided by LlamaIndex
    response = index.query(prompt,
                           text_qa_template=QA_PROMPT,
                           )
    end_time = time.time()

    print("Time taken to generate response: ", end_time - start_time)`

The response times are as below (in seconds)

Time taken to generate response:  9.692034006118774 (first call)

Time taken to generate response:  10.240139961242676 (second call)

Do I need to do something different? Just for reference I am using the text-davinci-003 model.

[SimFG]

Can you show me the import codes

[ashwinr1980]

from llama_index import GPTSimpleVectorIndex, QuestionAnswerPrompt
from langchain import PromptTemplate, LLMChain
from langchain.llms import OpenAI
from gptcache import cache
from gptcache.adapter import openai

One thing I realized is that index.query might not be using the gptcache's openai implementation.

[SimFG]

If you use the openai by the langchain, you can set the GPTCache to the langchain.llm_cache, like:

from gptcache import Cache
from gptcache.adapter.api import init_similar_cache
from langchain.cache import GPTCache

def init_gptcache(cache_obj: Cache, llm: str):
    init_similar_cache(cache_obj=cache_obj, data_dir=f"similar_cache_{llm}")

langchain.llm_cache = GPTCache(init_gptcache)

[ashwinr1980]

Thanks SimFG. What is llm str?

def init_gptcache(cache_obj: Cache, llm str):

That doesn't compile.

[SimFG]

sorry, it's a typo, it should be:

def init_gptcache(cache_obj: Cache, llm: str):

[ashwinr1980]

Oh, definitely update the docs. (or I am happy to do it). The same code is on the langchain docs as well.

Btw, I dug into the logs and found that the data is not being persisted to the sqllite db

Am I missing any other depedencies?

[SimFG]

Can you give a demo code and gptcache version? From the error stack, i guess the param usage problem.

[SimFG]

or you can reference the langchain doc, link: https://python.langchain.com/en/latest/modules/models/llms/examples/llm_caching.html

[ashwinr1980]

My suspicion is that GPTCache is not completely compatible with LlamaIndex. I'm digging into the LlamaIndex code and then the adapt function in adapter.py

This is the code snippet where the error is coming from in adapter.py

    if cache_enable:
        try:

            def update_cache_func(handled_llm_data, question=None):
                if question is None:
                    question = pre_store_data
                else:
                    question.content = pre_store_data
                chat_cache.data_manager.save(
                    question,
                    handled_llm_data,
                    embedding_data,
                    extra_param=context.get("save_func", None),
                    session=session,
                )

            llm_data = update_cache_callback(llm_data, update_cache_func, *args, **kwargs)
        except Exception as e:  # pylint: disable=W0703
            logging.warning("failed to save the data to cache, error: %s", e)
    return llm_data

[SimFG]

yes, maybe you need to change the pre_embendding_func, because the LlamaIndex isn't integrated by the GPTCache.

[SimFG]

If it's possible, please give a full demo code, and i will try to integrated the LlamaIndex

[ashwinr1980]

I’ll figure it out and post it here

…

On Thu, May 18, 2023 at 8:07 PM SimFG ***@***.***> wrote: If it's possible, please give a full demo code, and i will try to integrated the LlamaIndex — Reply to this email directly, view it on GitHub <#361 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL4L2Q5UXCAAOBDVD7HL6B3XGYXZ7ANCNFSM6AAAAAAYF6GHP4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[SimFG]

@ashwinr1980 Hello, I write a webpage qa example by using the GPTCache and llama_index, reference: https://gptcache.readthedocs.io/en/latest/bootcamp/llama_index/webpage_qa.html

[Towhid1]

NameError: name 'GPTCache' is not defined

[sireesha303]

NameError: name 'GPTCache' is not defined

[ashwinr1980]

This is very helpful. I'll try it today. One thing I do in my app is to save the index to disk, so that I can load the index json from my local folder to save time (don't have to recreate index everytime)

So the first time I do this

index = GPTSimpleVectorIndex.from_documents(documents, service_context=service_context)
index.save_to_disk('./sample_data/latet_index.json')

And in subsequent calls I just do this

index = GPTSimpleVectorIndex.load_from_disk("/Users/ashwin/Documents/Dev/chatbot/static/latest_index.json")

I'm going to try and if I can insert the cache in here somewhere.