[Bug]: Moderation api is not working

[kishanios123]

Current Behavior

modOutputres = openai.Moderation.create(input=question)
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/openai.py", line 319, in create
res = adapt(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/adapter.py", line 39, in adapt
pre_embedding_res = chat_cache.pre_embedding_func(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/processor/pre.py", line 19, in last_content
return data.get("messages")[-1]["content"]
TypeError: 'NoneType' object is not subscriptable

Expected Behavior

it should give moderation api output.

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

[SimFG]

hi, @kishanios123
When you want to use the Moderation api, you should use the get_openai_moderation_input pre-process

[kishanios123]

hi, @kishanios123 When you want to use the Moderation api, you should use the get_openai_moderation_input pre-process

@SimFG Yes, correct. Sorry for creating an issue by mistake ... And thank you very much for considering adding Moderation api.

BTW I have one question, Is Moderation also using Caching mechanism or not? If yes how does it work? Like exact word match or some other mechanism?

[SimFG]

@kishanios123 yes, it's same as other api. The biggest advantage may be that it can relieve the pressure on the network. Of course, this api is free, and you can determine whether to use the cache according to the actual scenario.

[kishanios123]

@SimFG Thanks for your response.

I am confused with init_similar_cache & pre_func

init_similar_cache(pre_func=get_openai_moderation_input)

1. Does this line mean that it is added as a pre-function? do I need to call moderation api separately or not?
2. This does not create any issue with my main openai completion or chat api cache right?

I am mentioning my code below to show how i am using this... please tell me if a problem exists in my code regarding caching...

  from gptcache import cache, Config
  from gptcache.manager import manager_factory
  from gptcache.embedding import Onnx
  from gptcache.processor.post import temperature_softmax
  from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
  from gptcache.similarity_evaluation.onnx import OnnxModelEvaluation
  from gptcache.adapter import openai
  from gptcache.adapter.api import init_similar_cache
  from gptcache.processor.pre import get_openai_moderation_input
  
  os.environ['OPENAI_API_KEY'] = ''
  cache.set_openai_key()
  
  onnx = Onnx()
  data_manager = manager_factory("sqlite,faiss", vector_params={"dimension": onnx.dimension})
  
  cache.init(
      embedding_func=onnx.to_embeddings,
      data_manager=data_manager,
      similarity_evaluation=OnnxModelEvaluation(),
      post_process_messages_func=temperature_softmax
      )
  init_similar_cache(pre_func=get_openai_moderation_input)

    modOutput = openai.Moderation.create(input=question)["results"][0]["flagged"]
    if not modOutput:
        start = time.time()
        response = openai.ChatCompletion.create(model="gpt-3.5-turbo",temperature = 1.0,  # Change temperature here
            messages=[
                                                                                    {"role": "user", "content": "My Question"}],)
        print("Time elapsed:", round(time.time() - start, 3))
        print("moderation passed answer = " + response['choices'][0]['message']['content'] + " - token = " + str(response['usage']['total_tokens']), flush=True)

    else:
        print("moderation failed question = " + question, flush=True)
        answer = "Your question violates the policy. Please ask only relevant and appropriate questions."
        return redirect(url_for("index", result=answer))

[SimFG]

in each cache, the preprocessing method is different, because each llm request format is different. The following is my suggestion:

os.environ['OPENAI_API_KEY'] = ''
cache.set_openai_key()

llm_cache = Cache()
onnx = Onnx()
data_manager = manager_factory("sqlite,faiss", vector_params={"dimension": onnx.dimension})
llm_cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=OnnxModelEvaluation(),
    post_process_messages_func=temperature_softmax
)
# maybe you can use the SearchDistanceEvaluation, because it should fit a lot of scenes,
# if you say yes, try:
# llm_cache = Cache()
# init_similar_cache(data_dir="llm_cache", cache_obj=llm_cache)

moderation_cache = Cache()
init_similar_cache(data_dir="moderation_cache", cache_obj=moderation_cache, pre_func=get_openai_moderation_input)

modOutput = openai.Moderation.create(input=question, cache_obj=moderation_cache)["results"][0]["flagged"]
if not modOutput:
    start = time.time()
    response = openai.ChatCompletion.create(model="gpt-3.5-turbo", temperature=1.0,  # Change temperature here
                                            messages=[
                                                {"role": "user", "content": "My Question"}], cache_obj=llm_cache)
    print("Time elapsed:", round(time.time() - start, 3))
    print("moderation passed answer = " + response['choices'][0]['message']['content'] + " - token = " + str(
        response['usage']['total_tokens']), flush=True)

else:
    print("moderation failed question = " + question, flush=True)
    answer = "Your question violates the policy. Please ask only relevant and appropriate questions."
    return redirect(url_for("index", result=answer))

Note: OnnxModelEvaluation only support the 512 token.

[kishanios123]

@SimFG

          llm_cache = Cache() 
          moderation_cache = Cache()

I am using api env. (Flask), is this code good? Like Cache() create a new object every time right ... so on the next start of my flask server respective old cache used for both LLM and moderation or I need to do something else ?

  Note: OnnxModelEvaluation only support the 512 token.

Oh, I don't know. I have tried SearchDistanceEvaluation, but it fails in many simple cases.
Like it found this both sentences are similar "Tell me a story that contains Dog." & "Tell me a story that does not contain Dog."
Can you suggest any other good option?

[SimFG]

When you start the server, you go to init the cache, which is no problem. In the next start, the cache data will restore from the cache directory.
About the next question, i will reply you lately, because i will checkout it.

[kishanios123]

@SimFG
Deleting issue message, I found mail that my api key is disabled, I have posted it here in the previous comment by mistake.

It's working well now.

Waiting for your suggestion on the similarity algorithm ...

Thanks again for your support.

[kishanios123]

@SimFG

I have found one more issue.

I want to use Exact Match Evaluation in Moderation API.

I don't know how to do this?

I have tried something like this but getting this error "TypeError: init() got an unexpected keyword argument 'pre_func'"

moderation_data_manager = manager_factory(data_dir="moderation_cache")
moderation_cache.init(
    data_manager=data_manager,
    pre_func=get_openai_moderation_input
)

I can not find "init_exact_cache" function similar to "init_similar_cache" ...

[SimFG]

change pre_func to pre_embedding_func

moderation_cache.init(
    data_manager=data_manager,
    pre_embedding_func=get_openai_moderation_input
)

[wxywb]

@SimFG

          llm_cache = Cache() 
          moderation_cache = Cache()

I am using api env. (Flask), is this code good? Like Cache() create a new object every time right ... so on the next start of my flask server respective old cache used for both LLM and moderation or I need to do something else ?

  Note: OnnxModelEvaluation only support the 512 token.

Oh, I don't know. I have tried SearchDistanceEvaluation, but it fails in many simple cases. Like it found this both sentences are similar "Tell me a story that contains Dog." & "Tell me a story that does not contain Dog." Can you suggest any other good option?

In this particular example, the two sentences are similar in structure, with the only difference being the negation in the second sentence. As a result, the embeddings of these sentences could be close in the embedding space, reflecting the shared context and structure.
It's important to note that embeddings alone may not capture the full semantics or meaning of the sentences. While the embeddings can indicate some degree of similarity, they may not capture the contradictory nature of the statements.

[SimFG]

@kishanios123 a possible way of the wrong cache answer, link: #388

[kishanios123]

While the embeddings can indicate some degree of similarity, they may not capture the contradictory nature of the statements.

@SimFG Got your point.

I am not an expert in this but I think ONXX manages it well right? Because it is ML Model? It is understanding the negation part?

[SimFG]

@kishanios123 yes, but due to limited training data, it has no way to support a large token.

[xiaofan-luan]

Anything we can do on the evaluation part?

[kishanios123]

Getting this error, can someone help ?

modRes = openai.Moderation.create(input=question, cache_obj=moderation_cache)
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/openai.py", line 340, in create
res = adapt(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/adapter.py", line 76, in adapt
search_data_list = time_cal(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/utils/time.py", line 9, in inner
res = func(*args, **kwargs)
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/manager/data_manager.py", line 321, in search
embedding_data = normalize(embedding_data)
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/manager/data_manager.py", line 172, in normalize
magnitude = np.linalg.norm(vec)
File "<array_function internals>", line 180, in norm
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 2511, in norm
x = x.astype(float)
ValueError: could not convert string to float: 'hi'

[SimFG]

@kishanios123 What embedding function did you use? I guess you should not set embedding.

[kishanios123]

@SimFG ... but it is working before few weeks ....i have followed this code - #386 (comment)

moderation_cache = Cache()
moderation_data_manager = manager_factory(data_dir="moderation_cache")
moderation_cache.init(
data_manager=data_manager,
pre_embedding_func=get_openai_moderation_input
)

[SimFG]

[SimFG]

maybe you should use the moderation_data_manager

[kishanios123]

@SimFG

Thanks for providing a solution, it works ...

maybe you should use the moderation_data_manager

[kishanios123]

@SimFG

I am getting this warning. does it cause any issues in the future?

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:

• Avoid using tokenizers before the fork if possible
• Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
  huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
  To disable this warning, you can either:
• Avoid using tokenizers before the fork if possible
• Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

I am using moderation cache like this :
moderation_cache = Cache()
moderation_data_manager = manager_factory(data_dir="moderation_cache")
moderation_cache.init(
data_manager=moderation_data_manager,
pre_embedding_func=get_openai_moderation_input
)

if i change

pre_embedding_func=get_openai_moderation_input

to

pre_func=get_openai_moderation_input

then warning is not showing ...

[kishanios123]

I got another error ...

File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/openai.py", line 100, in create
return adapt(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/adapter.py", line 172, in adapt
return_message = time_cal(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/utils/time.py", line 9, in inner
res = func(*args, **kwargs)
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/adapter/adapter.py", line 161, in post_process
return_message = chat_cache.post_process_messages_func(
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/processor/post.py", line 84, in temperature_softmax
scores = softmax([x / temperature for x in scores])
File "/opt/anaconda3/envs/openai/lib/python3.9/site-packages/gptcache/utils/softmax.py", line 6, in softmax
assert len(x.shape) == 1, f"Expect to get a shape of (len,) but got {x.shape}."
AssertionError: Expect to get a shape of (len,) but got (1, 1).

[SimFG]

161

it's just warning

[SimFG]

For the error, can you give a testing demo

[kishanios123]

For the error, can you give a testing demo

import os
import time

#import openai as origOpenai
from gptcache import cache, Config
from gptcache.core import Cache
from gptcache.manager import manager_factory
from gptcache.embedding import Onnx
from gptcache.processor.post import temperature_softmax
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
from gptcache.similarity_evaluation.onnx import OnnxModelEvaluation
from gptcache.adapter import openai
from gptcache.adapter.api import init_similar_cache
from gptcache.processor.pre import get_openai_moderation_input

cacheTemperature = 1.0
sysPrompt = ""
openaiKey = ''

app = Flask(__name__)

os.environ['OPENAI_API_KEY'] = openaiKey
#origOpenai.api_key = openai.api_key = os.getenv("OPENAI_API_KEY")
cache.set_openai_key()

llm_cache = Cache()
onnx = Onnx()
data_manager = manager_factory("sqlite,faiss", data_dir="llm_cache", vector_params={"dimension": onnx.dimension})
llm_cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=OnnxModelEvaluation(),
    post_process_messages_func=temperature_softmax
)

moderation_cache = Cache()
moderation_data_manager = manager_factory(data_dir="moderation_cache")
moderation_cache.init(
    data_manager=moderation_data_manager,
    pre_embedding_func=get_openai_moderation_input
)

        question = str()
        answer = getOepnAiResponse(question)
        return redirect(url_for("index", result=answer))
    
    question = ""
    answer = getOepnAiResponse(question)

def getOepnAiResponse(question):
    start = time.time()
    if not question.strip():
        print("empty question" + " ,,, " + "empty answer" + " ,,, " + str(round(time.time() - start, 2)) + " ,,, " + "0", flush=True)
        answer = "No input received. Please enter a valid input and try again."
        return answer

    try:
        modRes = openai.Moderation.create(input=question, cache_obj=moderation_cache)
        print(str(modRes), flush=True)
        modOutput = modRes["results"][0]["flagged"]
    except Exception as e:
        print("got error once - " + str(e), flush=True)
        modRes = openai.Moderation.create(input=question, cache_obj=moderation_cache)
        modOutput = modRes["results"][0]["flagged"]
        print("after error modOutput = " + str(modOutput), flush=True)
    
    if not modOutput:
        response = openai.ChatCompletion.create(model="gpt-3.5-turbo",temperature = cacheTemperature,  # Change temperature here
            messages=[{"role": "system", "content":sysPrompt},
            {"role": "user", "content": question}],cache_obj=llm_cache)
        answer = response['choices'][0]['message']['content']
        token = str(response['usage']['total_tokens'])
        print(question + " ,,, " + answer + " ,,, " + str(round(time.time() - start, 2)) + " ,,, " + token, flush=True)
        return answer
    else:
        print(question + " ,,, " + "moderation failed" + " ,,, " + str(round(time.time() - start, 2)) + " ,,, " + "0", flush=True)
        answer = "Your question violates the policy. Please ask only relevant and appropriate questions."
        return answer

[kishanios123]

sometimes error comes, and some time working well...

[SimFG]

I try to run the below example and don't reproduce the error:

from gptcache import Cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import manager_factory
from gptcache.processor.post import temperature_softmax
from gptcache.similarity_evaluation import OnnxModelEvaluation

llm_cache = Cache()
onnx = Onnx()
data_manager = manager_factory(
    "sqlite,faiss", data_dir="llm_cache", vector_params={"dimension": onnx.dimension}
)
llm_cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=OnnxModelEvaluation(),
    post_process_messages_func=temperature_softmax,
)

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    temperature=1.2,  # Change temperature here
    messages=[{"role": "user", "content": "My Question"}],
    cache_obj=llm_cache,
)

pprint(response)

[kishanios123]

please try on different inputs ... I found it many times...

[SimFG]

From the source code, this array dimension seems unlikely to be wrong.

[kishanios123]

From the source code, this array dimension seems unlikely to be wrong.

in which case this issue can happen? i am getting this issue many times now, don't know why. I have updated the GPTCache pip library.

[kishanios123]

it's not about updates. I tried now in GPTCache-0.1.26, getting the same error on the first run.

[SimFG]

I am not very clear when this will happen. The above is the output of my separate test softmax function. If temperature_softmax is used, the error is not this one.

[SimFG]

[x / temperature for x in scores]

Through the above code, it is impossible to get a 1*1 two-dimensional array, only a one-dimensional array. If the score is set to a two-dimensional array, this operation will report an error

[SimFG]

the latest version is 0.1.32, maybe you can try it. From the latest source code, I think this error should not exist

[kishanios123]

i received exception initially in 0.1.32, then i downgraded to 0.1.26. getting the exception in both versions.

the latest version is 0.1.32, maybe you can try it. From the latest source code, I think this error should not exist