Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About Exception raised in Job[]: TimeoutError() #1850

Open
erliang-sf opened this issue Jan 16, 2025 · 0 comments
Open

About Exception raised in Job[]: TimeoutError() #1850

erliang-sf opened this issue Jan 16, 2025 · 0 comments
Labels
bug Something isn't working module-metrics this is part of metrics module

Comments

@erliang-sf
Copy link

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When I use the evaluation function, the timeoutError is frequently displayed, and the final evaluation metrics all show NAN, I refer to the documentation to set up various runconfigs, but it has no effect

Ragas version:0.2.9
Python version:3.12

Code to Reproduce
faithfulness.llm = qwen_llm
answer_relevancy.llm = qwen_llm
answer_relevancy.embeddings = bge_m3
context_recall.llm = qwen_llm
context_precision.llm = qwen_llm

evalsets = load_test_data(uploaded_file)
if evalsets is not None:
    batch_size = 4
    final_df = pd.DataFrame()
    try:
        result = evaluate(
            evalsets,
            metrics=metrics,
            batch_size=batch_size,
            raise_exceptions=True,
            callbacks=[TestCallback()],
            run_config=RunConfig(timeout=60,max_retries=3, max_wait=60, max_workers=1)
        )
        final_df = result.to_pandas()
        print("final_df:", final_df.head())
    except Exception as e:
        print(f"评估时出错: {e}")

Error trace
Evaluating: 0%| | 0/8 [00:00<?, ?it/s]Exception raised in Job[0]: TimeoutError()
Evaluating: 25%|████████████████████▊ | 2/8 [02:03<06:10, 61.83s/it]Exception raised in Job[2]: TimeoutError()
Evaluating: 38%|███████████████████████████████▏ | 3/8 [03:03<05:04, 61.00s/it]Exception raised in Job[3]: TimeoutError()
Evaluating: 50%|█████████████████████████████████████████▌ | 4/8 [04:03<04:02, 60.61s/it]Exception raised in Job[4]: TimeoutError()
Evaluating: 62%|███████████████████████████████████████████████████▉ | 5/8 [05:03<03:01, 60.39s/it]Exception raised in Job[5]: TimeoutError()
Evaluating: 75%|██████████████████████████████████████████████████████████████▎ | 6/8 [06:03<02:00, 60.26s/it]Exception raised in Job[6]: TimeoutError()
Evaluating: 88%|████████████████████████████████████████████████████████████████████████▋ | 7/8 [07:03<01:00, 60.18s/it]Exception raised in Job[7]: TimeoutError()
Evaluating: 100%|███████████████████████████████████████████████████████████████████████████████████| 8/8 [08:03<00:00, 60.39s/it]
final_df: user_input retrieved_contexts ... context_precision context_recall
0 请详细介绍奥运会的历史与发展。 [奥运会,全称为奥林匹克运动会(Olympic Games),是世界上规模最大、影响最广的综... ...
NaN NaN
1 请详细介绍奥运会的历史与发展。 [奥运会,全称为奥林匹克运动会(Olympic Games),是世界上规模最大、影响最广的综... ...
NaN NaN

Expected behavior
This issue didn't come up at first, and I wondered if it was due to server load or other factors such as the network, or if there was a problem with the evaluation parameters I had configured. Thanks.

Additional context
Add any other context about the problem here.

@erliang-sf erliang-sf added the bug Something isn't working label Jan 16, 2025
@dosubot dosubot bot added the module-metrics this is part of metrics module label Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working module-metrics this is part of metrics module
Projects
None yet
Development

No branches or pull requests

1 participant