Improve readability of the quick tour. #501
Conversation
|
Actually, I'm not sure running 2 models at the same time is true. When I run it only evaluates the second model. It seems the code doc in is not correct? |
docs/source/quicktour.mdx
Outdated
|
|
||
| Tasks details can be found in the | ||
| All supported tasks can be found at the [tasks_list](available-tasks). For more details, you can have a look at the |
There was a problem hiding this comment.
We also support the tasks that are community provided in the extended folder
docs/source/quicktour.mdx
Outdated
| the [tasks_list](available-tasks) in the format: | ||
| Here, the first argument specifies which model(s) to run, and the second argument specifies how to evaluate them. | ||
|
|
||
| Multiple models can be evaluated at the same time by using a comma-separated list. For example: |
There was a problem hiding this comment.
Nope, we can only evaluate one model at a time - however we can specifiy precision, peft weights, ...
docs/source/quicktour.mdx
Outdated
| The task specification might be a bit hard to grasp as first. The format is as follows: | ||
|
|
||
| ```bash | ||
| {suite}|{task}|{num_few_shot}|{0 or 1 to automatically reduce `num_few_shot` if prompt is too long} |
There was a problem hiding this comment.
automatically adapt the number of few shot examples presented to the model if the prompt is too long for the context size of the task or the model
There was a problem hiding this comment.
(I would add this explanation on antoher line)
docs/source/quicktour.mdx
Outdated
| The syntax for the task specification might be a bit hard to grasp at first. The format is as follows: | ||
|
|
||
| ```txt | ||
| {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a reduction} |
There was a problem hiding this comment.
| {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a reduction} | |
| {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a truncation if context size is too small} |
docs/source/quicktour.mdx
Outdated
| Tasks details can be found in the | ||
| All officially supported tasks can be found at the [tasks_list](available-tasks). | ||
| Moreover, community-provided tasks can be found in the | ||
| [extended folder](https://github.com/huggingface/lighteval/tree/main/src/lighteval/tasks/extended) and the |
There was a problem hiding this comment.
Extended are not community provided but maintainer provided, however they are tasks which require added logic (such as an LLM as judge, or redefine new metrics like IFEval)
No description provided.