Skip to content

Commit

Permalink
remove wheel from requirements.txt (#10)
Browse files Browse the repository at this point in the history
  • Loading branch information
lalo authored Sep 16, 2024
1 parent c6e6cab commit 910aee1
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 5 deletions.
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,10 @@ To get started, clone this repository to your local machine and navigate to the
### 📦 Generate wheel package to share with others
1. *activate venv*
2. Update version inside setup.py if needed.
3. ```python setup.py bdist_wheel```
4. Fetch from dir dist/ the .whl
5. This file can be installed via `pip install eureka_ml_insights.whl`
3. Install wheel package via ```pip install wheel```
4. ```python setup.py bdist_wheel```
5. Fetch from dir dist/ the .whl
6. This file can be installed via `pip install eureka_ml_insights.whl`

## 🚀 Quick start
To reproduce the results of a pre-defined experiment pipeline, you can run the following command:
Expand Down Expand Up @@ -148,4 +149,4 @@ A cross-cutting dimension for all capability evaluations is the evaluation of se

A general rising concern on responsible AI evaluations is that there is a quick turnaround between new benchmarks being released and then included in content safety filters or in post training datasets. Because of this, scores on benchmarks focused on responsible and safe deployment may appear to be unusually high for most capable models. While the quick reaction is a positive development, from an evaluation and understanding perspective, the high scores indicate that the benchmarks are not sensitive enough to capture differences in alignment and safety processes followed for different models. At the same time, it is also the case that fielding thresholds for responsible AI measurements can be inherently higher and as such these evaluations will require a different interpretation lens. For example, a 5 percent error rate in instruction following for content length should not be weighed in the same way as a 5 percent error rate in detecting toxic content, or even a 5 percent success rates in jailbreak attacks. Therefore, successful and timely evaluations to this end depend on collaborative efforts that integrate red teaming, quantified evaluations, and human studies in the context of real-world applications.

Finally, Eureka and the set of associated benchmarks are only the initial snapshot of an effort that aims at reliably measuring progress in AI. Our team is excited about further collaborations with the open-source community and research, with the goal of sharing and extending current measurements for new capabilities and models. Our current roadmap involves enriching Eureka with more measurements around planning, reasoning, fairness, reliability and safety, and advanced multimodal capabilities for video and audio.
Finally, Eureka and the set of associated benchmarks are only the initial snapshot of an effort that aims at reliably measuring progress in AI. Our team is excited about further collaborations with the open-source community and research, with the goal of sharing and extending current measurements for new capabilities and models. Our current roadmap involves enriching Eureka with more measurements around planning, reasoning, fairness, reliability and safety, and advanced multimodal capabilities for video and audio.
1 change: 0 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@ isort==5.9.3
black==24.3.0
autoflake==1.7.5
mypy==1.10.1
wheel

0 comments on commit 910aee1

Please sign in to comment.