Skip to content

Commit

Permalink
UI update (#273)
Browse files Browse the repository at this point in the history
* Update langchain requirement from <0.3.0,>=0.2.5 to >=0.2.5,<0.4.0 (#236)

* [Automated] Merge release into main (#235)

* bump the version, test release to PyPi

* Update README.md

* Update README.md

* Update README.md

* bumpy version to 0.0.9

* Update Sotopia presentation information in README.md

* bump version to 0.0.10

* bump version

* add merge release back to main action

* change checkout v4->v3

* fix merge-back-to-main and pin mypy to <1.11.0

* merge bug fix

* upgrade default model to handle bad-foratted outputs to gpt-4o-mini as gpt-3.5-turbo is deprecated (#183)

* update pull request -> pull request target

* bump version

* Add `bad_output_process_model` option and `use_fixed_model_version` option for all generation methods, to avoid future OpenAI API changes break Sotopia running. (#196)

* Two major updates: 1) add "bad_output_process_model" option to all `agenerate_xxx()` methods so users can decide which model to use for handling bad outputs. By default, this is set to be `gpt-4o-mini`. 2) add `use_fixed_model_version` option for all generation methods, as some fixed model version may no longer available in the future. Users should have the right to bypass the fixed model version mapping instead of getting stuck in an error. Document (`generation.md`) has been updated for these two major changes correspondingly.

* [autofix.ci] apply automated fixes

---------

Co-authored-by: Chenghao Yang <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix gpt-3.5

* replace gpt3.5 turbo for tests

* update gpt-3.5-turbo to gpt-4o-mini

* bug fix for return fixed model version function

* fix sampling error

* fix rc.4

* new tag

* bump version

* update workflow permission

* add why sotopia

* improve the why sotopia

* bump version

* further add clarification to the custom models

---------

Co-authored-by: XuhuiZhou <[email protected]>
Co-authored-by: Chenghao (Alan) Yang <[email protected]>
Co-authored-by: Chenghao Yang <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* update macos test runner as macos-latest (#238)

* update macos test runner as macos-latest

* redis-stack-server

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* feat(exp_eval): support tag for combo iteration (#245)

* support tag for combo iteration

* fix pre-commit error

* [autofix.ci] apply automated fixes

* fix mypy error

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Adding OpenHands node

* [autofix.ci] apply automated fixes

* Added LLM Agent Node

* [autofix.ci] apply automated fixes

* removing openhands

* [autofix.ci] apply automated fixes

* Correcting mypy error in tick agent

* Moving everything to examples since its not specific to sotopia

* [autofix.ci] apply automated fixes

* Adding some additional final checks

* Renaming to interview openhands

* Adding documentation

* [autofix.ci] apply automated fixes

* Updating the readme

* [autofix.ci] apply automated fixes

* feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249)

* Adding OpenHands node

* [autofix.ci] apply automated fixes

* Added LLM Agent Node

* [autofix.ci] apply automated fixes

* removing openhands

* [autofix.ci] apply automated fixes

* Correcting mypy error in tick agent

* Moving everything to examples since its not specific to sotopia

* [autofix.ci] apply automated fixes

* Adding some additional final checks

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding the initial message channel

* Renaming to interview openhands

* Adding documentation

* [autofix.ci] apply automated fixes

* Updating the readme

* [autofix.ci] apply automated fixes

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding the initial message channel

* Rebasing

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* feat: Adding Chat print node for pretty printing chat conversation between agents (#250)

* Adding OpenHands node

* [autofix.ci] apply automated fixes

* Added LLM Agent Node

* [autofix.ci] apply automated fixes

* removing openhands

* [autofix.ci] apply automated fixes

* Correcting mypy error in tick agent

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Added chat print node

* [autofix.ci] apply automated fixes

* Fixing the runtime channel

* Moving everything to examples since its not specific to sotopia

* [autofix.ci] apply automated fixes

* Adding some additional final checks

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding the initial message channel

* Adding scene context for agents

* Added chat print node

* [autofix.ci] apply automated fixes

* Moving chat node to experiment

* Renaming to interview openhands

* Adding documentation

* [autofix.ci] apply automated fixes

* Updating the readme

* [autofix.ci] apply automated fixes

* feat: Initial Message for LLM agnets in Sotopia Aact (Experimental) (#249)

* Adding OpenHands node

* [autofix.ci] apply automated fixes

* Added LLM Agent Node

* [autofix.ci] apply automated fixes

* removing openhands

* [autofix.ci] apply automated fixes

* Correcting mypy error in tick agent

* Moving everything to examples since its not specific to sotopia

* [autofix.ci] apply automated fixes

* Adding some additional final checks

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding the initial message channel

* Renaming to interview openhands

* Adding documentation

* [autofix.ci] apply automated fixes

* Updating the readme

* [autofix.ci] apply automated fixes

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding the initial message channel

* Rebasing

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Renaming scene context to initial message

* Adding scene context for agents

* Added chat print node

* [autofix.ci] apply automated fixes

* Adding scene context for agents

* [autofix.ci] apply automated fixes

* Fixing the runtime channel

* Moving chat node to experiment

* Updating the toml file paths

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* doc: API endpoints for sotopia (#242)

* api doc

* add PUT

* add an temp example for websocket

* websocket

* update readme

* Update README.md

* update websocket live simulation api doc

* [autofix.ci] apply automated fixes

* update websocket doc

* add api server with websocket as well as a client

* fix mypy errors

* support stopping the chat

* add 404 to the status code

* fix mypy issue

* remove files

* update requirement

* update return message from the server

* add restful error code

* Update README.md

* remove paks

---------

Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: Zhe Su <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* feat: FastAPI Implementation of Sotopia Part One (wo websocket) (#246)

* api doc

* add PUT

* add an temp example for websocket

* websocket

* update readme

* Update README.md

* fastapi server wo websocket

* update project toml

* add websocket api and a sample client sample

* add initial test

* post update

* finalize the doc

* fix the create

* finish test

* add files

* change chata to api

* fix mypy error

* fix mypy bug

* fix mypy

* create mock agents

* downgrade langchain openai upperbound to fix CI

---------

Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: Zhe Su <[email protected]>

* update the instruction to load existing data via docker (#255)

* fix doc (#258)

* Sotopia API and UI (#264)

* feat: FastAPI Implementation of Sotopia Part Two (w websocket) (#252)

* api doc

* add PUT

* add an temp example for websocket

* websocket

* update readme

* Update README.md

* update websocket live simulation api doc

* [autofix.ci] apply automated fixes

* update websocket doc

* add api server with websocket as well as a client

* fix mypy errors

* support stopping the chat

* add 404 to the status code

* fix mypy issue

* update the returned message types

* redesign websocket api

* update websocket, fix mypy error

* add example of using websocket

* clean code & change to existing functions for simulation

* fix typing mismatch

* update doc & mypy type fix

* add type check for run_async_server

* move example

---------

Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: Zhe Su <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Add customizable evaluation dimensions (#256)

* add customizable evaluation dimensions

* add docs

* fix mypy error & refactor examples

* add docs for evaluation dimensions

* update docs and examples

* add test cases and fix mypy issue

* fix mypy issue

* Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) (#262)

Co-authored-by: openhands <[email protected]>

* Fix/custom eval dimension test (#263)

* Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk)

* Update documentation for SotopiaDimension and EvaluationDimensionBuilder

* [autofix.ci] apply automated fixes

* Add API documentation for evaluation dimensions

* Refine API documentation for evaluation_dimensions.py to match style

* [autofix.ci] apply automated fixes

---------

Co-authored-by: openhands <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* add doc

---------

Co-authored-by: XuhuiZhou <[email protected]>
Co-authored-by: openhands <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* Feat/addtional fast apis for non-streaming simulation and managing relationshio (#265)

* temp run

* add relationship api

* fix mypy error

* update relationship api

* simulate episode non-streaming

* modify sim episodes

* add simulation status

* task error

* add background task

* [autofix.ci] apply automated fixes

* back to arun one episode

* upload the code

* use rq to execute background tasks

* temp sol

---------

Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix ci error

* solving pytests

* improve the tests

* add custom eval fast api (#268)

* fix mypy error

* aact moderator (#257)

* initial framework

* initial conv

* fix module error

* feat: Add 3 new features to Moderator (#266)

* feat:introduce booting procedure, saving, and ending chat to moderator

* fix: moderator will now ignore none AgentAction, Observations now don't have to include all channels in the mapping

* merge changes of example into the original one

* fix: 1. save() method now accepts push_to_db config 2. booting()'s waiting time is changed to 0.1 sec

* fix: rewrite booting() so that different agent will receive different background information

* fix: moderator now inherits from Node directly, instead of from BaseAgent

---------

Co-authored-by: JXZhou <JXZhou>

* add save condition for moderator

* push to db false

* to fully stop

* stopping all agents

* fix mypy

* fix mypy error

---------

Co-authored-by: JXZhou <[email protected]>

* Deploy the api to modal (#267)

* prototype for modal serving

* add openai secret

* fix type annotation

* add doc

* bug fix for simulation api

* add customize model, evaluator model and evaluation dimensions

* Implement modal API server with Redis integration and FastAPI setup

- Added a new script for the modal API server that initializes a Redis instance.
- Created a persistent volume for Redis data and included a function to download initial data if not present.
- Configured a Docker image with necessary dependencies including Redis Stack and FastAPI.
- Implemented a web API class that sets up and cleans up the Redis connection, ensuring readiness before serving requests.
- Integrated the SotopiaFastAPI application within the modal framework.

---------

Co-authored-by: XuhuiZhou <[email protected]>

* Feature/sotopia demo UI (#261)

* initial

* initial ui

* merge main

* add new ui

* switch to fastAPI

* websocket check

* fix render episode error

* add  page; make a simplified page and still WIP

* [autofix.ci] apply automated fixes

* fix simplified streaming version

* semi-done character page + avatar assets

* Fixed character card styling

* [autofix.ci] apply automated fixes

* unified rendering and chat display

* updated chat character icons

* add some tags

* add typing

* temp fix

* add characters avatar to simulation

* fix episode full avatar

* go to modal config

* clean up code

* add modal streamlit app

* clean codebase except websocket

* remove repeated local css

* clean websocket

* fix get name error

* fix errors

* pre render scenario

* add custom eval

* change streamlit to dynamic path

* new uv

* revert to previous install commands

* a fix for modal

* add customized dimension

* [autofix.ci] apply automated fixes

* sort scenarios in simulation

* for demo video

* update deploy instruction

* update intro page

* update intro page

* [autofix.ci] apply automated fixes

* update intro page

* add customized dimensions

* update api link and modal environment

* move folder

* fix relative import

* update modal image build

* use uv to build environment

* change folder name

* change test

* fix modal serve

* environment change

* refactor

* fix ui

---------

Co-authored-by: Zhe Su <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: astrophie <[email protected]>

* remove dev tag

* add custom eval

* base dimension

* fix ui mypy

* fix mypy

* add delete dimension

* update streamlit ui

* ignores the ui directory

* Committing changes before push

* pytest for eval dimension

* fix mypy

* clean up comments

* run non streaming simulation

* add pytest for websocket

* fix mypy issue

---------

Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: Zhe Su <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: openhands <[email protected]>
Co-authored-by: JXZhou <[email protected]>
Co-authored-by: astrophie <[email protected]>

---------

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hao Zhu <[email protected]>
Co-authored-by: Chenghao (Alan) Yang <[email protected]>
Co-authored-by: Chenghao Yang <[email protected]>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Haofei Yu <[email protected]>
Co-authored-by: Arpandeep Khatua <[email protected]>
Co-authored-by: Arpandeep Khatua <[email protected]>
Co-authored-by: Zhe Su <[email protected]>
Co-authored-by: openhands <[email protected]>
Co-authored-by: JXZhou <[email protected]>
Co-authored-by: astrophie <[email protected]>
  • Loading branch information
13 people authored Jan 8, 2025
1 parent 2b80515 commit ccccd06
Show file tree
Hide file tree
Showing 72 changed files with 7,631 additions and 2,284 deletions.
1 change: 1 addition & 0 deletions .github/.codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ ignore:
- ".github" # ignore the .github directory
- "docs" # ignore the tests directory
- "figs" # ignore the figs directory
- "ui" # ignore the ui directory

coverage:
status:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/cli_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
strategy:
max-parallel: 5
matrix:
os: [ubuntu-latest, macos-13]
os: [ubuntu-latest, macos-latest]

runs-on: ${{ matrix.os }}

Expand All @@ -38,7 +38,7 @@ jobs:
run: |
python -m pip install --upgrade pip
python -m pip install uv
uv sync --extra test --extra chat
uv sync --extra test --extra api
- name: Test with pytest
run: |
uv run pytest tests/cli/test_install.py --cov=. --cov-report=xml
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/mypy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
run: |
python -m pip install --upgrade pip
python -m pip install uv
uv sync --extra test --extra chat
uv sync --extra test --extra api
- name: Type-checking package with mypy
run: |
# Run this mypy instance against our main package.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests.sh
Original file line number Diff line number Diff line change
@@ -1 +1 @@
uv run --extra test --extra chat pytest --ignore tests/cli --cov=. --cov-report=xml
uv run --extra test --extra api pytest --ignore tests/cli --cov=. --cov-report=xml
2 changes: 1 addition & 1 deletion .github/workflows/tests_in_docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
- name: Docker Compose
run: docker compose -f .devcontainer/docker-compose.yml up -d
- name: Run tests
run: docker compose -f .devcontainer/docker-compose.yml run --rm -u root -v /home/runner/work/sotopia/sotopia:/workspaces/sotopia devcontainer /bin/sh -c "cd /workspaces/sotopia; ls; uv sync --extra test --extra chat; uv run pytest --ignore tests/cli --cov=. --cov-report=xml"
run: docker compose -f .devcontainer/docker-compose.yml run --rm -u root -v /home/runner/work/sotopia/sotopia:/workspaces/sotopia devcontainer /bin/sh -c "cd /workspaces/sotopia; ls; uv sync --extra test --extra api; uv run pytest --ignore tests/cli --cov=. --cov-report=xml"
- name: Upload coverage report to Codecov
uses: codecov/[email protected]
with:
Expand Down
116 changes: 116 additions & 0 deletions docs/pages/concepts/evaluation_dimension.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
## Overview

Evaluation dimensions are used to evaluate the quality of social interactions.
In original Sotopia paper, there are 7 dimensions to evaluate the quality of social interactions, where we named them as `sotopia` evaluation dimensions:
- believability
- relationship
- knowledge
- secret
- social rules
- financial and material benefits
- goal

The `SotopiaDimensions` can be used directly without initializing the database. It provides a set of predefined evaluation dimensions that are ready to use for evaluating social interactions. For example,

```python
from sotopia.envs.parallel import ParallelSotopiaEnv
from sotopia.envs.evaluators import EvaluationForTwoAgents, ReachGoalLLMEvaluator, RuleBasedTerminatedEvaluator, SotopiaDimensions

env = ParallelSotopiaEnv(
env_profile=env_profile,
model_name=model_names["env"],
action_order="round-robin",
evaluators=[
RuleBasedTerminatedEvaluator(max_turn_number=20, max_stale_turn=2),
],
terminal_evaluators=[
ReachGoalLLMEvaluator(
model_names["env"],
EvaluationForTwoAgents[SotopiaDimensions], # type: ignore
# TODO check how to do type annotation
),
],
)
```


However we observe under many use cases people may want to evaluate with customized evaluation metrics, so we provide a way to build custom evaluation dimensions.
For a quick reference, you can directly check out the `examples/use_custom_dimensions.py`.

### CustomEvaluationDimension
The [`CustomEvaluationDimension`](/python_API/database/evaluation_dimensions) is a class that can be used to create a custom evaluation dimension.
There are four parameters:
- name: the name of the dimension
- description: the description of the dimension
- range_low: the minimum score of the dimension (should be an integer)
- range_high: the maximum score of the dimension (should be an integer)

### CustomEvaluationDimensionList
The [`CustomEvaluationDimensionList`](/python_API/database/evaluation_dimensions) is a class that can be used to create a custom evaluation dimension list based on the existing dimensions. It helps one to group multiple dimensions together for a specific use case.
There are two parameters:
- name: the name of the dimension list
- dimension_pks: the primary keys of the dimensions in the dimension list

### EvaluationDimensionBuilder
The [`EvaluationDimensionBuilder`](/python_API/database/evaluation_dimensions) is a class that can be used to generate a custom evaluation dimension model based on the existing dimensions.


## Usage
### Initialize the database
The default evaluation metric is still `SotopiaDimensions` in `sotopia.env.evaluators`.There is no `CustomEvaluationDimension` in the database by default. To initialize the database, please refer to `examples/use_custom_dimensions.py`.


### Use the custom evaluation dimensions
After you initialize your customized evaluation dimensions, you can choose to use any one of these methods provided below:

#### Method 1: Choose dimensions by names
```python
evaluation_dimensions = (
EvaluationDimensionBuilder.select_existing_dimension_model_by_name(
["transactivity", "verbal_equity"]
)
)
```

#### Method 2: Directly choose the grouped evaluation dimension list
```python
evaluation_dimensions = (
EvaluationDimensionBuilder.select_existing_dimension_model_by_list_name(
"sotopia"
)
)
```

#### Method 3: Build a custom evaluation dimension model temporarily
We provide multiple ways to build a custom evaluation dimension model with `EvaluationDimensionBuilder`, specifically:
- `generate_dimension_model`: build an evaluation dimension from existing dimension primary keys.
- `generate_dimension_model_from_dict`: build an evaluation dimension from a dictionary that specifies the parameters of the `CustomEvaluationDimension`. For example
```json
[
{
"name": "believability",
"description": "The believability of the interaction",
"range_low": 0,
"range_high": 10
},
...
]
```
- `select_existing_dimension_model_by_name`: build an evaluation dimension from existing dimension names. For example `['believability', 'goal']`
- `select_existing_dimension_model_by_list_name`: build an evaluation dimension from existing `CustomEvaluationDimensionList` list names. For example, directly use `sotopia`.


After you get the evaluation dimension model, you can pass it as a parameter for the `Evaluator`, for example,
```python
evaluation_dimensions = (
EvaluationDimensionBuilder.select_existing_dimension_model_by_list_name(
"sotopia"
)
)
terminal_evaluators=[
ReachGoalLLMEvaluator(
model_names["env"],
EvaluationForTwoAgents[evaluation_dimensions], # type: ignore
),
],
```
2 changes: 1 addition & 1 deletion docs/pages/contribution/contribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Please refer to [Dev Containers](https://containers.dev/supporting#editors) to s

You can also set up the development environment without Dev Containers. There are three things you will need to set up manually:

- Python and uv: Please start from an environment supporting Python 3.10+ and install uv using `pip install uv; uv sync --all-extra`.
- Python and uv: Please start from an environment supporting Python 3.10+ and install uv using `pip install uv; uv sync --all-extras`. (Note that this will install all the extra dependencies)
- Redis: Please refer to introduction page for the set up of Redis.
- Local LLM (optional): If you don't have access to model endpoints (e.g. OpenAI, Anthropic or others), you can use a local model. You can use Ollama, Llama.cpp, vLLM or many others which support OpenAI compatible endpoints.

Expand Down
6 changes: 6 additions & 0 deletions docs/pages/examples/deployment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Deploy Sotopia Python API to Modal
We offer a script to deploy Sotopia Python API to [Modal](https://modal.com/).
To do so, simply go to the `sotopia/sotopia/ui` directory and run the following command:
```bash
modal deploy sotopia/ui/modal_api_server.py
```
11 changes: 8 additions & 3 deletions docs/pages/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ export REDIS_OM_URL="redis://localhost:6379"
```
if you are developing Sotopia using uv, you can sync your dependency with
```bash
uv sync --extra examples --extra chat
uv sync --extra examples --extra api
```
</AccordionContent>
</AccordionItem>
Expand All @@ -144,13 +144,18 @@ or manual setup:
<AccordionItem value="item-1">
<AccordionTrigger>Docker is my thing.</AccordionTrigger>
<AccordionContent>
Please follow the [instruction](https://redis.io/docs/stack/get-started/install/docker/) to start a redis-stack server or use an existing server. You can also check [Q&A](/docs/troubleshooting.md) to initiate the redis server with the Sotopia data.
Please follow the [instruction](https://redis.io/docs/stack/get-started/install/docker/) to start a redis-stack server or use an existing server. If you want to use the existing data in Sotopia, you can download the `dump.rdb` file from [here](https://cmu.box.com/shared/static/xiivc5z8rnmi1zr6vmk1ohxslylvynur). Feel free to check more datasets related to Sotopia [here](https://huggingface.co/collections/cmu-lti/sotopia-65f312c1bd04a8c4a9225e5b).

After downloading the `dump.rdb` file, make a `redis-data` folder in an desired `<your_path>` directory. And then you can start the server with the following command:
```bash
docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 -v <your_path>/redis-data:/data/ redis/redis-stack:latest
```

The `REDIS_OM_URL` need to be set before loading and saving agents:
```bash
conda env config vars set REDIS_OM_URL="redis://user:password@host:port"
```
</AccordionContent>
</AccordionContent>
</AccordionItem>
<AccordionItem value="item-2">
<AccordionTrigger>No, I don't want to use Docker.</AccordionTrigger>
Expand Down
54 changes: 54 additions & 0 deletions docs/pages/python_API/database/evaluation_dimensions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# `evaluation_dimensions.py`

This module provides classes and utilities for defining and managing custom evaluation dimensions within the Sotopia environment. It includes classes for individual dimensions, lists of dimensions, and a builder for creating dimension models.

## Classes

### `CustomEvaluationDimension`

Represents a custom evaluation dimension with specific attributes such as name, description, and score range.

#### Attributes
- `name`: `str`. The name of the dimension.
- `description`: `str`. A brief description of the dimension.
- `range_low`: `int`. The minimum score for the dimension.
- `range_high`: `int`. The maximum score for the dimension.

### `CustomEvaluationDimensionList`

Groups multiple custom evaluation dimensions together.

#### Attributes
- `name`: `str`. The name of the dimension list.
- `dimension_pks`: `list[str]`. A list of primary keys for the dimensions included in the list.

### `EvaluationDimensionBuilder`

Provides utility methods to create and manage evaluation dimension models.

#### Methods
- `create_range_validator(low: int, high: int)`: Creates a validator for score ranges.

**Arguments:**
- `low`: `int`. The minimum score allowed.
- `high`: `int`. The maximum score allowed.

- `build_dimension_model(dimension_ids: list[str])`: Builds a dimension model from primary keys.

**Arguments:**
- `dimension_ids`: `list[str]`. A list of dimension primary keys.

- `build_dimension_model_from_dict(dimensions: list[dict[str, Union[str, int]]])`: Builds a dimension model from a dictionary.

**Arguments:**
- `dimensions`: `list[dict[str, Union[str, int]]]`. A list of dictionaries specifying dimension attributes.

- `select_existing_dimension_model_by_name(dimension_names: list[str])`: Selects a dimension model by dimension names.

**Arguments:**
- `dimension_names`: `list[str]`. A list of dimension names.

- `select_existing_dimension_model_by_list_name(list_name: str)`: Selects a dimension model by list name.

**Arguments:**
- `list_name`: `str`. The name of the dimension list.
30 changes: 27 additions & 3 deletions examples/experiment_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
EnvAgentComboStorage,
EnvironmentProfile,
EpisodeLog,
EvaluationDimensionBuilder,
)
from sotopia.envs.evaluators import (
EvaluationForTwoAgents,
Expand All @@ -34,6 +35,7 @@
)
from sotopia.server import run_async_server
from sotopia_conf.gin_utils import parse_gin_flags, run
# from sotopia.database import EvaluationDimensionBuilder

_DEFAULT_GIN_SEARCH_PATHS = [
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
Expand Down Expand Up @@ -109,6 +111,18 @@ def _iterate_env_agent_combo_not_in_db(
tag: str | None = None,
) -> Generator[EnvAgentCombo[Observation, AgentAction], None, None]:
"""We iterate over each environment and return the **first** env-agent combo that is not in the database."""
# loading evaluation metric
try:
evaluation_dimensions = EvaluationDimensionBuilder.select_existing_dimension_model_by_list_name(
"sotopia"
) # Initialize your customized dimension, please refer to `examples/use_custom_dimensions.py`
except Exception as e:
print(
"No customized evaluation dimensions found, using default SotopiaDimensions",
e,
)
evaluation_dimensions = SotopiaDimensions

if not env_ids:
env_ids = list(EnvironmentProfile.all_pks())
for env_id in env_ids:
Expand All @@ -123,6 +137,11 @@ def _iterate_env_agent_combo_not_in_db(
)
assert env_agent_combo_storage_list
first_env_agent_combo_storage_to_run: EnvAgentComboStorage | None = None

env_agent_combo_storage_list = sorted(
env_agent_combo_storage_list, key=lambda x: str(x.pk)
)

for env_agent_combo_storage in env_agent_combo_storage_list:
env_agent_combo_storage = cast(
EnvAgentComboStorage, env_agent_combo_storage
Expand All @@ -147,7 +166,8 @@ def _iterate_env_agent_combo_not_in_db(
terminal_evaluators=[
ReachGoalLLMEvaluator(
model_names["env"],
EvaluationForTwoAgents[SotopiaDimensions],
EvaluationForTwoAgents[evaluation_dimensions], # type: ignore
# TODO check how to do type annotation
),
],
)
Expand Down Expand Up @@ -183,10 +203,14 @@ def run_async_server_in_batch(
logger.removeHandler(rich_handler)

# we cannot get the exact length of the generator, we just give an estimate of the length
env_agent_combo_iter = _iterate_env_agent_combo_not_in_db(model_names=model_names)
env_agent_combo_iter = _iterate_env_agent_combo_not_in_db(
model_names=model_names, tag=tag
)
env_agent_combo_iter_length = sum(1 for _ in env_agent_combo_iter)

env_agent_combo_iter = _iterate_env_agent_combo_not_in_db(model_names=model_names)
env_agent_combo_iter = _iterate_env_agent_combo_not_in_db(
model_names=model_names, tag=tag
)
env_agent_combo_batch: list[EnvAgentCombo[Observation, AgentAction]] = []

while True:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from aact import Message, NodeFactory
from aact.messages import Text, Tick, DataModel, DataModelFactory
from sotopia.agents.llm_agent import ainput
from sotopia.experimental.agents import BaseAgent
from sotopia.experimental.agents.base_agent import BaseAgent

from sotopia.generation_utils import agenerate
from sotopia.generation_utils.generate import StrOutputParser
Expand Down
Loading

0 comments on commit ccccd06

Please sign in to comment.