Benchmarking o1-preview-2024-09-12

[haesleinhuepf]

If anyone knows anyone who is tier 5 in openai (@royerloic maybe ?), they could benchmark the new o1 model. I am just tier3 and have to wait...

https://x.com/OpenAI/status/1834278218888872042

https://platform.openai.com/docs/models/o1

[jkh1]

It seems that this model has hidden tokens that you still get billed for (see note in the docs here: https://platform.openai.com/docs/guides/reasoning/how-reasoning-works) which may explain why it's for tier 5 users 😃
This could become an expensive experiment.

[haesleinhuepf]

Yeah I know. I think claude does something similar. That's why I'm curious if it's really so much better than the models we tested so far.

[haesleinhuepf]

Ok, I have access now. Just FYI: My first 12 prompts cost $6.68 for o1-preview, hence running the entire benchmark would cost about $300. (updated)