Skip to content
This repository has been archived by the owner on Oct 16, 2023. It is now read-only.

Update tiktoken requirement from 0.3.3 to 0.4.0 #73

Merged
merged 1 commit into from
Jul 27, 2023

Conversation

dependabot[bot]
Copy link
Contributor

@dependabot dependabot bot commented on behalf of github Jul 27, 2023

Updates the requirements on tiktoken to permit the latest version.

Changelog

Sourced from tiktoken's changelog.

[v0.4.0]

  • Add decode_batch and decode_bytes_batch
  • Improve error messages and handling

[v0.3.3]

  • tiktoken will now make a best effort attempt to replace surrogate pairs with the corresponding Unicode character and will replace lone surrogates with the Unicode replacement character.

[v0.3.2]

  • Add encoding for GPT-4

[v0.3.1]

  • Build aarch64 wheels
  • Make blobfile an optional dependency

Thank you to @​messense for the environment variable that makes cargo not OOM under emulation!

[v0.3.0]

  • Improve performance by 5-20%; thank you to @​nistath!
  • Add gpt-3.5-turbo models to encoding_for_model
  • Add prefix matching to encoding_for_model to better support future model versions
  • Fix a bug in the README instructions on extending tiktoken
  • Update the set of available encodings
  • Add packaging metadata

[v0.2.0]

  • Add tiktoken.encoding_for_model to get the encoding for a specific model
  • Improve portability of caching logic

Thank you to @​fritzo, @​arvid220u, @​khanhvu207, @​henriktorget for various small corrections

[v0.1.2]

  • Avoid use of blobfile for public files
  • Add support for Python 3.8
  • Add py.typed
  • Improve the public tests

[v0.1.1]

  • Initial release
Commits

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Updates the requirements on [tiktoken](https://github.com/openai/tiktoken) to permit the latest version.
- [Release notes](https://github.com/openai/tiktoken/releases)
- [Changelog](https://github.com/openai/tiktoken/blob/main/CHANGELOG.md)
- [Commits](openai/tiktoken@0.3.3...0.4.0)

---
updated-dependencies:
- dependency-name: tiktoken
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot dependabot bot added dependencies Pull requests that update a dependency file python Issues that require code changes labels Jul 27, 2023
@Daethyra
Copy link
Owner

Remove unused modules from deps

@Daethyra Daethyra merged commit a2078e0 into master Jul 27, 2023
2 of 3 checks passed
@dependabot dependabot bot deleted the dependabot/pip/master/tiktoken-0.4.0 branch July 27, 2023 18:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dependencies Pull requests that update a dependency file python Issues that require code changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant