Skip to content

Latest commit

 

History

History
75 lines (51 loc) · 4.14 KB

README_en.md

File metadata and controls

75 lines (51 loc) · 4.14 KB

ChatGLM Application Based on Local Knowledge

Introduction

🌍 中文文档

🤖️ A local knowledge based LLM Application with ChatGLM-6B and langchain.

💡 Inspired by document.ai by GanymedeNil and ChatGLM-6B Pull Request by AlexZhangji.

✅ In this project, GanymedeNil/text2vec-large-chinese is used as Embedding Model，and ChatGLM-6B used as LLM。Based on those models，this project can be deployed offline with all open source models。

Update

[2023/04/07]

  1. Fix bug which costs twice gpu memory (Thanks to @suc16 and @myml).
  2. Add gpu memory clear function after each call of ChatGLM.
  3. Add nghuyong/ernie-3.0-nano-zh and nghuyong/ernie-3.0-base-zh as Embedding model alternatives，costing less gpu than GanymedeNil/text2vec-large-chinese (Thanks to @lastrei)

[2023/04/09]

  1. Using RetrievalQA in langchain to replace the previously selected ChatVectorDBChain, the replacement can effectively solve the problem of program stopping after 2-3 questions due to insufficient gpu memory.
  2. Add EMBEDDING_MODEL, VECTOR_SEARCH_TOP_K, LLM_MODEL, LLM_HISTORY_LEN, REPLY_WITH_SOURCE parameter value settings in knowledge_based_chatglm.py.
  3. Add chatglm-6b-int4, chatglm-6b-int4-qe with smaller GPU memory requirements as LLM model alternatives.
  4. Correct code errors in README.md (Thanks to @calcitem).

Usage

Hardware Requirements

  • ChatGLM Hardware Requirements

    Quantization Level GPU Memory
    FP16（no quantization） 13 GB
    INT8 10 GB
    INT4 6 GB

  • Embedding Hardware Requirements

    The default Embedding model in this repo is GanymedeNil/text2vec-large-chinese, 3GB GPU Memory required when running on GPU.

Software Requirements

This repo has been tested in python 3.8 environment。

1. install python packages

pip install -r requirements.txt

Attention: With langchain.document_loaders.UnstructuredFileLoader used to connect with local knowledge file, you may need some other dependencies as mentioned in langchain documentation

2. Run knowledge_based_chatglm.py script

python knowledge_based_chatglm.py

Known issues

  • Currently tested to support txt, docx, md format files, for more file formats please refer to langchain documentation. If the document contains special characters, the file may not be correctly loaded.
  • When running this project with macOS, it may not work properly due to incompatibility with pytorch caused by macOS version 13.3 and above.

FAQ

Q: How to solve Resource punkt not found.?

A: Unzip packages/tokenizers in https://github.com/nltk/nltk_data/raw/gh-pages/packages/tokenizers/punkt.zip and put it in the corresponding directory of Searched in:.

Q: How to solve Resource averaged_perceptron_tagger not found.?

A: Download https://github.com/nltk/nltk_data/blob/gh-pages/packages/taggers/averaged_perceptron_tagger.zip, decompress it and put it in the corresponding directory of Searched in:.

Roadmap

  • local knowledge based application with langchain + ChatGLM-6B
  • unstructured files loaded with langchain
  • more different file format loaded with langchain
  • implement web ui DEMO with gradio/streamlit
  • implement API with fastapi，and web ui DEMO with API