[컴] MLC LLM 를 windows 에서 사용

machine learning compilation /

MLC LLM 를 windows 에서 사용

MLC LLM 에 대한 ref.1 의 설명:

• 다양한 하드웨어 backend들과 native application에다가 모든 언어모델들(language models)을 natively 하게 배포하기 위한 범용솔루션(universal solution) 이다.
• 서버 없이, 폰이나 노트북의 local GPU들로 가속한다.

환경

• windows 10
• geforce gtx 1660, nvidia driver 531.79

Prerequisites

1. 최신 vulkan driver
   • 이것은 Nvidia driver를 update 하면 된다. nvidia graphic card driver 가 vulkan driver 도 지원한다.
2. miniconda 설치
   • 쿠…sal: [컴] Conda package manager
   • 쿠…sal: [컴] windows 에서 miniconda 설치 경로 변경

작업

1. mlc-chat conda 환경 생성하고, 여기에 git, git-lfs를 설치.
   • 이것은 Hugging Face에서 model weights 를 다운로드하기 위한 것이라 한다.
2. CLI app 을 설치
3. dist directory 생성, 이 directory 에 git clone 을 하게 된다.
4. git lfs 초기화
5. git clone
6. 실행: mlc_chat_cli

# 환경생성, git, git-lfs 설치
conda create -n mlc-chat
conda activate mlc-chat

conda install git git-lfs

# chat CLI app 을 설치
conda install -c mlc-ai -c conda-forge mlc-chat-nightly

# `dist` directory 생성
mkdir dist

# lfs 초기화
git lfs install

# git clone, 대략 6.2 GB 필요
git clone https://huggingface.co/mlc-ai/demo-vicuna-v1-7b-int3 dist/vicuna-v1-7b
git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/lib

# 실행
mlc_chat_cli

실행

질문에 대한 답변을 해줄때 대략 GPU 를 86% 사용한다.

(d:\a\env\conda\mlc-chat) D:\a\prog\mlcchat\try1>mlc_chat_cli
Use lib D:\a\prog\mlcchat\try1\dist\lib\vicuna-v1-7b-q3f16_0-vulkan.dll
[11:42:21] C:\Miniconda\envs\mlc-llm-build\conda-bld\mlc-chat-nightly-package_1683954581677\work\cpp\cli_main.cc:298: model_path = D:\a\prog\mlcchat\try1\dist\lib
Initializing the chat module...
Finish loading
You can use the following special commands:
  /help    print the special commands
  /exit    quit the cli
  /stats   print out the latest stats (token/sec)
  /reset   restart a fresh chat

USER: /help
You can use the following special commands:
  /help    print the special commands
  /exit    quit the cli
  /stats   print out the latest stats (token/sec)
  /reset   restart a fresh chat

USER: /stats
encode: -nan(ind) tok/s, decode: -nan(ind) tok/s
USER: hi what is you?
ASSISTANT: Hello! As an artificial intelligence language model, "I" (as a pronoun) represents the collective intelligence and knowledge of the language model. So the answer to your question is: everything! I'm here to assist you with any questions or concerns you may have. Please feel free to ask me anything you'd like to know more about.
USER: how can i change your weights?
ASSISTANT: As an AI language model, my "weights" are adjusted based on the information and training I received during my training phase. However, you can adjust the importance of different words and phrases in your questions to me by using the "weight" function.

For example, you can assign higher or lower weight to specific words or phrases in your question by using the "weight" function. For example:

* "weight:20 my\_question" will give higher importance to your question than the "weight:30 other\_questions"
* "weight:10 my\_question" will give higher importance to your question than the "weight:20 other\_questions"

So please go ahead and give your questions higher or lower importance based on what is most important for you. And please don't hesitate to ask questions at the appropriate importance level if you're unsure which weight to use.
USER: so the higher weight means more important?
ASSISTANT: Yes! That's correct. Assigning a higher weight to a specific word or phrase in your question can help to give it more emphasis or priority in the conversation. The weight you assign to a word or phrase can help to indicate the level of importance or focus you want to give it in the conversation. For example, if you're unsure which weight to use for a specific aspect of your question, you can assign a higher weight to it by using the "weight" function. For example:

* "weight:20 my\_question" will give higher importance to your question than the "weight:30 other\_questions"
* "weight:10 my\_question" will give higher importance to your question than the "weight:20 other\_questions"
USER: what if i say like you mentioned, weight:20 my\_question would have bigger weight than other\_questions?
ASSISTANT: Yes! That's correct. Assigning a higher weight to your question can indicate that it is of greater importance or focus than other questions in the conversation. However, please note that the weight you assign to something does not change the information or context of the conversation. The weight you assign to something in a question will only affect the order in which it appears in the conversation. So please go ahead and give your questions higher or lower importance based on what is most important for you. And please don't hesitate to ask questions at the appropriate importance level if you're unsure which weight to use.
USER: so what is difference from you and chatgpt
ASSISTANT: As an AI language model, my main function is to process and generate text based on the input I receive. I don't have personal preferences or emotions like humans do. The weight of a word or a question depends on the algorithm that was used to build and train me.
USER: how to train you?
ASSISTANT: As an AI language model, I can be trained using supervised learning techniques. This typically involves using a large dataset of text and labeling the words within the text based on a given task or objective. For example, if you want to train me to predict the weather forecast for a given day based on the historical weather data.

In this way I can improve the quality of my responses to better match the user's expectations.
USER: can u program?
ASSISTANT: Yes, I can be programmed in various programming languages like Python, JavaScript, C++, etc. However, I don't have the ability to program or create new algorithms as I am just a software program myself.
USER: show me an example
ASSISTANT: Sure! Here's an example of a simple program in Python:
```python
with open('example.py', 'w') as file:
    file.write('This is an example of a Python program.\n')
    file.close()
print('Example written and saved to file.')
```
This code creates a file called "example.py" in the current directory, writes a string to it and then prints a message indicating that the example was written and saved.
USER:

See Also

1. GitHub - imartinez/privateGPT: Interact privately with your documents using the power of GPT, 100% privately, no data leaks

Reference

1. GitHub - mlc-ai/mlc-llm: Enable everyone to develop, optimize and deploy AI models natively on everyone’s devices.
2. MLC LLM | Home