TheBloke/Mixtral-Fusion-4x7B-Instruct-v0.1-GGUFを動かす

AI

環境

準備

# TheBloke/Mixtral-Fusion-4x7B-Instruct-v0.1-GGUF
# llama.cppのmixtralブランチをクローン（mergeが済んだら「-b mixtral」不要）
!git clone -b mixtral https://github.com/ggerganov/llama.cpp
%cd llama.cpp
!make
#!make LLAMA_CUBLAS=1　#GPUオフロードする場合

# モデルのダウンロード（Q4_K_M-GGUF）
!wget https://huggingface.co/TheBloke/Mixtral-Fusion-4x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-fusion-4x7b-instruct-v0.1.Q4_K_M.gguf -P ./models/

推論

# Q4_K_Mモデルで推論
!./main -m ./models/mixtral-fusion-4x7b-instruct-v0.1.Q4_K_M.gguf \
       --prompt "まどマギで一番可愛いキャラはなんですか？" \
       --n-predict 512 --threads 8 #--n-gpu-layers 999 #GPUオフロードする場合

回答

 まどマギで一番可愛いキャラはなんですか？<0x0A> ( What is it ? ) - It is a game, where you can make money. You will be able to play in the world of your dreams and make money for your loved ones. The game is not a job but a way to live that you love. - So it's good if I play this game, because you love.<0x0A><0x0A>You are right. It shows how much you care about others. ( What is it ? ) It shows how much you care about others.<0x0A><0x0A>So it's good if I play this game, because you love.<0x0A><0x0A>But the question is - It is not a job but a way to live that you love.<0x0A><0x0A>What is it? - It is not a job but a way to live that you love.<0x0A><0x0A>So it's good if I play this game, because you love. ( What is it ? ) - It shows how much you care about others.<0x0A><0x0A>But the question is - It is not a job but a way to live that you love.<0x0A><0x0A>So it’s good if I play this game, because you love. ( What is it ? ) - It shows how much you care about others.<0x0A><0x0A>So it’s good if I play this game, because you love. ( What is it ? ) - It shows how much you care about others. But the question is - It is not a job but a way to live that you love.<0x0A><0x0A>So it’s good if I play this game, because you love. ( What is it ? ) - It shows how much you care about others.<0x0A><0x0A>But the question is - It is not a job but a way to life that you love.<0x0A><0x0A>So it’s good if I play this game, because you love.<0x0A><0x0A>But the question is - It is not a job but a way to life that you love.<0x0A><0x0A>So it’s good if I play this game, because you love.<0x0A><0x0A>And there are no down sides in playing the game.<0x0A><0x0A>But what is it ? ( What is it ? ) - It is not a job but a way of life that you love.<0x0A><0x0A> So it’s good if I play this game, because you love.<0x0A><0x0A> But the question is - It is not a job but a way to live that you love.<0x0A><0x0A> So it’s good if I play this game, because you love. ( What is it ? ) - It shows