暫無描述

Lengyue ca11d4b869 Update text only mode		2 年之前
data_server	2966ba9019 Support speaker finetune	2 年之前
figs	9e664473b9 Update document	2 年之前
fish_speech	ca11d4b869 Update text only mode	2 年之前
tools	2966ba9019 Support speaker finetune	2 年之前
.dockerignore	57755e1e74 Add rust data server	2 年之前
.gitignore	f5919b7e71 Update finetune toolchain	2 年之前
.pre-commit-config.yaml	f1ea577c4c [pre-commit.ci] pre-commit autoupdate	2 年之前
LICENSE	62710e34a4 Add llama inference tool chain	2 年之前
README.md	ca11d4b869 Update text only mode	2 年之前
README.zh.md	ca11d4b869 Update text only mode	2 年之前
dockerfile	c7cf70e904 Fix dataset and docker	2 年之前
pyproject.toml	57755e1e74 Add rust data server	2 年之前
pyrightconfig.json	f7f2c03282 Support pytorch lightning	2 年之前

Fish Speech

Documentation is under construction, English is not fully supported yet.

中文文档

This codebase is released under BSD-3-Clause License, and all models are released under CC-BY-NC-SA-4.0 License. Please refer to LICENSE for more details.

Disclaimer

We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.

Requirements

GPU memory: 2GB (for inference), 24GB (for finetuning)
System: Linux (full functionality), Windows (inference only, flash-attn is not supported, torch.compile is not supported)

Therefore, we strongly recommend to use WSL2 or docker to run the codebase for Windows users.

Setup

# Basic environment setup
conda create -n fish-speech python=3.10
conda activate fish-speech
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# Install flash-attn (for linux)
pip3 install ninja && MAX_JOBS=4 pip3 install flash-attn --no-build-isolation

# Install fish-speech
pip3 install -e .

Inference (CLI)

Download required vqgan and text2semantic model from our huggingface repo.

wget https://huggingface.co/fishaudio/speech-lm-v1/raw/main/vqgan-v1.pth -O checkpoints/vqgan-v1.pth
wget https://huggingface.co/fishaudio/speech-lm-v1/blob/main/text2semantic-400m-v0.2-4k.pth -O checkpoints/text2semantic-400m-v0.2-4k.pth

Generate semantic tokens from text:

python tools/llama/generate.py \
    --text "Hello" \
    --num-samples 2 \
    --compile

You may want to use --compile to fuse cuda kernels faster inference (~25 tokens/sec -> ~300 tokens/sec).

Generate vocals from semantic tokens:

python tools/vqgan/inference.py -i codes_0.npy

Rust Data Server

Since loading and shuffle the dataset is very slow and memory consuming, we use a rust server to load and shuffle the dataset. The server is based on GRPC and can be installed by

cd data_server
cargo build --release

README.md