Update for OpenAudio S1 (#986)

* update for s1

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

README.md

@@ -36,69 +36,65 @@
 
 This codebase is released under Apache License and all model weights are released under CC-BY-NC-SA-4.0 License. Please refer to [LICENSE](LICENSE) for more details.
 
----
-## Fish Agent
-We are very excited to announce that we have made our self-research agent demo open source, you can now try our agent demo for instant English and Chinese chat locally by following the [docs](https://speech.fish.audio/start_agent/).
+We are excited to announce that we have changed our name into OpenAudio, this will be a brand new series of Text-to-Speech model.
 
-You should mention that the content is released under a **CC BY-NC-SA 4.0 licence**. And the demo is an early alpha test version, the inference speed needs to be optimised, and there are a lot of bugs waiting to be fixed. If you've found a bug or want to fix it, we'd be very happy to receive an issue or a pull request.
+Demo available at [Fish Audio Playground](https://fish.audio).
+
+Visit the [OpenAudio website](https://openaudio.com) for blog & tech report.
 
 ## Features
-### Fish Speech
+### OpenAudio-S1 (Fish-Speech's new verison)
+
+1. This model has **ALL FEATURES** that fish-speech had.
 
-1. **Zero-shot & Few-shot TTS:** Input a 10 to 30-second vocal sample to generate high-quality TTS output. **For detailed guidelines, see [Voice Cloning Best Practices](https://docs.fish.audio/text-to-speech/voice-clone-best-practices).**
+2. OpenAudio S1 supports a variety of emotional, tone, and special markers to enhance speech synthesis:
+   
+   (angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused)
 
-2. **Multilingual & Cross-lingual Support:** Simply copy and paste multilingual text into the input box—no need to worry about the language. Currently supports English, Japanese, Korean, Chinese, French, German, Arabic, and Spanish.
+   Also supports tone marker:
 
-3. **No Phoneme Dependency:** The model has strong generalization capabilities and does not rely on phonemes for TTS. It can handle text in any language script.
+   (in a hurry tone) (shouting) (screaming) (whispering) (soft tone)
 
-4. **Highly Accurate:** Achieves a low CER (Character Error Rate) and WER (Word Error Rate) of around 2% for 5-minute English texts.
+    There's a few special markers that are supported:
 
-5. **Fast:** With fish-tech acceleration, the real-time factor is approximately 1:5 on an Nvidia RTX 4060 laptop and 1:15 on an Nvidia RTX 4090.
+    (laughing) (chuckling) (sobbing) (crying loudly) (sighing) (panting) (groaning) (crowd laughing) (background laughter) (audience laughing)
 
-6. **WebUI Inference:** Features an easy-to-use, Gradio-based web UI compatible with Chrome, Firefox, Edge, and other browsers.
+    You can also use **Ha,ha,ha** to control, there's many other cases waiting to be explored by yourself.
 
-7. **GUI Inference:** Offers a PyQt6 graphical interface that works seamlessly with the API server. Supports Linux, Windows, and macOS. [See GUI](https://github.com/AnyaCoder/fish-speech-gui).
+3. The OpenAudio S1 includes the following sizes:
+-   **S1 (4B, proprietary):** The full-sized model.
+-   **S1-mini (0.5B, open-sourced):** A distilled version of S1.
 
-8. **Deploy-Friendly:** Easily set up an inference server with native support for Linux, Windows and MacOS, minimizing speed loss.
+    Both S1 and S1-mini incorporate online Reinforcement Learning from Human Feedback (RLHF).
 
-### Fish Agent
-1. **Completely End to End:** Automatically integrates ASR and TTS parts, no need to plug-in other models, i.e., true end-to-end, not three-stage (ASR+LLM+TTS).
+4. Evaluations
 
-2. **Timbre Control:** Can use reference audio to control the speech timbre.
+    **Seed TTS Eval Metrics (English, auto eval, based on OpenAI gpt-4o-transcribe, speaker distance using Revai/pyannote-wespeaker-voxceleb-resnet34-LM):**
 
-3. **Emotional:** The model can generate speech with strong emotion.
+    -   **S1:**
+        -   WER (Word Error Rate): **0.008**
+        -   CER (Character Error Rate): **0.004**
+        -   Distance: **0.332**
+    -   **S1-mini:**
+        -   WER (Word Error Rate): **0.011**
+        -   CER (Character Error Rate): **0.005**
+        -   Distance: **0.380**
+    
 
 ## Disclaimer
 
 We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.
 
-## Online Demo
-
-[Fish Audio](https://fish.audio)
-
-[Fish Agent](https://fish.audio/demo/live)
-
-## Quick Start for Local Inference 
-
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/fishaudio/fish-speech/blob/main/inference.ipynb)
-
 ## Videos
 
-#### V1.5 Demo Video: [Watch the video on X (Twitter).](https://x.com/FishAudio/status/1864370933496205728)
+#### To be continued.
 
 ## Documents
 
-- [English](https://speech.fish.audio/)
-- [中文](https://speech.fish.audio/zh/)
-- [日本語](https://speech.fish.audio/ja/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/)
-
-## Samples (2024/10/02 V1.4)
+- [Build Envrionment](docs/en/install.md)
+- [Inference](docs/en/inference.md)
 
-- [English](https://speech.fish.audio/samples/)
-- [中文](https://speech.fish.audio/zh/samples/)
-- [日本語](https://speech.fish.audio/ja/samples/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/samples/)
+It should be noted that the current model **DOESN'T SUPPORT FINETUNE**.
 
 ## Credits
 
@@ -121,13 +117,3 @@ We do not hold any responsibility for any illegal usage of the codebase. Please
       url={https://arxiv.org/abs/2411.01156},
 }
 ```
-
-## Sponsor
-
-<div>
-  <a href="https://6block.com/">
-    <img src="https://avatars.githubusercontent.com/u/60573493" width="100" height="100" alt="6Block Avatar"/>
-  </a>
-  <br>
-  <a href="https://6block.com/">Data Processing sponsor by 6Block</a>
-</div>

docs/README.ja.md

@@ -1,7 +1,7 @@
 <div align="center">
 <h1>Fish Speech</h1>
 
-[English](../README.md) | [简体中文](README.zh.md) | [Portuguese](README.pt-BR.md) | **日本語** | [한국어](README.ko.md)<br>
+[English](../README.md) | [简体中文](README.zh.md) | [Portuguese](README.pt-BR.md) | **日本語** | [한국어](README.ko.md) <br>
 
 <a href="https://www.producthunt.com/posts/fish-speech-1-4?embed=true&utm_source=badge-featured&utm_medium=badge&utm_souce=badge-fish&#0045;speech&#0045;1&#0045;4" target="_blank">
     <img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=488440&theme=light" alt="Fish&#0032;Speech&#0032;1&#0046;4 - Open&#0045;Source&#0032;Multilingual&#0032;Text&#0045;to&#0045;Speech&#0032;with&#0032;Voice&#0032;Cloning | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" />
@@ -16,6 +16,7 @@
 <div align="center">
     <img src="https://count.getloli.com/get/@fish-speech?theme=asoul" /><br>
 </div>
+
 <br>
 
 <div align="center">
@@ -28,52 +29,72 @@
     <a target="_blank" href="https://huggingface.co/spaces/fishaudio/fish-speech-1">
         <img alt="Huggingface" src="https://img.shields.io/badge/🤗%20-space%20demo-yellow"/>
     </a>
+    <a target="_blank" href="https://pd.qq.com/s/bwxia254o">
+      <img alt="QQ Channel" src="https://img.shields.io/badge/QQ-blue?logo=tencentqq">
+    </a>
 </div>
 
-このコードリポジトリはApache 2.0ライセンスの下で公開されており、モデルはCC-BY-NC-SA-4.0ライセンスの下で公開されています。詳細については[LICENSE](../LICENSE)をご参照ください。
+このコードベースはApache Licenseの下でリリースされ、すべてのモデルウェイトはCC-BY-NC-SA-4.0 Licenseの下でリリースされています。詳細については[LICENSE](../LICENSE)をご参照ください。
+
+私たちは名前をOpenAudioに変更したことをお知らせでき、嬉しく思います。これは全く新しいText-to-Speechモデルシリーズになります。
+
+デモは[Fish Audio Playground](https://fish.audio)で利用可能です。
 
----
+ブログと技術レポートについては[OpenAudioウェブサイト](https://openaudio.com)をご覧ください。
 
 ## 機能
+### OpenAudio-S1 (Fish-Speechの新バージョン)
 
-1. **ゼロショット & フューショット TTS**：10〜30 秒の音声サンプルを入力して、高品質の TTS 出力を生成します。**詳細は [音声クローンのベストプラクティス](https://docs.fish.audio/text-to-speech/voice-clone-best-practices) を参照してください。**
-2. **多言語 & クロスリンガル対応**：多言語テキストを入力ボックスにコピーペーストするだけで、言語を気にする必要はありません。現在、英語、日本語、韓国語、中国語、フランス語、ドイツ語、アラビア語、スペイン語に対応しています。
-3. **音素依存なし**：このモデルは強力な汎化能力を持ち、TTS に音素を必要としません。あらゆる言語スクリプトに対応可能です。
-4. **高精度**：5 分間の英語テキストに対し、CER（文字誤り率）と WER（単語誤り率）は約 2%の精度を達成します。
-5. **高速**：fish-tech アクセラレーションにより、Nvidia RTX 4060 ラップトップではリアルタイムファクターが約 1:5、Nvidia RTX 4090 では約 1:15 です。
-6. **WebUI 推論**：使いやすい Gradio ベースの Web ユーザーインターフェースを搭載し、Chrome、Firefox、Edge などのブラウザに対応しています。
-7. **GUI 推論**：PyQt6 のグラフィカルインターフェースを提供し、API サーバーとシームレスに連携します。Linux、Windows、macOS に対応しています。[GUI を見る](https://github.com/AnyaCoder/fish-speech-gui)。
-8. **デプロイしやすい**：Linux、Windows、macOS にネイティブ対応した推論サーバーを簡単にセットアップでき、速度の低下を最小限に抑えます。
+1. このモデルはfish-speechが持っていた**すべての機能**を持っています。
 
-## 免責事項
+2. OpenAudio S1は音声合成を強化するための様々な感情、トーン、特別なマーカーをサポートしています：
+   
+      (angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused)
 
-コードベースの違法な使用については一切責任を負いません。DMCA（デジタルミレニアム著作権法）およびその他の関連法については、地域の法律を参照してください。
+   またトーンマーカーもサポートしています：
 
-## オンラインデモ
+   (急いだトーン) (叫び) (絶叫) (ささやき) (柔らかいトーン)
 
-[Fish Audio](https://fish.audio)
+    サポートされているいくつかの特別なマーカーがあります：
 
-## ローカル推論のクイックスタート
+    (笑い) (くすくす笑い) (すすり泣き) (大声で泣く) (ため息) (あえぎ) (うめき) (群衆の笑い) (背景の笑い) (観客の笑い)
 
-[inference.ipynb](/inference.ipynb)
+    また、**ハ、ハ、ハ**を使って制御することもでき、あなた自身が探索を待っている他の多くのケースがあります。
 
-## ビデオ
+3. OpenAudio S1には以下のサイズが含まれています：
+-   **S1 (4B, プロプライエタリ):** フルサイズのモデル。
+-   **S1-mini (0.5B, オープンソース):** S1の蒸留版。
 
-#### V1.5 デモビデオ: [Watch the video on X (Twitter).](https://x.com/FishAudio/status/1864370933496205728)
+    S1とS1-miniの両方がオンライン人間フィードバック強化学習（RLHF）を組み込んでいます。
 
-## ドキュメント
+4. 評価
+
+    **Seed TTS評価メトリクス（英語、自動評価、OpenAI gpt-4o-transcribeベース、Revai/pyannote-wespeaker-voxceleb-resnet34-LMを使用したスピーカー距離）：**
+
+    -   **S1:**
+        -   WER（単語誤り率）：**0.008**
+        -   CER（文字誤り率）：**0.004**
+        -   距離：**0.332**
+    -   **S1-mini:**
+        -   WER（単語誤り率）：**0.011**
+        -   CER（文字誤り率）：**0.005**
+        -   距離：**0.380**
+    
 
-- [英語](https://speech.fish.audio/)
-- [中文](https://speech.fish.audio/zh/)
-- [日本語](https://speech.fish.audio/ja/)
-- [ポルトガル語 (ブラジル)](https://speech.fish.audio/pt/)
+## 免責事項
+
+コードベースの違法な使用について、いかなる責任も負いません。DMCAおよびその他の関連法律に関する現地の法律をご参照ください。
+
+## 動画
+
+#### 続く予定。
 
-## サンプル (2024/10/02 V1.4)
+## ドキュメント
+
+- [環境構築](en/install.md)
+- [推論](en/inference.md)
 
-- [英語](https://speech.fish.audio/samples/)
-- [中文](https://speech.fish.audio/zh/samples/)
-- [日本語](https://speech.fish.audio/ja/samples/)
-- [ポルトガル語 (ブラジル)](https://speech.fish.audio/pt/samples/)
+現在のモデルは**ファインチューニングをサポートしていない**ことに注意してください。
 
 ## クレジット
 
@@ -84,12 +105,15 @@
 - [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
 - [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
 
-## スポンサー
-
-<div>
-  <a href="https://6block.com/">
-    <img src="https://avatars.githubusercontent.com/u/60573493" width="100" height="100" alt="6Block Avatar"/>
-  </a>
-  <br>
-  <a href="https://6block.com/">データ処理スポンサー：6Block</a>
-</div>
+## 技術レポート (V1.4)
+```bibtex
+@misc{fish-speech-v1.4,
+      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
+      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
+      year={2024},
+      eprint={2411.01156},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD},
+      url={https://arxiv.org/abs/2411.01156},
+}
+```

docs/README.ko.md

@@ -16,6 +16,7 @@
 <div align="center">
     <img src="https://count.getloli.com/get/@fish-speech?theme=asoul" /><br>
 </div>
+
 <br>
 
 <div align="center">
@@ -28,63 +29,74 @@
     <a target="_blank" href="https://huggingface.co/spaces/fishaudio/fish-speech-1">
         <img alt="Huggingface" src="https://img.shields.io/badge/🤗%20-space%20demo-yellow"/>
     </a>
+    <a target="_blank" href="https://pd.qq.com/s/bwxia254o">
+      <img alt="QQ Channel" src="https://img.shields.io/badge/QQ-blue?logo=tencentqq">
+    </a>
 </div>
 
-이 코드 저장소는 Apache 2.0 라이선스 하에 배포되며, 모델은 CC-BY-NC-SA-4.0 라이선스 하에 배포됩니다. 자세한 내용은 [LICENSE](../LICENSE)를 참조하십시오.
+이 코드베이스는 Apache License 하에 릴리스되며, 모든 모델 가중치는 CC-BY-NC-SA-4.0 License 하에 릴리스됩니다. 자세한 내용은 [LICENSE](../LICENSE)를 참조하세요.
+
+저희는 이름을 OpenAudio로 변경했다고 발표하게 되어 기쁩니다. 이는 완전히 새로운 Text-to-Speech 모델 시리즈가 될 것입니다.
 
----
+데모는 [Fish Audio Playground](https://fish.audio)에서 사용할 수 있습니다.
+
+블로그와 기술 보고서는 [OpenAudio 웹사이트](https://openaudio.com)를 방문하세요.
 
 ## 기능
+### OpenAudio-S1 (Fish-Speech의 새 버전)
 
-1. **Zero-shot & Few-shot TTS:** 10초에서 30초의 음성 샘플을 입력하여 고품질의 TTS 출력을 생성합니다. **자세한 가이드는 [모범 사례](https://docs.fish.audio/text-to-speech/voice-clone-best-practices)를 참조하시길 바랍니다.**
+1. 이 모델은 fish-speech가 가지고 있던 **모든 기능**을 가지고 있습니다.
 
-2. **다국어 및 교차 언어 지원:** 다국어 걱정 없이, 텍스트를 입력창에 복사하여 붙여넣기만 하면 됩니다. 현재 영어, 일본어, 한국어, 중국어, 프랑스어, 독일어, 아랍어, 스페인어를 지원합니다.
+2. OpenAudio S1은 음성 합성을 향상시키기 위한 다양한 감정, 톤, 특별한 마커를 지원합니다:
+   
+      (angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused)
 
-3. **음소 의존성 제거:** 이 모델은 강력한 일반화 능력을 가지고 있으며, TTS가 음소에 의존하지 않습니다. 모든 언어 스크립트 텍스트를 손쉽게 처리할 수 있습니다.
+   또한 톤 마커도 지원합니다:
 
-4. **높은 정확도:** 영어 텍스트 기준 5분 기준에서 단, 2%의 문자 오류율(CER)과 단어 오류율(WER)을 달성합니다.
+   (급한 톤) (외치기) (비명지르기) (속삭이기) (부드러운 톤)
 
-5. **빠른 속도:** fish-tech 가속을 통해 실시간 인자(RTF)는 Nvidia RTX 4060 노트북에서는 약 1:5, Nvidia RTX 4090에서는 1:15입니다.
+    지원되는 몇 가지 특별한 마커가 있습니다:
 
-6. **웹 UI 추론:** Chrome, Firefox, Edge 등 다양한 브라우저에서 호환되는 Gradio 기반의 사용하기 쉬운 웹 UI를 제공합니다.
+    (웃음) (킥킥거림) (흐느낌) (큰 소리로 우는 것) (한숨) (헐떡거림) (신음) (군중 웃음) (배경 웃음) (관객 웃음)
 
-7. **GUI 추론:** PyQt6 그래픽 인터페이스를 제공하여 API 서버와 원활하게 작동합니다. Linux, Windows 및 macOS를 지원합니다. [GUI 참조](https://github.com/AnyaCoder/fish-speech-gui).
+    또한 **하, 하, 하**를 사용하여 제어할 수도 있으며, 여러분이 직접 탐험할 수 있는 많은 다른 경우들이 있습니다.
 
-8. **배포 친화적:** Linux, Windows, macOS에서 네이티브로 지원되는 추론 서버를 쉽게 설정할 수 있어 속도 손실을 최소화합니다.
+3. OpenAudio S1은 다음 크기를 포함합니다:
+-   **S1 (4B, 독점):** 전체 크기 모델.
+-   **S1-mini (0.5B, 오픈소스):** S1의 증류 버전.
 
-## 면책 조항
+    S1과 S1-mini 모두 온라인 인간 피드백 강화학습(RLHF)을 통합하고 있습니다.
 
-이 코드베이스의 불법적 사용에 대해 어떠한 책임도 지지 않습니다. DMCA 및 관련 법률에 대한 로컬 법률을 참조하십시오.
+4. 평가
 
-## 온라인 데모
+    **Seed TTS 평가 메트릭 (영어, 자동 평가, OpenAI gpt-4o-transcribe 기반, Revai/pyannote-wespeaker-voxceleb-resnet34-LM을 사용한 화자 거리):**
 
-[Fish Audio](https://fish.audio)
+    -   **S1:**
+        -   WER (단어 오류율): **0.008**
+        -   CER (문자 오류율): **0.004**
+        -   거리: **0.332**
+    -   **S1-mini:**
+        -   WER (단어 오류율): **0.011**
+        -   CER (문자 오류율): **0.005**
+        -   거리: **0.380**
+    
 
-## 로컬 추론을 위한 빠른 시작
+## 면책 조항
 
-[inference.ipynb](/inference.ipynb)
+저희는 코드베이스의 불법적인 사용에 대해 어떠한 책임도 지지 않습니다. DMCA 및 기타 관련 법률에 대한 현지 법률을 참조하세요.
 
-## 영상
+## 비디오
 
-#### V1.5 데모 영상: [Watch the video on X (Twitter).](https://x.com/FishAudio/status/1864370933496205728)
+#### 계속될 예정입니다.
 
 ## 문서
 
-- [English](https://speech.fish.audio/)
-- [中文](https://speech.fish.audio/zh/)
-- [日本語](https://speech.fish.audio/ja/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/)
-- [한국어](https://speech.fish.audio/ko/)
-
-## Samples (2024/10/02 V1.4)
+- [환경 구축](en/install.md)
+- [추론](en/inference.md)
 
-- [English](https://speech.fish.audio/samples/)
-- [中文](https://speech.fish.audio/zh/samples/)
-- [日本語](https://speech.fish.audio/ja/samples/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/samples/)
-- [한국어](https://speech.fish.audio/ko/samples/)
+현재 모델은 **파인튜닝을 지원하지 않는다**는 점에 유의해야 합니다.
 
-## Credits
+## 크레딧
 
 - [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
 - [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
@@ -93,12 +105,15 @@
 - [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
 - [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
 
-## Sponsor
-
-<div>
-  <a href="https://6block.com/">
-    <img src="https://avatars.githubusercontent.com/u/60573493" width="100" height="100" alt="6Block Avatar"/>
-  </a>
-  <br>
-  <a href="https://6block.com/">데이터 처리 후원: 6Block</a>
-</div>
+## 기술 보고서 (V1.4)
+```bibtex
+@misc{fish-speech-v1.4,
+      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
+      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
+      year={2024},
+      eprint={2411.01156},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD},
+      url={https://arxiv.org/abs/2411.01156},
+}
+```

docs/README.pt-BR.md

@@ -1,7 +1,7 @@
 <div align="center">
 <h1>Fish Speech</h1>
 
-[English](../README.md) | [简体中文](README.zh.md) | **Portuguese** | [日本語](README.ja.md) | [한국어](README.ko.md)<br>
+[English](../README.md) | [简体中文](README.zh.md) | **Portuguese** | [日本語](README.ja.md) | [한국어](README.ko.md) <br>
 
 <a href="https://www.producthunt.com/posts/fish-speech-1-4?embed=true&utm_source=badge-featured&utm_medium=badge&utm_souce=badge-fish&#0045;speech&#0045;1&#0045;4" target="_blank">
     <img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=488440&theme=light" alt="Fish&#0032;Speech&#0032;1&#0046;4 - Open&#0045;Source&#0032;Multilingual&#0032;Text&#0045;to&#0045;Speech&#0032;with&#0032;Voice&#0032;Cloning | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" />
@@ -29,61 +29,74 @@
     <a target="_blank" href="https://huggingface.co/spaces/fishaudio/fish-speech-1">
         <img alt="Huggingface" src="https://img.shields.io/badge/🤗%20-space%20demo-yellow"/>
     </a>
+    <a target="_blank" href="https://pd.qq.com/s/bwxia254o">
+      <img alt="QQ Channel" src="https://img.shields.io/badge/QQ-blue?logo=tencentqq">
+    </a>
 </div>
 
-Este repositório de código é disponibilizado sob a licença Apache 2.0, e o modelo sob a licença CC-BY-NC-SA-4.0. Consulte [LICENSE](../LICENSE) para mais detalhes.
+Esta base de código é lançada sob a Licença Apache e todos os pesos dos modelos são lançados sob a Licença CC-BY-NC-SA-4.0. Consulte [LICENSE](../LICENSE) para mais detalhes.
 
----
+Estamos animados em anunciar que mudamos nosso nome para OpenAudio, esta será uma nova série de modelos Text-to-Speech.
 
-## Funcionalidades
+Demo disponível em [Fish Audio Playground](https://fish.audio).
 
-1. **TTS Zero-shot & Few-shot**: Insira uma amostra vocal de 10 a 30 segundos para gerar saída de TTS de alta qualidade. **Para diretrizes detalhadas, veja [Melhores Práticas para Clonagem de Voz](https://docs.fish.audio/text-to-speech/voice-clone-best-practices).**
+Visite o [site OpenAudio](https://openaudio.com) para blog e relatório técnico.
 
-2. **Suporte Multilíngue e Interlingual**: Basta copiar e colar o texto multilíngue na caixa de entrada—não se preocupe com o idioma. Atualmente suporta inglês, japonês, coreano, chinês, francês, alemão, árabe e espanhol.
+## Recursos
+### OpenAudio-S1 (Nova versão do Fish-Speech)
 
-3. **Sem Dependência de Fonemas**: O modelo tem forte capacidade de generalização e não depende de fonemas para TTS. Ele pode lidar com textos em qualquer script de idioma.
+1. Este modelo possui **TODOS OS RECURSOS** que o fish-speech tinha.
 
-4. **Alta Precisão**: Alcança uma CER (Taxa de Erro de Caracteres) e WER (Taxa de Erro de Palavras) de cerca de 2% para textos de 5 minutos em inglês.
+2. O OpenAudio S1 suporta uma variedade de marcadores emocionais, de tom e especiais para aprimorar a síntese de fala:
+   
+      (angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused)
 
-5. **Rápido**: Com a aceleração fish-tech, o fator de tempo real é de aproximadamente 1:5 em um laptop Nvidia RTX 4060 e 1:15 em uma Nvidia RTX 4090.
+   Também suporta marcadores de tom:
 
-6. **Inferência WebUI**: Apresenta uma interface de usuário web baseada em Gradio, fácil de usar e compatível com navegadores como Chrome, Firefox e Edge.
+   (tom apressado) (gritando) (berrando) (sussurrando) (tom suave)
 
-7. **Inferência GUI**: Oferece uma interface gráfica PyQt6 que funciona perfeitamente com o servidor API. Suporta Linux, Windows e macOS. [Veja o GUI](https://github.com/AnyaCoder/fish-speech-gui).
+    Há alguns marcadores especiais que são suportados:
 
-8. **Fácil de Implantar**: Configura facilmente um servidor de inferência com suporte nativo para Linux, Windows e macOS, minimizando a perda de velocidade.
+    (rindo) (dando risadinhas) (soluçando) (chorando alto) (suspirando) (ofegando) (gemendo) (multidão rindo) (riso de fundo) (audiência rindo)
 
-## Isenção de Responsabilidade
+    Você também pode usar **Ha,ha,ha** para controlar, há muitos outros casos esperando para serem explorados por você mesmo.
 
-Não nos responsabilizamos por qualquer uso ilegal do código-fonte. Consulte as leis locais sobre DMCA (Digital Millennium Copyright Act) e outras leis relevantes em sua região.
+3. O OpenAudio S1 inclui os seguintes tamanhos:
+-   **S1 (4B, proprietário):** O modelo de tamanho completo.
+-   **S1-mini (0.5B, código aberto):** Uma versão destilada do S1.
 
-## Demonstração Online
+    Tanto S1 quanto S1-mini incorporam Aprendizado por Reforço online a partir de Feedback Humano (RLHF).
 
-[Fish Audio](https://fish.audio)
+4. Avaliações
 
-## Início Rápido de Inferência Local
+    **Métricas de Avaliação Seed TTS (Inglês, avaliação automática, baseada no OpenAI gpt-4o-transcribe, distância do locutor usando Revai/pyannote-wespeaker-voxceleb-resnet34-LM):**
 
-[inference.ipynb](/inference.ipynb)
+    -   **S1:**
+        -   WER (Taxa de Erro de Palavra): **0.008**
+        -   CER (Taxa de Erro de Caractere): **0.004**
+        -   Distância: **0.332**
+    -   **S1-mini:**
+        -   WER (Taxa de Erro de Palavra): **0.011**
+        -   CER (Taxa de Erro de Caractere): **0.005**
+        -   Distância: **0.380**
+    
 
-## Vídeos
+## Aviso Legal
+
+Não assumimos qualquer responsabilidade por qualquer uso ilegal da base de código. Consulte suas leis locais sobre DMCA e outras leis relacionadas.
 
-#### 1.5 Introdução: [Watch the video on X (Twitter).](https://x.com/FishAudio/status/1864370933496205728)
+## Vídeos
 
-## Documentação
+#### A ser continuado.
 
-- [Inglês](https://speech.fish.audio/)
-- [Chinês](https://speech.fish.audio/zh/)
-- [Japonês](https://speech.fish.audio/ja/)
-- [Português (Brasil)](https://speech.fish.audio/pt/)
+## Documentos
 
-## Exemplos
+- [Construir Ambiente](en/install.md)
+- [Inferência](en/inference.md)
 
-- [Inglês](https://speech.fish.audio/samples/)
-- [Chinês](https://speech.fish.audio/zh/samples/)
-- [Japonês](https://speech.fish.audio/ja/samples/)
-- [Português (Brasil)](https://speech.fish.audio/pt/samples/)
+Deve-se notar que o modelo atual **NÃO SUPORTA AJUSTE FINO**.
 
-## Agradecimentos
+## Créditos
 
 - [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
 - [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
@@ -92,12 +105,15 @@ Não nos responsabilizamos por qualquer uso ilegal do código-fonte. Consulte as
 - [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
 - [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
 
-## Patrocinadores
-
-<div>
-  <a href="https://6block.com/">
-    <img src="https://avatars.githubusercontent.com/u/60573493" width="100" height="100" alt="6Block Avatar"/>
-  </a>
-  <br>
-  <a href="https://6block.com/">Servidores de processamento de dados fornecidos por 6Block</a>
-</div>
+## Relatório Técnico (V1.4)
+```bibtex
+@misc{fish-speech-v1.4,
+      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
+      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
+      year={2024},
+      eprint={2411.01156},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD},
+      url={https://arxiv.org/abs/2411.01156},
+}
+```

docs/README.zh.md

@@ -1,7 +1,7 @@
 <div align="center">
 <h1>Fish Speech</h1>
 
-[English](../README.md) | **简体中文** | [Portuguese](README.pt-BR.md) | [日本語](README.ja.md) | [한국어](README.ko.md)<br>
+[English](../README.md) | **简体中文** | [Portuguese](README.pt-BR.md) | [日本語](README.ja.md) | [한국어](README.ko.md) <br>
 
 <a href="https://www.producthunt.com/posts/fish-speech-1-4?embed=true&utm_source=badge-featured&utm_medium=badge&utm_souce=badge-fish&#0045;speech&#0045;1&#0045;4" target="_blank">
     <img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=488440&theme=light" alt="Fish&#0032;Speech&#0032;1&#0046;4 - Open&#0045;Source&#0032;Multilingual&#0032;Text&#0045;to&#0045;Speech&#0032;with&#0032;Voice&#0032;Cloning | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" />
@@ -29,62 +29,74 @@
     <a target="_blank" href="https://huggingface.co/spaces/fishaudio/fish-speech-1">
         <img alt="Huggingface" src="https://img.shields.io/badge/🤗%20-space%20demo-yellow"/>
     </a>
-    <br>
-
+    <a target="_blank" href="https://pd.qq.com/s/bwxia254o">
+      <img alt="QQ Channel" src="https://img.shields.io/badge/QQ-blue?logo=tencentqq">
+    </a>
 </div>
 
-此代码库根据 Apache 2.0 许可证发布，模型根据 CC-BY-NC-SA-4.0 许可证发布。请参阅 [LICENSE](../LICENSE) 了解更多细节.
+此代码库在 Apache License 下发布，所有模型权重在 CC-BY-NC-SA-4.0 License 下发布。更多详情请参考 [LICENSE](../LICENSE)。
+
+我们很高兴地宣布，我们已将名字更改为 OpenAudio，这将是一个全新的文本转语音模型系列。
+
+演示可在 [Fish Audio Playground](https://fish.audio) 获得。
 
----
+访问 [OpenAudio 网站](https://openaudio.com) 获取博客和技术报告。
 
 ## 特性
+### OpenAudio-S1 (Fish-Speech 的新版本)
 
-1. **零样本 & 小样本 TTS**：输入 10 到 30 秒的声音样本即可生成高质量的 TTS 输出。**详见 [语音克隆最佳实践指南](https://docs.fish.audio/text-to-speech/voice-clone-best-practices)。**
-2. **多语言 & 跨语言支持**：只需复制并粘贴多语言文本到输入框中，无需担心语言问题。目前支持英语、日语、韩语、中文、法语、德语、阿拉伯语和西班牙语。
-3. **无音素依赖**：模型具备强大的泛化能力，不依赖音素进行 TTS，能够处理任何文字表示的语言。
-4. **高准确率**：在 5 分钟的英文文本上，达到了约 2% 的 CER（字符错误率）和 WER（词错误率）。
-5. **快速**：通过 fish-tech 加速，在 Nvidia RTX 4060 笔记本上的实时因子约为 1:5，在 Nvidia RTX 4090 上约为 1:15。
-6. **WebUI 推理**：提供易于使用的基于 Gradio 的网页用户界面，兼容 Chrome、Firefox、Edge 等浏览器。
-7. **GUI 推理**：提供 PyQt6 图形界面，与 API 服务器无缝协作。支持 Linux、Windows 和 macOS。[查看 GUI](https://github.com/AnyaCoder/fish-speech-gui)。
-8. **易于部署**：轻松设置推理服务器，原生支持 Linux、Windows 和 macOS，最大程度减少速度损失。
+1. 此模型具有 fish-speech 的**所有功能**。
 
-## 免责声明
+2. OpenAudio S1 支持多种情感、语调和特殊标记来增强语音合成：
+   
+      (angry) (sad) (disdainful) (excited) (surprised) (satisfied) (unhappy) (anxious) (hysterical) (delighted) (scared) (worried) (indifferent) (upset) (impatient) (nervous) (guilty) (scornful) (frustrated) (depressed) (panicked) (furious) (empathetic) (embarrassed) (reluctant) (disgusted) (keen) (moved) (proud) (relaxed) (grateful) (confident) (interested) (curious) (confused) (joyful) (disapproving) (negative) (denying) (astonished) (serious) (sarcastic) (conciliative) (comforting) (sincere) (sneering) (hesitating) (yielding) (painful) (awkward) (amused) PS:中文也支持
 
-我们不对代码库的任何非法使用承担任何责任. 请参阅您当地关于 DMCA (数字千年法案) 和其他相关法律法规.
+   同时支持语调标记：
 
-## 在线 DEMO
+   (急促的语调) (大喊) (尖叫) (低语) (温柔的语调)
 
-[Fish Audio](https://fish.audio)
+    还有一些特殊标记得到支持：
 
-## 快速开始本地推理
+    (笑声) (轻笑) (抽泣) (大声哭泣) (叹气) (喘气) (呻吟) (人群笑声) (背景笑声) (观众笑声)
 
-[inference.ipynb](/inference.ipynb)
+    您也可以使用 **哈，哈，哈** 来控制，还有许多其他情况等待您自己探索。
 
-## 视频
+3. OpenAudio S1 包含以下规模：
+-   **S1 (4B, 专有)：** 完整规模的模型。
+-   **S1-mini (0.5B, 开源)：** S1 的蒸馏版本。
 
-#### 1.5 介绍: https://www.bilibili.com/video/BV1EKiDYBE4o
+    S1 和 S1-mini 都结合了在线人类反馈强化学习（RLHF）。
 
-#### 1.4 介绍: https://www.bilibili.com/video/BV1pu46eVEk7
+4. 评估
 
-#### 1.2 介绍: https://www.bilibili.com/video/BV1wz421B71D
+    **Seed TTS 评估指标（英语，自动评估，基于 OpenAI gpt-4o-transcribe，使用 Revai/pyannote-wespeaker-voxceleb-resnet34-LM 的说话人距离）：**
 
-#### 1.1 介绍: https://www.bilibili.com/video/BV1zJ4m1K7cj
+    -   **S1：**
+        -   WER（词错误率）：**0.008**
+        -   CER（字符错误率）：**0.004**
+        -   距离：**0.332**
+    -   **S1-mini：**
+        -   WER（词错误率）：**0.011**
+        -   CER（字符错误率）：**0.005**
+        -   距离：**0.380**
+    
 
-## 文档
+## 免责声明
 
-- [English](https://speech.fish.audio/)
-- [中文](https://speech.fish.audio/zh/)
-- [日本語](https://speech.fish.audio/ja/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/)
+我们不对代码库的任何非法使用承担责任。请参考您当地关于 DMCA 和其他相关法律的规定。
 
-## 例子 (2024/10/02 V1.4)
+## 视频
+
+#### 待续。
+
+## 文档
 
-- [English](https://speech.fish.audio/samples/)
-- [中文](https://speech.fish.audio/zh/samples/)
-- [日本語](https://speech.fish.audio/ja/samples/)
-- [Portuguese (Brazil)](https://speech.fish.audio/pt/samples/)
+- [构建环境](zh/install.md)
+- [推理](zh/inference.md)
 
-## 鸣谢
+需要注意的是，当前模型**不支持微调**。
+
+## 致谢
 
 - [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
 - [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
@@ -93,12 +105,15 @@
 - [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
 - [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
 
-## 赞助
-
-<div>
-  <a href="https://6block.com/">
-    <img src="https://avatars.githubusercontent.com/u/60573493" width="100" height="100" alt="6Block Avatar"/>
-  </a>
-  <br>
-  <a href="https://6block.com/">数据处理服务器由 6Block 提供</a>
-</div>
+## 技术报告 (V1.4)
+```bibtex
+@misc{fish-speech-v1.4,
+      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
+      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
+      year={2024},
+      eprint={2411.01156},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD},
+      url={https://arxiv.org/abs/2411.01156},
+}
+```

docs/en/finetune.md

@@ -1,128 +0,0 @@
-# Fine-tuning
-
-Obviously, when you opened this page, you were not satisfied with the performance of the few-shot pre-trained model. You want to fine-tune a model to improve its performance on your dataset.
-
-In current version, you only need to finetune the 'LLAMA' part.
-
-## Fine-tuning LLAMA
-### 1. Prepare the dataset
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 30.1-32.71.lab
-│   └── 30.1-32.71.mp3
-└── SPK2
-    ├── 38.79-40.85.lab
-    └── 38.79-40.85.mp3
-```
-
-You need to convert your dataset into the above format and place it under `data`. The audio file can have the extensions `.mp3`, `.wav`, or `.flac`, and the annotation file should have the extensions `.lab`.
-
-!!! info "Dataset Format"
-    The `.lab` annotation file only needs to contain the transcription of the audio, with no special formatting required. For example, if `hi.mp3` says "Hello, goodbye," then the `hi.lab` file would contain a single line of text: "Hello, goodbye."
-
-!!! warning
-    It's recommended to apply loudness normalization to the dataset. You can use [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) to do this.
-
-    ```bash
-    fap loudness-norm data-raw data --clean
-    ```
-
-
-### 2. Batch extraction of semantic tokens
-
-Make sure you have downloaded the VQGAN weights. If not, run the following command:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-You can then run the following command to extract semantic tokens:
-
-```bash
-python tools/vqgan/extract_vq.py data \
-    --num-workers 1 --batch-size 16 \
-    --config-name "firefly_gan_vq" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-!!! note
-    You can adjust `--num-workers` and `--batch-size` to increase extraction speed, but please make sure not to exceed your GPU memory limit.  
-    For the VITS format, you can specify a file list using `--filelist xxx.list`.
-
-This command will create `.npy` files in the `data` directory, as shown below:
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 21.15-26.44.npy
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 27.51-29.98.npy
-│   ├── 30.1-32.71.lab
-│   ├── 30.1-32.71.mp3
-│   └── 30.1-32.71.npy
-└── SPK2
-    ├── 38.79-40.85.lab
-    ├── 38.79-40.85.mp3
-    └── 38.79-40.85.npy
-```
-
-### 3. Pack the dataset into protobuf
-
-```bash
-python tools/llama/build_dataset.py \
-    --input "data" \
-    --output "data/protos" \
-    --text-extension .lab \
-    --num-workers 16
-```
-
-After the command finishes executing, you should see the `quantized-dataset-ft.protos` file in the `data` directory.
-
-### 4. Finally, fine-tuning with LoRA
-
-Similarly, make sure you have downloaded the `LLAMA` weights. If not, run the following command:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-Finally, you can start the fine-tuning by running the following command:
-
-```bash
-python fish_speech/train.py --config-name text2semantic_finetune \
-    project=$project \
-    +lora@model.model.lora_config=r_8_alpha_16
-```
-
-!!! note
-    You can modify the training parameters such as `batch_size`, `gradient_accumulation_steps`, etc. to fit your GPU memory by modifying `fish_speech/configs/text2semantic_finetune.yaml`.
-
-!!! note
-    For Windows users, you can use `trainer.strategy.process_group_backend=gloo` to avoid `nccl` issues.
-
-After training is complete, you can refer to the [inference](inference.md) section to generate speech.
-
-!!! info
-    By default, the model will only learn the speaker's speech patterns and not the timbre. You still need to use prompts to ensure timbre stability.
-    If you want to learn the timbre, you can increase the number of training steps, but this may lead to overfitting.
-
-After training, you need to convert the LoRA weights to regular weights before performing inference.
-
-```bash
-python tools/llama/merge_lora.py \
-	--lora-config r_8_alpha_16 \
-	--base-weight checkpoints/fish-speech-1.5 \
-	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
-	--output checkpoints/fish-speech-1.5-yth-lora/
-```
-!!! note
-    You may also try other checkpoints. We suggest using the earliest checkpoint that meets your requirements, as they often perform better on out-of-distribution (OOD) data.

docs/en/index.md

@@ -1,174 +0,0 @@
-# Introduction
-
-<div>
-<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
-<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
-<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
-<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
-</a>
-</div>
-
-!!! warning
-    We assume no responsibility for any illegal use of the codebase. Please refer to the local laws regarding DMCA (Digital Millennium Copyright Act) and other relevant laws in your area. <br/>
-    This codebase and all models are released under the CC-BY-NC-SA-4.0 license.
-
-<p align="center">
-   <img src="../assets/figs/diagram.png" width="75%">
-</p>
-
-## Requirements
-
-- GPU Memory: 4GB (for inference), 8GB (for fine-tuning)
-- System: Linux, Windows
-
-## Windows Setup
-
-!!! info "Attention"
-
-    We strongly suggest non-professional windows users use our official GUI to run the project. [GUI is here](https://github.com/AnyaCoder/fish-speech-gui).
-
-Professional Windows users may consider using WSL2 or Docker to run the codebase.
-
-```bash
-# Create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# Install pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
-
-# Install fish-speech
-pip3 install -e .
-
-# (Enable acceleration) Install triton-windows
-pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
-```
-
-## Linux Setup
-
-See [pyproject.toml](../../pyproject.toml) for details.
-```bash
-# Create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# Install pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-
-# (Ubuntu / Debian User) Install sox + ffmpeg
-apt install libsox-dev ffmpeg 
-
-# (Ubuntu / Debian User) Install pyaudio 
-apt install build-essential \
-    cmake \
-    libasound-dev \
-    portaudio19-dev \
-    libportaudio2 \
-    libportaudiocpp0
-    
-# Install fish-speech
-pip3 install -e .[stable]
-```
-
-## MacOS setup
-
-If you want to perform inference on MPS, please add the `--device mps` flag.
-Please refer to [this PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772) for a comparison of inference speeds.
-
-!!! warning
-    The `compile` option is not officially supported on Apple Silicon devices, so there is no guarantee that inference speed will improve.
-
-```bash
-# install dependencies
-brew install portaudio
-# create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-# install pytorch
-pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-# install fish-speech
-pip install -e .[stable]
-```
-
-## Docker Setup
-
-1. Install NVIDIA Container Toolkit:
-
-    To use GPU for model training and inference in Docker, you need to install NVIDIA Container Toolkit:
-
-    For Ubuntu users:
-
-    ```bash
-    # Add repository
-    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
-        && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
-            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
-            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-    # Install nvidia-container-toolkit
-    sudo apt-get update
-    sudo apt-get install -y nvidia-container-toolkit
-    # Restart Docker service
-    sudo systemctl restart docker
-    ```
-
-    For users of other Linux distributions, please refer to: [NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
-
-2. Pull and run the fish-speech image
-
-    ```shell
-    # Pull the image
-    docker pull fishaudio/fish-speech:latest-dev
-    # Run the image
-    docker run -it \
-        --name fish-speech \
-        --gpus all \
-        -p 7860:7860 \
-        fishaudio/fish-speech:latest-dev \
-        zsh
-    # If you need to use a different port, please modify the -p parameter to YourPort:7860
-    ```
-
-3. Download model dependencies
-
-    Make sure you are in the terminal inside the docker container, then download the required `vqgan` and `llama` models from our huggingface repository.
-
-    ```bash
-    huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-4. Configure environment variables and access WebUI
-
-    In the terminal inside the docker container, enter `export GRADIO_SERVER_NAME="0.0.0.0"` to allow external access to the gradio service inside docker.
-    Then in the terminal inside the docker container, enter `python tools/run_webui.py` to start the WebUI service.
-
-    If you're using WSL or MacOS, visit [http://localhost:7860](http://localhost:7860) to open the WebUI interface.
-
-    If it's deployed on a server, replace localhost with your server's IP.
-
-## Changelog
-
-- 2024/12/03: Updated Fish-Speech to 1.5 version, supports more languages, and reaches SOTA in the Open-Source field.
-- 2024/09/10: Updated Fish-Speech to 1.4 version, with an increase in dataset size and a change in the quantizer's n_groups from 4 to 8.
-- 2024/07/02: Updated Fish-Speech to 1.2 version, remove VITS Decoder, and greatly enhanced zero-shot ability.
-- 2024/05/10: Updated Fish-Speech to 1.1 version, implement VITS decoder to reduce WER and improve timbre similarity.
-- 2024/04/22: Finished Fish-Speech 1.0 version, significantly modified VQGAN and LLAMA models.
-- 2023/12/28: Added `lora` fine-tuning support.
-- 2023/12/27: Add `gradient checkpointing`, `causual sampling`, and `flash-attn` support.
-- 2023/12/19: Updated webui and HTTP API.
-- 2023/12/18: Updated fine-tuning documentation and related examples.
-- 2023/12/17: Updated `text2semantic` model, supporting phoneme-free mode.
-- 2023/12/13: Beta version released, includes VQGAN model and a language model based on LLAMA (phoneme support only).
-
-## Acknowledgements
-
-- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
-- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
-- [GPT VITS](https://github.com/innnky/gpt-vits)
-- [MQTTS](https://github.com/b04901014/MQTTS)
-- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
-- [Transformers](https://github.com/huggingface/transformers)
-- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/en/inference.md

@@ -1,144 +0,0 @@
-# Inference
-
-Inference support command line, HTTP API and web UI.
-
-!!! note
-    Overall, reasoning consists of several parts:
-
-    1. Encode a given ~10 seconds of voice using VQGAN.
-    2. Input the encoded semantic tokens and the corresponding text into the language model as an example.
-    3. Given a new piece of text, let the model generate the corresponding semantic tokens.
-    4. Input the generated semantic tokens into VITS / VQGAN to decode and generate the corresponding voice.
-
-## Download Models
-Download the required `vqgan` and `llama` models from our Hugging Face repository.
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-## Command Line Inference
-### 1. Generate prompt from voice:
-
-!!! note
-    If you plan to let the model randomly choose a voice timbre, you can skip this step.
-
-!!! warning "Future Warning"
-    We have kept the interface accessible from the original path (tools/vqgan/inference.py), but this interface may be removed in subsequent releases, so please change your code as soon as possible.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "paimon.wav" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-You should get a `fake.npy` file.
-
-### 2. Generate semantic tokens from text:
-
-!!! warning "Future Warning"
-    We have kept the interface accessible from the original path (tools/llama/generate.py), but this interface may be removed in subsequent releases, so please change your code as soon as possible.
-
-```bash
-python fish_speech/models/text2semantic/inference.py \
-    --text "The text you want to convert" \
-    --prompt-text "Your reference text" \
-    --prompt-tokens "fake.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5" \
-    --num-samples 2 \
-    --compile
-```
-
-This command will create a `codes_N` file in the working directory, where N is an integer starting from 0.
-
-!!! note
-    You may want to use `--compile` to fuse CUDA kernels for faster inference (~30 tokens/second -> ~500 tokens/second).
-    Correspondingly, if you do not plan to use acceleration, you can comment out the `--compile` parameter.
-
-!!! info
-    For GPUs that do not support bf16, you may need to use the `--half` parameter.
-
-### 3. Generate vocals from semantic tokens:
-
-#### VQGAN Decoder
-
-!!! warning "Future Warning"
-    We have kept the interface accessible from the original path (tools/vqgan/inference.py), but this interface may be removed in subsequent releases, so please change your code as soon as possible.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "codes_0.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-## HTTP API Inference
-
-We provide a HTTP API for inference. You can use the following command to start the server:
-
-```bash
-python -m tools.api_server \
-    --listen 0.0.0.0:8080 \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-
-> If you want to speed up inference, you can add the `--compile` parameter.
-
-After that, you can view and test the API at http://127.0.0.1:8080/.
-
-Below is an example of sending a request using `tools/api_client.py`.
-
-```bash
-python -m tools.api_client \
-    --text "Text to be input" \
-    --reference_audio "Path to reference audio" \
-    --reference_text "Text content of the reference audio" \
-    --streaming True
-```
-
-The above command indicates synthesizing the desired audio according to the reference audio information and returning it in a streaming manner.
-
-The following example demonstrates that you can use **multiple** reference audio paths and reference audio texts at once. Separate them with spaces in the command.
-
-```bash
-python -m tools.api_client \
-    --text "Text to input" \
-    --reference_audio "reference audio path1" "reference audio path2" \
-    --reference_text "reference audio text1" "reference audio text2"\
-    --streaming False \
-    --output "generated" \
-    --format "mp3"
-```
-
-The above command synthesizes the desired `MP3` format audio based on the information from multiple reference audios and saves it as `generated.mp3` in the current directory.
-
-You can also use `--reference_id` (only one can be used) instead of `--reference-audio` and `--reference_text`, provided that you create a `references/<your reference_id>` folder in the project root directory, which contains any audio and annotation text. 
-The currently supported reference audio has a maximum total duration of 90 seconds.
-
-
-!!! info 
-    To learn more about available parameters, you can use the command `python -m tools.api_client -h`
-
-## GUI Inference 
-[Download client](https://github.com/AnyaCoder/fish-speech-gui/releases)
-
-## WebUI Inference
-
-You can start the WebUI using the following command:
-
-```bash
-python -m tools.run_webui \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-> If you want to speed up inference, you can add the `--compile` parameter.
-
-!!! note
-    You can save the label file and reference audio file in advance to the `references` folder in the main directory (which you need to create yourself), so that you can directly call them in the WebUI.
-
-!!! note
-    You can use Gradio environment variables, such as `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME` to configure WebUI.
-
-Enjoy!

docs/en/install.md

@@ -0,0 +1,50 @@
+# Introduction
+
+<div>
+<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
+<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
+<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
+<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
+</a>
+</div>
+
+!!! warning
+    We assume no responsibility for any illegal use of the codebase. Please refer to the local laws regarding DMCA (Digital Millennium Copyright Act) and other relevant laws in your area. <br/>
+    This codebase is released under Apache 2.0 license and all models are released under the CC-BY-NC-SA-4.0 license.
+
+## Requirements
+
+- GPU Memory: 12GB (Inference)
+- System: Linux, Windows
+
+## Setup
+
+First, we need to create a conda environment to install the packages.
+
+```bash
+
+conda create -n fish-speech python=3.12
+conda activate fish-speech
+
+pip install sudo apt-get install portaudio19-dev # For pyaudio
+pip install -e . # This will download all rest packages.
+
+apt install libsox-dev ffmpeg # If needed.
+```
+
+!!! warning
+    The `compile` option is not supported on windows and macOS, if you want to run with compile, you need to install trition by yourself.
+
+## Acknowledgements
+
+- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
+- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
+- [GPT VITS](https://github.com/innnky/gpt-vits)
+- [MQTTS](https://github.com/b04901014/MQTTS)
+- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
+- [Transformers](https://github.com/huggingface/transformers)
+- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/en/samples.md

@@ -1,137 +0,0 @@
-# Samples
-
-ver 1.4
-
-## Credits
-Special thanks to [Seed-TTS (2024)](https://bytedancespeech.github.io/seedtts_tech_report/)  for providing the evaluation data for demonstration.
-
-All prompt audio is from the Seed-TTS effect demo page, and all generated audio is from the first generation of fish-speech version 1.4.
-
-## Zero-shot In-context Learning
-<table>
-    <thead>
-    <tr>
-    <th style="vertical-align : middle;text-align: center">Language </th>
-    <th style="vertical-align : middle;text-align: center">Prompt </th>
-    <th style="vertical-align : middle;text-align: center">Same Language Generation</th>
-    <th style="vertical-align : middle;text-align: center">Cross-linugal Generation</th>
-    </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="3">EN</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/4245145269330795065.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2486365921931244890.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Dealing with family secrets is never easy. Yet, sometimes, omission is a form of protection, intending to safeguard some from the harsh truths. One day, I hope you understand the reasons behind my actions. Until then, Anna, please, bear with me.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>处理家庭秘密从来都不是一件容易的事。然而，有时候，隐瞒是一种保护形式，旨在保护一些人免受残酷的真相伤害。有一天，我希望你能理解我行为背后的原因。在那之前，安娜，请容忍我。</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/-9102975986427238220.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>The combinations of different textures and flavors create a perfect harmony. The succulence of the steak, the tartness of the cranberries, the crunch of pine nuts, and creaminess of blue cheese make it a truly delectable delight. Enjoy your culinary adventure!</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>听着你的话，我心里五味杂陈。虽然我愿意一直在你身边，承担一切不幸，但我知道只有让你自己面对，才能真正让你变得更强大。所以，你要记得，无论面对何种困难，都请你坚强，我会在心里一直支持你的。</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="3">ZH</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2648200402409733590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道："我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？"</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, there was a burst of laughter beside me. I looked at them, stood up straight with high spirit, shook the slightly fleshy arms, and smiled lightly, saying, "The flesh on my body is to hide my bursting charm. Otherwise, wouldn't it scare you?"</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/8913957783621352198.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>他闭上眼睛，期望这一切都能过去。然而，当他再次睁开眼睛，眼前的景象让他不禁倒吸一口气。雾气中出现的禁闭岛，陌生又熟悉，充满未知的危险。他握紧拳头，心知他的生活即将发生翻天覆地的改变。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>He closed his eyes, expecting that all of this could pass. However, when he opened his eyes again, the sight in front of him made him couldn't help but take a deep breath. The closed island that appeared in the fog, strange and familiar, was full of unknown dangers. He tightened his fist, knowing that his life was about to undergo earth-shaking changes. </td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2631296891109983590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, the atmosphere became gloomy. At first glance, all the troubles seemed to surround me. I frowned, feeling that pressure, but I know I can't give up, can't admit defeat. So, I took a deep breath, and the voice in my heart told me, "Anyway, must calm down and start again."</td>
-        </tr>
-    </tbody>
-</table>
-
-
-
-## Speaker Fine-tune
-
-<table>
-    <thead>
-    <tr>
-    <th style="text-align: center"> </th>
-    <th style="text-align: center">Text </th>
-    <th style="text-align: center">Generated</th>
-    </tr>
-    </thead>
-    <tbody>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;" rowspan="3">Speaker1</td>
-        <td style="vertical-align : middle;text-align:center;">好呀，哈哈哈哈哈，喜欢笑的人运气都不会差哦，希望你每天笑口常开~</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/4781135337205789117.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">哇！恭喜你中了大乐透，八百万可真不少呢！有什么特别的计划或想法吗？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">哼，你这么问是想请本小姐吃饭吗？如果对象是你的话，那也不是不可以。</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;" rowspan="3">Speaker2</td>
-        <td style="vertical-align : middle;text-align:center;">是呀，他还想换个地球仪哈哈哈，看来给你积累了一些快乐值了，你还想不想再听一个其他的笑话呀？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/-1325430967143158944.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">嘿嘿，你是不是也想拥有甜甜的恋爱呢？《微微一笑很倾城》是你的不二选择，男女主是校花校草类型，他们通过游戏结识，再到两人见面，全程没有一点误会，真的齁甜，想想都忍不住“姨妈笑”~</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">小傻瓜，嗯……算是个很可爱很亲切的名字，有点“独特”哦，不过我有些好奇，你为什么会给我选这个昵称呢？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-    </tbody>
-</table>
-<br>
-
-## Content Editing
-
-<table>
-    <thead>
-    <tr><th style="text-align: center">Language</th>
-    <th style="text-align: center">Original Text</th>
-    <th style="text-align: center">Original Audio</th>
-    <th style="text-align: center">Target Text</th>
-    <th style="text-align: center">Edited Audio</th>
-    </tr></thead>
-    <tbody>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="2">EN</td>
-            <td style="vertical-align : middle;text-align:center;">They can't order me to stop dreaming. If you dream a thing more than once, it's sure to come true. Have faith in your dreams, and someday your rainbow will come shining through.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/2372076002032794455.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">They can't <b>require</b> me to stop <b>imagining.</b> If you envision a thing more than once, it's <b>bound</b> to come <b>about</b>. Have <b>trust</b> in your <b>visions</b>, and someday your <b>radiance</b> will come <b>beaming</b> through.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/2372076002032794455/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;">Are you familiar with it? Slice the steak and place the strips on top, then garnish with the dried cranberries, pine nuts, and blue cheese. I wonder how people rationalise the decision?</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/3347127306902202498.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">Are you <b>acquainted</b> with it? <b>Cut the pork</b> and place the strips on top, then garnish with the dried <b>cherries, almonds,</b> and <b>feta</b> cheese. I <b>query</b> how people <b>justify</b> the <b>choice?</b></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/3347127306902202498/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="2">ZH</td>
-            <td style="vertical-align : middle;text-align:center;">自古以来，庸君最怕党政了，可圣君他就不怕，不但不怕，反能利用。要我说，你就让明珠索额图互相争宠，只要你心里明白，左右逢源，你就能立于不败之地。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/1297014176484007082.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><b>从古至今</b>，庸君最怕<b>朝纲了</b>，可<b>明</b>君他就不怕，不但不怕，反能<b>借助</b>。要我说，你就让<b>李四张三</b>互相争宠，只要你心里<b>清楚</b>，左右<b>周旋</b>，你就能<b>处</b>于不败之<b>境</b>。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/1297014176484007082/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;">对，这就是我，万人敬仰的太乙真人，虽然有点婴儿肥，但也掩不住我逼人的帅气。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/-40165564411515767.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">对，这就是我，<b>众人尊崇</b>的太<b>白金星</b>，虽然有点<b>娃娃脸</b>，但也<b>遮</b>不住我<b>迷人</b>的<b>魅力。</b></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/-40165564411515767/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-    </tbody>
-</table>

docs/en/start_agent.md

@@ -1,77 +0,0 @@
-# Start Agent
-
-## Requirements
-
-- GPU memory: At least 8GB(under quanization), 16GB or more is recommanded.
-- Disk usage: 10GB
-
-## Download Model
-
-You can get the model by:
-
-```bash
-huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
-```
-
-Put them in the 'checkpoints' folder.
-
-You also need the fish-speech model which you can download instructed by [inference](inference.md).
-
-So there will be 2 folder in the checkpoints.
-
-The `checkpoints/fish-speech-1.4` and `checkpoints/fish-agent-v0.1-3b`
-
-## Environment Prepare
-
-If you already have Fish-speech, you can directly use by adding the follow instruction:
-```bash
-pip install cachetools
-```
-
-!!! note
-    Please use the Python version below 3.12 for compile.
-
-If you don't have, please use the below commands to build your environment:
-
-```bash
-sudo apt-get install portaudio19-dev
-
-pip install -e .[stable]
-```
-
-## Launch The Agent Demo.
-
-To build fish-agent, please use the command below under the main folder:
-
-```bash
-python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
-```
-
-The `--compile` args only support Python < 3.12 , which will greatly speed up the token generation.
-
-It won't compile at once (remember).
-
-Then open another terminal and use the command:
-
-```bash
-python -m tools.e2e_webui
-```
-
-This will create a Gradio WebUI on the device.
-
-When you first use the model, it will come to compile (if the `--compile` is True) for a short time, so please wait with patience.
-
-## Gradio Webui
-<p align="center">
-   <img src="../assets/figs/agent_gradio.png" width="75%">
-</p>
-
-Have a good time!
-
-## Performance
-
-Under our test, a 4060 laptop just barely runs, but is very stretched, which is only about 8 tokens/s. The 4090 is around 95 tokens/s under compile, which is what we recommend.
-
-# About Agent
-
-The demo is an early alpha test version, the inference speed needs to be optimised, and there are a lot of bugs waiting to be fixed. If you've found a bug or want to fix it, we'd be very happy to receive an issue or a pull request.

docs/ja/finetune.md

@@ -1,128 +0,0 @@
-# 微調整
-
-明らかに、このページを開いたとき、few-shot 事前トレーニングモデルのパフォーマンスに満足していなかったことでしょう。データセット上でのパフォーマンスを向上させるためにモデルを微調整したいと考えています。
-
-現在のバージョンでは、「LLAMA」部分のみを微調整する必要があります。
-
-## LLAMAの微調整
-### 1. データセットの準備
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 30.1-32.71.lab
-│   └── 30.1-32.71.mp3
-└── SPK2
-    ├── 38.79-40.85.lab
-    └── 38.79-40.85.mp3
-```
-
-データセットを上記の形式に変換し、「data」ディレクトリに配置する必要があります。音声ファイルの拡張子は「.mp3」、「.wav」、または「.flac」にすることができ、注釈ファイルの拡張子は「.lab」にする必要があります。
-
-!!! info
-     標準ファイル `.lab` には、音声の転写テキストのみを含め、特別なフォーマットは必要ありません。例えば、`hi.mp3` で「こんにちは、さようなら」と言っている場合、`hi.lab` ファイルには「こんにちは、さようなら」という一行のテキストを含めるだけです。
-
-!!! warning
-    データセットにラウドネス正規化を適用することをお勧めします。これを行うには、[fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) を使用できます。
-
-    ```bash
-    fap loudness-norm data-raw data --clean
-    ```
-
-
-### 2. セマンティックトークンのバッチ抽出
-
-VQGANの重みをダウンロードしたことを確認してください。まだダウンロードしていない場合は、次のコマンドを実行してください。
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-次に、次のコマンドを実行してセマンティックトークンを抽出できます。
-
-```bash
-python tools/vqgan/extract_vq.py data \
-    --num-workers 1 --batch-size 16 \
-    --config-name "firefly_gan_vq" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-!!! note
-    `--num-workers` と `--batch-size` を調整して抽出速度を上げることができますが、GPUメモリの制限を超えないようにしてください。  
-    VITS形式の場合、`--filelist xxx.list` を使用してファイルリストを指定できます。
-
-このコマンドは、`data`ディレクトリに`.npy`ファイルを作成します。以下のように表示されます。
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 21.15-26.44.npy
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 27.51-29.98.npy
-│   ├── 30.1-32.71.lab
-│   ├── 30.1-32.71.mp3
-│   └── 30.1-32.71.npy
-└── SPK2
-    ├── 38.79-40.85.lab
-    ├── 38.79-40.85.mp3
-    └── 38.79-40.85.npy
-```
-
-### 3. データセットをprotobufにパックする
-
-```bash
-python tools/llama/build_dataset.py \
-    --input "data" \
-    --output "data/protos" \
-    --text-extension .lab \
-    --num-workers 16
-```
-
-コマンドの実行が完了すると、`data`ディレクトリに`quantized-dataset-ft.protos`ファイルが表示されます。
-
-### 4. 最後に、LoRAを使用して微調整する
-
-同様に、`LLAMA`の重みをダウンロードしたことを確認してください。まだダウンロードしていない場合は、次のコマンドを実行してください。
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-最後に、次のコマンドを実行して微調整を開始できます。
-
-```bash
-python fish_speech/train.py --config-name text2semantic_finetune \
-    project=$project \
-    +lora@model.model.lora_config=r_8_alpha_16
-```
-
-!!! note
-    `fish_speech/configs/text2semantic_finetune.yaml` を変更して、`batch_size`、`gradient_accumulation_steps` などのトレーニングパラメータを変更し、GPUメモリに適合させることができます。
-
-!!! note
-    Windowsユーザーの場合、`trainer.strategy.process_group_backend=gloo` を使用して `nccl` の問題を回避できます。
-
-トレーニングが完了したら、[推論](inference.md)セクションを参照し、音声を生成します。
-
-!!! info
-    デフォルトでは、モデルは話者の発話パターンのみを学習し、音色は学習しません。音色の安定性を確保するためにプロンプトを使用する必要があります。
-    音色を学習したい場合は、トレーニングステップ数を増やすことができますが、これにより過学習が発生する可能性があります。
-
-トレーニングが完了したら、推論を行う前にLoRAの重みを通常の重みに変換する必要があります。
-
-```bash
-python tools/llama/merge_lora.py \
-	--lora-config r_8_alpha_16 \
-	--base-weight checkpoints/fish-speech-1.5 \
-	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
-	--output checkpoints/fish-speech-1.5-yth-lora/
-```
-!!! note
-    他のチェックポイントを試すこともできます。要件を満たす最も早いチェックポイントを使用することをお勧めします。これらは通常、分布外（OOD）データでより良いパフォーマンスを発揮します。

docs/ja/index.md

@@ -1,172 +0,0 @@
-# Fish Speech の紹介
-
-<div>
-<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
-<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
-<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
-<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
-</a>
-</div>
-
-!!! warning
-    私たちは、コードベースの違法な使用について一切の責任を負いません。お住まいの地域の DMCA（デジタルミレニアム著作権法）およびその他の関連法を参照してください。 <br/>
-    このコードベースとモデルは、CC-BY-NC-SA-4.0 ライセンス下でリリースされています。
-
-<p align="center">
-   <img src="../assets/figs/diagram.png" width="75%">
-</p>
-
-## 要件
-
-- GPU メモリ: 4GB（推論用）、8GB（ファインチューニング用）
-- システム: Linux、Windows
-
-## Windowsセットアップ
-
-!!! info "注意"
-    Windowsの専門ユーザー以外の方には、GUIを使用してプロジェクトを実行することを強くお勧めします。[GUIはこちら](https://github.com/AnyaCoder/fish-speech-gui).
-
-プロフェッショナルなWindowsユーザーは、WSL2またはDockerを使用してコードベースを実行することを検討してください。
-
-```bash
-# Python 3.10の仮想環境を作成（virtualenvも使用可能）
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# PyTorchをインストール
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
-
-# fish-speechをインストール
-pip3 install -e .
-
-# (アクセラレーションを有効にする) triton-windowsをインストール
-pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
-```
-
-## Linux セットアップ
-
-詳細については、[pyproject.toml](../../pyproject.toml)  を参照してください。
-```bash
-# python 3.10の仮想環境を作成します。virtualenvも使用できます。
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# pytorchをインストールします。
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-
-# (Ubuntu / Debianユーザー) sox + ffmpegをインストールします。
-apt install libsox-dev ffmpeg
-
-# (Ubuntu / Debianユーザー) pyaudio をインストールします。
-apt install build-essential \
-    cmake \
-    libasound-dev \
-    portaudio19-dev \
-    libportaudio2 \
-    libportaudiocpp0
-    
-# fish-speechをインストールします。
-pip3 install -e .[stable]
-
-```
-
-## macos setup
-
-推論をMPS上で行う場合は、`--device mps`フラグを追加してください。
-推論速度の比較は[こちらのPR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772)を参考にしてください。
-
-!!! warning
-    AppleSiliconのデバイスでは、compileオプションに正式に対応していませんので、推論速度が向上する保証はありません。
-
-```bash
-# create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-# install pytorch
-pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-# install fish-speech
-pip install -e .[stable]
-```
-
-## Docker セットアップ
-
-1. NVIDIA Container Toolkit のインストール：
-
-    Docker で GPU を使用してモデルのトレーニングと推論を行うには、NVIDIA Container Toolkit をインストールする必要があります：
-
-    Ubuntu ユーザーの場合：
-
-    ```bash
-    # リポジトリの追加
-    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
-        && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
-            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
-            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-    # nvidia-container-toolkit のインストール
-    sudo apt-get update
-    sudo apt-get install -y nvidia-container-toolkit
-    # Docker サービスの再起動
-    sudo systemctl restart docker
-    ```
-
-    他の Linux ディストリビューションを使用している場合は、以下のインストールガイドを参照してください：[NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)。
-
-2. fish-speech イメージのプルと実行
-
-    ```shell
-    # イメージのプル
-    docker pull fishaudio/fish-speech:latest-dev
-    # イメージの実行
-    docker run -it \
-        --name fish-speech \
-        --gpus all \
-        -p 7860:7860 \
-        fishaudio/fish-speech:latest-dev \
-        zsh
-    # 他のポートを使用する場合は、-p パラメータを YourPort:7860 に変更してください
-    ```
-
-3. モデルの依存関係のダウンロード
-
-    Docker コンテナ内のターミナルにいることを確認し、huggingface リポジトリから必要な `vqgan` と `llama` モデルをダウンロードします。
-
-    ```bash
-    huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-4. 環境変数の設定と WebUI へのアクセス
-
-    Docker コンテナ内のターミナルで、`export GRADIO_SERVER_NAME="0.0.0.0"` と入力して、外部から Docker 内の gradio サービスにアクセスできるようにします。
-    次に、Docker コンテナ内のターミナルで `python tools/run_webui.py` と入力して WebUI サービスを起動します。
-
-    WSL または MacOS の場合は、[http://localhost:7860](http://localhost:7860) にアクセスして WebUI インターフェースを開くことができます。
-
-    サーバーにデプロイしている場合は、localhost をサーバーの IP に置き換えてください。
-
-## 変更履歴
-
-- 2024/12/03: Fish-Speech を 1.5にアップデートし、より多くの言語をサポートするようになりました。オープンソース領域ではSOTA（最先端）となっています。
-- 2024/09/10: Fish-Speech を Ver.1.4 に更新し、データセットのサイズを増加させ、quantizer n_groups を 4 から 8 に変更しました。
-- 2024/07/02: Fish-Speech を Ver.1.2 に更新し、VITS デコーダーを削除し、ゼロショット能力を大幅に強化しました。
-- 2024/05/10: Fish-Speech を Ver.1.1 に更新し、VITS デコーダーを実装して WER を減少させ、音色の類似性を向上させました。
-- 2024/04/22: Fish-Speech Ver.1.0 を完成させ、VQGAN および LLAMA モデルを大幅に修正しました。
-- 2023/12/28: `lora`微調整サポートを追加しました。
-- 2023/12/27: `gradient checkpointing`、`causual sampling`、および`flash-attn`サポートを追加しました。
-- 2023/12/19: webui および HTTP API を更新しました。
-- 2023/12/18: 微調整ドキュメントおよび関連例を更新しました。
-- 2023/12/17: `text2semantic`モデルを更新し、自由音素モードをサポートしました。
-- 2023/12/13: ベータ版をリリースし、VQGAN モデルおよび LLAMA に基づく言語モデル（音素のみサポート）を含みます。
-
-## 謝辞
-
-- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
-- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
-- [GPT VITS](https://github.com/innnky/gpt-vits)
-- [MQTTS](https://github.com/b04901014/MQTTS)
-- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
-- [Transformers](https://github.com/huggingface/transformers)
-- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/ja/inference.md

@@ -1,123 +0,0 @@
-# 推論
-
-推論は、コマンドライン、HTTP API、および Web UI をサポートしています。
-
-!!! note
-    全体として、推論は次のいくつかの部分で構成されています：
-
-    1. VQGANを使用して、与えられた約10秒の音声をエンコードします。
-    2. エンコードされたセマンティックトークンと対応するテキストを例として言語モデルに入力します。
-    3. 新しいテキストが与えられた場合、モデルに対応するセマンティックトークンを生成させます。
-    4. 生成されたセマンティックトークンをVITS / VQGANに入力してデコードし、対応する音声を生成します。
-
-## モデルをダウンロード
-必要な`vqgan`および`llama`モデルを Hugging Face リポジトリからダウンロードします。
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-## コマンドライン推論
-### 1. 音声からプロンプトを生成する：
-
-!!! note
-    モデルにランダムに音声の音色を選ばせる場合、このステップをスキップできます。
-
-!!! warning "将来のバージョンに関する警告"
-    元のパス（tools/vqgan/infernce.py）からアクセスできるインターフェースは残していますが、このインターフェースは将来のいくつかのバージョンで削除される可能性があります。お早めにコードを変更してください。
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "paimon.wav" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-`fake.npy`ファイルが生成されるはずです。
-
-### 2. テキストからセマンティックトークンを生成する：
-
-!!! warning "将来のバージョンに関する警告"
-    元のパス（tools/llama/generate.py）からアクセスできるインターフェースは残していますが、このインターフェースは将来のいくつかのバージョンで削除される可能性があります。お早めにコードを変更してください。
-
-```bash
-python fish_speech/models/text2semantic/inference.py \
-    --text "変換したいテキスト" \
-    --prompt-text "参照テキスト" \
-    --prompt-tokens "fake.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5" \
-    --num-samples 2 \
-    --compile
-```
-
-このコマンドは、作業ディレクトリに`codes_N`ファイルを作成します。ここで、N は 0 から始まる整数です。
-
-!!! note
-    `--compile`を使用して CUDA カーネルを融合し、より高速な推論を実現することができます（約 30 トークン/秒 -> 約 500 トークン/秒）。
-    それに対応して、加速を使用しない場合は、`--compile`パラメータをコメントアウトできます。
-
-!!! info
-    bf16 をサポートしていない GPU の場合、`--half`パラメータを使用する必要があるかもしれません。
-
-### 3. セマンティックトークンから音声を生成する：
-
-#### VQGAN デコーダー
-
-!!! warning "将来のバージョンに関する警告"
-    元のパス（tools/vqgan/infernce.py）からアクセスできるインターフェースは残していますが、このインターフェースは将来のいくつかのバージョンで削除される可能性があります。お早めにコードを変更してください。
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "codes_0.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-## HTTP API 推論
-
-推論のための HTTP API を提供しています。次のコマンドを使用してサーバーを起動できます：
-
-```bash
-python -m tools.api_server \
-    --listen 0.0.0.0:8080 \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-
-> 推論を高速化したい場合は、`--compile` パラメータを追加できます。
-
-その後、`http://127.0.0.1:8080/`で API を表示およびテストできます。
-
-以下は、`tools/api_client.py` を使用してリクエストを送信する例です。
-
-```bash
-python -m tools.api_client \
-    --text "入力するテキスト" \
-    --reference_audio "参照音声へのパス" \
-    --reference_text "参照音声テキスト" \
-    --streaming True
-```
-
-上記のコマンドは、参照音声の情報に基づいて必要な音声を合成し、ストリーミング方式で返すことを示しています。
-
-!!! info
-    使用可能なパラメータの詳細については、コマンド` python -m tools.api_client -h `を使用してください
-
-## WebUI 推論
-
-次のコマンドを使用して WebUI を起動できます：
-
-```bash
-python -m tools.run_webui \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-> 推論を高速化したい場合は、`--compile` パラメータを追加できます。
-
-!!! note
-    ラベルファイルと参照音声ファイルをメインディレクトリの `references` フォルダ（自分で作成する必要があります）に事前に保存しておくことで、WebUI で直接呼び出すことができます。
-
-!!! note
-    Gradio 環境変数（`GRADIO_SHARE`、`GRADIO_SERVER_PORT`、`GRADIO_SERVER_NAME`など）を使用して WebUI を構成できます。
-
-お楽しみください！

docs/ja/install.md

@@ -0,0 +1,50 @@
+# 紹介
+
+<div>
+<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
+<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
+<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
+<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
+</a>
+</div>
+
+!!! warning
+    このコードベースの違法な使用について、当方は一切の責任を負いません。お住まいの地域のDMCA（デジタルミレニアム著作権法）およびその他の関連法規をご参照ください。<br/>
+    このコードベースはApache 2.0ライセンスの下でリリースされ、すべてのモデルはCC-BY-NC-SA-4.0ライセンスの下でリリースされています。
+
+## システム要件
+
+- GPU メモリ：12GB（推論）
+- システム：Linux、Windows
+
+## セットアップ
+
+まず、パッケージをインストールするためのconda環境を作成する必要があります。
+
+```bash
+
+conda create -n fish-speech python=3.12
+conda activate fish-speech
+
+pip install sudo apt-get install portaudio19-dev # pyaudio用
+pip install -e . # これにより残りのパッケージがすべてダウンロードされます。
+
+apt install libsox-dev ffmpeg # 必要に応じて。
+```
+
+!!! warning
+    `compile`オプションはWindowsとmacOSでサポートされていません。compileで実行したい場合は、tritionを自分でインストールする必要があります。
+
+## 謝辞
+
+- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
+- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
+- [GPT VITS](https://github.com/innnky/gpt-vits)
+- [MQTTS](https://github.com/b04901014/MQTTS)
+- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
+- [Transformers](https://github.com/huggingface/transformers)
+- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/ja/samples.md

@@ -1,225 +0,0 @@
-# サンプル
-
-v1.4デモは[こちら](https://speech.fish.audio/samples/)に更新されています
-
-v1.2のサンプルは[Bilibili](https://www.bilibili.com/video/BV1wz421B71D/)で利用可能です。
-
-以下のサンプルはv1.1モデルからのものです。
-
-## 中国語の文1
-```
-人間灯火倒映湖中，她的渴望让静水泛起涟漪。若代价只是孤独，那就让这份愿望肆意流淌。
-流入她所注视的世间，也流入她如湖水般澄澈的目光。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ナヒーダ (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>鍾離 (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>フリナ (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>ランダム話者1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/4_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>ランダム話者2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/5_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## 中国語の文2
-```
-你们这个是什么群啊，你们这是害人不浅啊你们这个群！谁是群主，出来！真的太过分了。你们搞这个群干什么？
-我儿子每一科的成绩都不过那个平均分呐，他现在初二，你叫我儿子怎么办啊？他现在还不到高中啊？
-你们害死我儿子了！快点出来你这个群主！再这样我去报警了啊！我跟你们说你们这一帮人啊，一天到晚啊，
-搞这些什么游戏啊，动漫啊，会害死你们的，你们没有前途我跟你说。你们这九百多个人，好好学习不好吗？
-一天到晚在上网。有什么意思啊？麻烦你重视一下你们的生活的目标啊？有一点学习目标行不行？一天到晚上网是不是人啊？
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ナヒーダ (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/6_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>ランダム話者</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/7_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## 中国語の文3
-```
-大家好，我是 Fish Audio 开发的开源文本转语音模型。经过十五万小时的数据训练，
-我已经能够熟练掌握中文、日语和英语，我的语言处理能力接近人类水平，声音表现形式丰富多变。
-作为一个仅有亿级参数的模型，我相信社区成员能够在个人设备上轻松运行和微调，让我成为您的私人语音助手。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ランダム話者</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/8_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 英語の文1
-
-```
-In the realm of advanced technology, the evolution of artificial intelligence stands as a 
-monumental achievement. This dynamic field, constantly pushing the boundaries of what 
-machines can do, has seen rapid growth and innovation. From deciphering complex data 
-patterns to driving cars autonomously, AI's applications are vast and diverse.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ランダム話者1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>ランダム話者2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 英語の文2
-```
-Hello everyone, I am an open-source text-to-speech model developed by 
-Fish Audio. After training with 150,000 hours of data, I have become proficient 
-in Chinese, Japanese, and English, and my language processing abilities 
-are close to human level. My voice is capable of a wide range of expressions. 
-As a model with only hundreds of millions of parameters, I believe community 
-members can easily run and fine-tune me on their personal devices, allowing 
-me to serve as your personal voice assistant.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ランダム話者</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 日本語の文1
-
-```
-先進技術の領域において、人工知能の進化は画期的な成果として立っています。常に機械ができることの限界を
-押し広げているこのダイナミックな分野は、急速な成長と革新を見せています。複雑なデータパターンの解読か
-ら自動運転車の操縦まで、AIの応用は広範囲に及びます。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ランダム話者1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>ランダム話者2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 日本語の文2
-```
-皆さん、こんにちは。私はフィッシュオーディオによって開発されたオープンソースのテ
-キストから音声への変換モデルです。15万時間のデータトレーニングを経て、
-中国語、日本語、英語を熟知しており、言語処理能力は人間に近いレベルです。
-声の表現も多彩で豊かです。数億のパラメータを持つこのモデルは、コミュニティ
-のメンバーが個人のデバイスで簡単に実行し、微調整することができると
-信じています。これにより、私を個人の音声アシスタントとして活用できます。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>話者</th>
-        <th>入力音声</th>
-        <th>合成音声</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>ランダム話者</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>

docs/ja/start_agent.md

@@ -1,80 +0,0 @@
-# エージェントの開始
-
-!!! note
-    もしあなたがネイティブ・スピーカーで、翻訳に問題があるとお感じでしたら、issueかpull requestをお送りください！
-
-## 要件
-
-- GPUメモリ: 最低8GB(量子化使用時)、16GB以上推奨
-- ディスク使用量: 10GB
-
-## モデルのダウンロード
-
-以下のコマンドでモデルを取得できます:
-
-```bash
-huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
-```
-
-これらを'checkpoints'フォルダに配置してください。
-
-また、[inference](inference.md)の手順に従ってfish-speechモデルもダウンロードする必要があります。
-
-checkpointsには2つのフォルダが必要です。
-
-`checkpoints/fish-speech-1.4`と`checkpoints/fish-agent-v0.1-3b`です。
-
-## 環境準備
-
-すでにFish-speechをお持ちの場合は、以下の指示を追加するだけで直接使用できます:
-```bash
-pip install cachetools
-```
-
-!!! note
-    コンパイルにはPythonバージョン3.12未満を使用してください。
-
-お持ちでない場合は、以下のコマンドで環境を構築してください:
-
-```bash
-sudo apt-get install portaudio19-dev
-
-pip install -e .[stable]
-```
-
-## エージェントデモの起動
-
-fish-agentを構築するには、メインフォルダで以下のコマンドを使用してください:
-
-```bash
-python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
-```
-
-`--compile`引数はPython < 3.12でのみサポートされており、トークン生成を大幅に高速化します。
-
-一度にコンパイルは行われません(覚えておいてください)。
-
-次に、別のターミナルを開いて以下のコマンドを使用します:
-
-```bash
-python -m tools.e2e_webui
-```
-
-これにより、デバイス上にGradio WebUIが作成されます。
-
-モデルを初めて使用する際は、(`--compile`がTrueの場合)しばらくコンパイルが行われますので、お待ちください。
-
-## Gradio Webui
-<p align="center">
-   <img src="../../assets/figs/agent_gradio.png" width="75%">
-</p>
-
-お楽しみください！
-
-## パフォーマンス
-
-テストでは、4060搭載のラップトップではかろうじて動作しますが、非常に厳しい状態で、約8トークン/秒程度です。4090ではコンパイル時に約95トークン/秒で、これが推奨環境です。
-
-# エージェントについて
-
-このデモは初期アルファテストバージョンで、推論速度の最適化が必要で、修正を待つバグが多数あります。バグを発見した場合や修正したい場合は、issueやプルリクエストをいただけると大変嬉しく思います。

docs/ko/finetune.md

@@ -1,128 +0,0 @@
-# 파인튜닝
-
-이 페이지를 열었다는 것은, 사전 학습된 퓨샷(Few-shot) 모델의 성능에 만족하지 못했다는 의미일 것입니다. 데이터셋의 성능을 향상시키기 위해 모델을 파인튜닝하고 싶으시겠죠.
-
-현재 버전에서는 'LLAMA' 부분만 파인튜닝하시면 됩니다.
-
-## LLAMA 파인튜닝
-### 1. 데이터셋 준비
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 30.1-32.71.lab
-│   └── 30.1-32.71.mp3
-└── SPK2
-    ├── 38.79-40.85.lab
-    └── 38.79-40.85.mp3
-```
-
-위와 같은 형식으로 데이터셋을 변환하여 `data` 디렉토리 안에 배치하세요. 오디오 파일의 확장자는 `.mp3`, `.wav`, `.flac` 중 하나여야 하며, 주석 파일은 `.lab` 확장자를 사용해야 합니다.
-
-!!! info "데이터셋 형식"
-    `.lab` 주석 파일은 오디오의 전사 내용만 포함하면 되며, 특별한 형식이 필요하지 않습니다. 예를 들어, `hi.mp3`에서 "Hello, goodbye"라는 대사를 말한다면, `hi.lab` 파일에는 "Hello, goodbye"라는 한 줄의 텍스트만 있어야 합니다.
-
-!!! warning
-    데이터셋에 대한 음량 정규화(loudness normalization)를 적용하는 것이 좋습니다. 이를 위해 [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess)를 사용할 수 있습니다.
-
-    ```bash
-    fap loudness-norm data-raw data --clean
-    ```
-
-### 2. 시맨틱 토큰 배치 추출
-
-VQGAN 가중치를 다운로드했는지 확인하세요. 다운로드하지 않았다면 아래 명령어를 실행하세요:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-이후 시맨틱 토큰을 추출하기 위해 아래 명령어를 실행하세요:
-
-```bash
-python tools/vqgan/extract_vq.py data \
-    --num-workers 1 --batch-size 16 \
-    --config-name "firefly_gan_vq" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-!!! note
-    추출 속도를 높이기 위해 `--num-workers`와 `--batch-size` 값을 조정할 수 있지만, GPU 메모리 한도를 초과하지 않도록 주의하세요.  
-    VITS 형식의 경우, `--filelist xxx.list`를 사용하여 파일 목록을 지정할 수 있습니다.
-
-이 명령을 실행하면 `data` 디렉토리 안에 `.npy` 파일이 생성됩니다. 다음과 같이 표시됩니다:
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 21.15-26.44.npy
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 27.51-29.98.npy
-│   ├── 30.1-32.71.lab
-│   ├── 30.1-32.71.mp3
-│   └── 30.1-32.71.npy
-└── SPK2
-    ├── 38.79-40.85.lab
-    ├── 38.79-40.85.mp3
-    └── 38.79-40.85.npy
-```
-
-### 3. 데이터셋을 protobuf로 패킹
-
-```bash
-python tools/llama/build_dataset.py \
-    --input "data" \
-    --output "data/protos" \
-    --text-extension .lab \
-    --num-workers 16
-```
-
-명령이 완료되면 `data` 디렉토리 안에 `quantized-dataset-ft.protos` 파일이 생성됩니다.
-
-### 4. 마지막으로, LoRA를 이용한 파인튜닝
-
-마찬가지로, `LLAMA` 가중치를 다운로드했는지 확인하세요. 다운로드하지 않았다면 아래 명령어를 실행하세요:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-마지막으로, 아래 명령어를 실행하여 파인튜닝을 시작할 수 있습니다:
-
-```bash
-python fish_speech/train.py --config-name text2semantic_finetune \
-    project=$project \
-    +lora@model.model.lora_config=r_8_alpha_16
-```
-
-!!! note
-    `batch_size`, `gradient_accumulation_steps` 등의 학습 매개변수를 GPU 메모리에 맞게 조정하려면 `fish_speech/configs/text2semantic_finetune.yaml` 파일을 수정할 수 있습니다.
-
-!!! note
-    Windows 사용자의 경우, `nccl` 문제를 피하려면 `trainer.strategy.process_group_backend=gloo`를 사용할 수 있습니다.
-
-훈련이 완료되면 [추론](inference.md) 섹션을 참고하여 음성을 생성할 수 있습니다.
-
-!!! info
-    기본적으로 모델은 화자의 말하는 패턴만 학습하고 음색은 학습하지 않습니다. 음색의 안정성을 위해 프롬프트를 사용해야 합니다.
-    음색을 학습하려면 훈련 단계를 늘릴 수 있지만, 이는 과적합의 위험을 초래할 수 있습니다.
-
-훈련이 끝나면 LoRA 가중치를 일반 가중치로 변환한 후에 추론을 수행해야 합니다.
-
-```bash
-python tools/llama/merge_lora.py \
-	--lora-config r_8_alpha_16 \
-	--base-weight checkpoints/fish-speech-1.5 \
-	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
-	--output checkpoints/fish-speech-1.5-yth-lora/
-```
-
-!!! note
-    다른 체크포인트도 시도해 볼 수 있습니다. 요구 사항에 맞는 가장 초기 체크포인트를 사용하는 것이 좋습니다. 이들은 종종 분포 밖(OOD) 데이터에서 더 좋은 성능을 발휘합니다.

docs/ko/index.md

@@ -1,171 +0,0 @@
-# 소개
-
-<div>
-<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
-<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
-<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
-<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
-</a>
-</div>
-
-!!! warning
-    이 코드베이스의 불법적인 사용에 대해서는 책임을 지지 않습니다. DMCA(Digital Millennium Copyright Act) 및 해당 지역의 관련 법률을 참조하십시오. <br/>
-    이 코드베이스와 모든 모델은 CC-BY-NC-SA-4.0 라이선스에 따라 배포됩니다.
-
-<p align="center">
-   <img src="../assets/figs/diagram.png" width="75%">
-</p>
-
-## 요구 사항
-
-- GPU 메모리: 4GB (추론용), 8GB (파인튜닝용)
-- 시스템: Linux, Windows
-
-## Windows 설정
-
-!!! info "주의"
-    Windows 전문가가 아닌 사용자는 GUI를 통해 프로젝트를 실행할 것을 강력히 권장합니다. [GUI는 여기에서](https://github.com/AnyaCoder/fish-speech-gui) 확인하세요.
-
-고급 Windows 사용자는 WSL2 또는 Docker를 사용하여 코드베이스를 실행하는 것을 고려할 수 있습니다.
-
-```bash
-# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# pytorch 설치
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
-
-# fish-speech 설치
-pip3 install -e .
-
-# (가속 활성화) triton-windows 설치
-pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
-```
-
-## Linux 설정
-
-[pyproject.toml](../../pyproject.toml)에서 자세한 내용을 확인하세요.
-```bash
-# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# (Ubuntu / Debian 사용자) sox + ffmpeg 설치
-apt install libsox-dev ffmpeg 
-
-# (Ubuntu / Debian 사용자) pyaudio 설치
-apt install build-essential \
-    cmake \
-    libasound-dev \
-    portaudio19-dev \
-    libportaudio2 \
-    libportaudiocpp0
-
-# pytorch 설치
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-
-# fish-speech 설치
-pip3 install -e .[stable]
-```
-
-## macos 설정
-
-MPS에서 추론을 수행하려면 `--device mps` 플래그를 추가하세요.
-추론 속도 비교는 [이 PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772)을 참조하십시오.
-
-!!! warning
-    Apple Silicon 장치에서는 `compile` 옵션이 공식적으로 지원되지 않으므로 추론 속도가 향상된다는 보장은 없습니다.
-
-```bash
-# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-# pytorch 설치
-pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-# fish-speech 설치
-pip install -e .[stable]
-```
-
-## Docker 설정
-
-1. NVIDIA Container Toolkit 설치:
-
-    Docker에서 모델 훈련 및 추론에 GPU를 사용하려면 NVIDIA Container Toolkit을 설치해야 합니다:
-
-    Ubuntu 사용자:
-
-    ```bash
-    # 저장소 추가
-    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
-        && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
-            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
-            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-    # nvidia-container-toolkit 설치
-    sudo apt-get update
-    sudo apt-get install -y nvidia-container-toolkit
-    # Docker 서비스 재시작
-    sudo systemctl restart docker
-    ```
-
-    다른 Linux 배포판 사용자는: [NVIDIA Container Toolkit 설치 가이드](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)를 참조하십시오.
-
-2. fish-speech 이미지 가져오기 및 실행
-
-    ```bash
-    # 이미지 가져오기
-    docker pull fishaudio/fish-speech:latest-dev
-    # 이미지 실행
-    docker run -it \
-        --name fish-speech \
-        --gpus all \
-        -p 7860:7860 \
-        fishaudio/fish-speech:latest-dev \
-        zsh
-    # 다른 포트를 사용하려면 -p 매개변수를 YourPort:7860으로 수정하세요
-    ```
-
-3. 모델 종속성 다운로드
-
-    Docker 컨테이너 내부의 터미널에서 아래 명령어를 사용하여 필요한 `vqgan` 및 `llama` 모델을 Huggingface 리포지토리에서 다운로드합니다.
-
-    ```bash
-    huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-4. 환경 변수 설정 및 WebUI 접근
-
-    Docker 컨테이너 내부의 터미널에서 `export GRADIO_SERVER_NAME="0.0.0.0"`를 입력하여 Docker 내부에서 Gradio 서비스에 외부 접근을 허용합니다.
-    이후, 터미널에서 `python tools/run_webui.py` 명령어를 입력하여 WebUI 서비스를 시작합니다.
-
-    WSL 또는 macOS를 사용하는 경우 [http://localhost:7860](http://localhost:7860)에서 WebUI 인터페이스를 열 수 있습니다.
-
-    서버에 배포된 경우, localhost를 서버의 IP로 교체하세요.
-
-## 변경 사항
-
-- 2024/12/03: Fish-Speech를 1.5 로 업데이트하여 더 많은 언어를 지원하게 되었으며, 오픈소스 영역에서 SOTA(최첨단 기술)에 속합니다.
-- 2024/09/10: Fish-Speech 1.4 버전으로 업데이트, 데이터셋 크기 증가 및 양자화기의 n_groups를 4에서 8로 변경.
-- 2024/07/02: Fish-Speech 1.2 버전으로 업데이트, VITS 디코더 제거 및 제로샷 능력 크게 향상.
-- 2024/05/10: Fish-Speech 1.1 버전으로 업데이트, WER 감소 및 음색 유사성을 개선하기 위해 VITS 디코더 구현.
-- 2024/04/22: Fish-Speech 1.0 버전 완료, VQGAN 및 LLAMA 모델 대폭 수정.
-- 2023/12/28: `lora` 파인튜닝 지원 추가.
-- 2023/12/27: `gradient checkpointing`, `causual sampling`, 및 `flash-attn` 지원 추가.
-- 2023/12/19: WebUI 및 HTTP API 업데이트.
-- 2023/12/18: 파인튜닝 문서 및 관련 예시 업데이트.
-- 2023/12/17: `text2semantic` 모델 업데이트, 음소 없는 모드 지원.
-- 2023/12/13: 베타 버전 출시, VQGAN 모델 및 LLAMA 기반 언어 모델(음소 지원만 포함).
-
-## 감사의 말
-
-- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
-- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
-- [GPT VITS](https://github.com/innnky/gpt-vits)
-- [MQTTS](https://github.com/b04901014/MQTTS)
-- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
-- [Transformers](https://github.com/huggingface/transformers)
-- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/ko/inference.md

@@ -1,143 +0,0 @@
-# 추론
-
-추론은 명령줄, HTTP API, 그리고 웹 UI에서 지원됩니다.
-
-!!! note
-    전체 추론 과정은 다음의 여러 단계로 구성됩니다:
-
-    1. VQGAN을 사용하여 약 10초 분량의 음성을 인코딩합니다.
-    2. 인코딩된 시맨틱 토큰과 해당 텍스트를 예시로 언어 모델에 입력합니다.
-    3. 새로운 텍스트를 입력하면, 모델이 해당하는 시맨틱 토큰을 생성합니다.
-    4. 생성된 시맨틱 토큰을 VITS / VQGAN에 입력하여 음성을 디코딩하고 생성합니다.
-
-## 모델 다운로드
-필요한 `vqgan` 및 `llama` 모델을 Hugging Face 리포지토리에서 다운로드하세요.
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-## 명령줄 추론
-### 1. 음성에서 프롬프트 생성:
-
-!!! note
-    모델이 음색을 무작위로 선택하도록 하려면 이 단계를 건너뛸 수 있습니다.
-
-!!! warning "향후 버전 경고"
-    원래 경로(tools/vqgan/infernce.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "paimon.wav" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-이 명령을 실행하면 `fake.npy` 파일을 얻게 됩니다.
-
-### 2. 텍스트에서 시맨틱 토큰 생성:
-
-!!! warning "향후 버전 경고"
-    원래 경로(tools/llama/generate.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
-
-```bash
-python fish_speech/models/text2semantic/inference.py \
-    --text "변환할 텍스트" \
-    --prompt-text "참고할 텍스트" \
-    --prompt-tokens "fake.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5" \
-    --num-samples 2 \
-    --compile
-```
-
-이 명령을 실행하면 작업 디렉토리에 `codes_N` 파일이 생성되며, N은 0부터 시작하는 정수입니다.
-
-!!! note
-    빠른 추론을 위해 `--compile` 옵션을 사용하여 CUDA 커널을 결합할 수 있습니다 (~초당 30 토큰 -> ~초당 500 토큰).
-    `--compile` 매개변수를 주석 처리하여 가속화 옵션을 사용하지 않을 수도 있습니다.
-
-!!! info
-    bf16을 지원하지 않는 GPU의 경우 `--half` 매개변수를 사용해야 할 수 있습니다.
-
-### 3. 시맨틱 토큰에서 음성 생성:
-
-#### VQGAN 디코더
-
-!!! warning "향후 버전 경고"
-    원래 경로(tools/vqgan/infernce.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "codes_0.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-## HTTP API 추론
-
-추론을 위한 HTTP API를 제공하고 있습니다. 아래의 명령어로 서버를 시작할 수 있습니다:
-
-```bash
-python -m tools.api_server \
-    --listen 0.0.0.0:8080 \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-
-추론 속도를 높이고 싶다면 `--compile` 매개변수를 추가할 수 있습니다.
-
-이후, http://127.0.0.1:8080/ 에서 API를 확인하고 테스트할 수 있습니다.
-
-아래는 `tools/api_client.py`를 사용하여 요청을 보내는 예시입니다.
-
-```bash
-python -m tools.api_client \
-    --text "입력할 텍스트" \
-    --reference_audio "참고 음성 경로" \
-    --reference_text "참고 음성의 텍스트 내용" \
-    --streaming True
-```
-
-위 명령은 참고 음성 정보를 바탕으로 원하는 음성을 합성하고, 스트리밍 방식으로 반환합니다.
-
-다음 예시는 여러 개의 참고 음성 경로와 텍스트를 한꺼번에 사용할 수 있음을 보여줍니다. 명령에서 공백으로 구분하여 입력합니다.
-
-```bash
-python -m tools.api_client \
-    --text "입력할 텍스트" \
-    --reference_audio "참고 음성 경로1" "참고 음성 경로2" \
-    --reference_text "참고 음성 텍스트1" "참고 음성 텍스트2"\
-    --streaming False \
-    --output "generated" \
-    --format "mp3"
-```
-
-위 명령어는 여러 참고 음성 정보를 바탕으로 `MP3` 형식의 음성을 합성하여, 현재 디렉토리에 `generated.mp3`로 저장합니다.
-
-`--reference_audio`와 `--reference_text` 대신에 `--reference_id`(하나만 사용 가능)를 사용할 수 있습니다. 프로젝트 루트 디렉토리에 `references/<your reference_id>` 폴더를 만들어 해당 음성과 주석 텍스트를 넣어야 합니다. 참고 음성은 최대 90초까지 지원됩니다.
-
-!!! info 
-    제공되는 파라미터는 `python -m tools.api_client -h`를 사용하여 확인할 수 있습니다.
-
-## GUI 추론 
-[클라이언트 다운로드](https://github.com/AnyaCoder/fish-speech-gui/releases)
-
-## WebUI 추론
-
-다음 명령으로 WebUI를 시작할 수 있습니다:
-
-```bash
-python -m tools.run_webui \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-
-> 추론 속도를 높이고 싶다면 `--compile` 매개변수를 추가할 수 있습니다.
-
-!!! note
-    라벨 파일과 참고 음성 파일을 미리 메인 디렉토리의 `references` 폴더에 저장해 두면, WebUI에서 바로 호출할 수 있습니다. (해당 폴더는 직접 생성해야 합니다.)
-
-!!! note
-    WebUI를 구성하기 위해 `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME`과 같은 Gradio 환경 변수를 사용할 수 있습니다.
-
-즐기세요!

docs/ko/install.md

@@ -0,0 +1,50 @@
+# 소개
+
+<div>
+<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
+<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
+<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
+<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
+</a>
+</div>
+
+!!! warning
+    코드베이스의 불법적인 사용에 대해서는 일체 책임을 지지 않습니다. 귀하의 지역의 DMCA(디지털 밀레니엄 저작권법) 및 기타 관련 법률을 참고하시기 바랍니다. <br/>
+    이 코드베이스는 Apache 2.0 라이선스 하에 배포되며, 모든 모델은 CC-BY-NC-SA-4.0 라이선스 하에 배포됩니다.
+
+## 시스템 요구사항
+
+- GPU 메모리: 12GB (추론)
+- 시스템: Linux, Windows
+
+## 설치
+
+먼저 패키지를 설치하기 위한 conda 환경을 만들어야 합니다.
+
+```bash
+
+conda create -n fish-speech python=3.12
+conda activate fish-speech
+
+pip install sudo apt-get install portaudio19-dev # pyaudio용
+pip install -e . # 나머지 모든 패키지를 다운로드합니다.
+
+apt install libsox-dev ffmpeg # 필요한 경우.
+```
+
+!!! warning
+    `compile` 옵션은 Windows와 macOS에서 지원되지 않습니다. compile로 실행하려면 trition을 직접 설치해야 합니다.
+
+## 감사의 말
+
+- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
+- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
+- [GPT VITS](https://github.com/innnky/gpt-vits)
+- [MQTTS](https://github.com/b04901014/MQTTS)
+- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
+- [Transformers](https://github.com/huggingface/transformers)
+- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/ko/samples.md

@@ -1,137 +0,0 @@
-# 샘플
-
-ver 1.4
-
-## Credits
-[Seed-TTS (2024)](https://bytedancespeech.github.io/seedtts_tech_report/)에 감사드리며, 평가 데이터를 제공해 주셔서 이 데모를 완성할 수 있었습니다.
-
-모든 프롬프트 음성은 Seed-TTS 효과 데모 페이지에서 가져왔으며, 모든 생성된 음성은 fish-speech 버전 1.4에서 첫 번째로 생성된 것입니다.
-
-## 제로샷 인컨텍스트 학습
-- TODO: 한국어 제로샷 인컨텍스트 학습 샘플 추가. (현재는 영어와 중국어 데모만 제공됩니다.)
-
-<table>
-    <thead>
-    <tr>
-    <th style="vertical-align : middle;text-align: center">언어</th>
-    <th style="vertical-align : middle;text-align: center">프롬프트</th>
-    <th style="vertical-align : middle;text-align: center">동일 언어 생성</th>
-    <th style="vertical-align : middle;text-align: center">교차 언어 생성</th>
-    </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="3">EN</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/4245145269330795065.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2486365921931244890.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Dealing with family secrets is never easy. Yet, sometimes, omission is a form of protection, intending to safeguard some from the harsh truths. One day, I hope you understand the reasons behind my actions. Until then, Anna, please, bear with me.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>处理家庭秘密从来都不是一件容易的事。然而，有时候，隐瞒是一种保护形式，旨在保护一些人免受残酷的真相伤害。有一天，我希望你能理解我行为背后的原因。在那之前，安娜，请容忍我。</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/-9102975986427238220.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>The combinations of different textures and flavors create a perfect harmony. The succulence of the steak, the tartness of the cranberries, the crunch of pine nuts, and creaminess of blue cheese make it a truly delectable delight. Enjoy your culinary adventure!</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>听着你的话，我心里五味杂陈。虽然我愿意一直在你身边，承担一切不幸，但我知道只有让你自己面对，才能真正让你变得更强大。所以，你要记得，无论面对何种困难，都请你坚强，我会在心里一直支持你的。</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="3">ZH</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2648200402409733590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道："我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？"</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, there was a burst of laughter beside me. I looked at them, stood up straight with high spirit, shook the slightly fleshy arms, and smiled lightly, saying, "The flesh on my body is to hide my bursting charm. Otherwise, wouldn't it scare you?"</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/8913957783621352198.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>他闭上眼睛，期望这一切都能过去。然而，当他再次睁开眼睛，眼前的景象让他不禁倒吸一口气。雾气中出现的禁闭岛，陌生又熟悉，充满未知的危险。他握紧拳头，心知他的生活即将发生翻天覆地的改变。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>He closed his eyes, expecting that all of this could pass. However, when he opened his eyes again, the sight in front of him made him couldn't help but take a deep breath. The closed island that appeared in the fog, strange and familiar, was full of unknown dangers. He tightened his fist, knowing that his life was about to undergo earth-shaking changes.</td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2631296891109983590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, the atmosphere became gloomy. At first glance, all the troubles seemed to surround me. I frowned, feeling that pressure, but I know I can't give up, can't admit defeat. So, I took a deep breath, and the voice in my heart told me, "Anyway, must calm down and start again."</td>
-        </tr>
-    </tbody>
-</table>
-
-## 화자 파인튜닝
-
-<table>
-    <thead>
-    <tr>
-    <th style="text-align: center"> </th>
-    <th style="text-align: center">텍스트</th>
-    <th style="text-align: center">생성된 음성</th>
-    </tr>
-    </thead>
-    <tbody>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;" rowspan="3">화자1</td>
-        <td style="vertical-align : middle;text-align:center;">好呀，哈哈哈哈哈，喜欢笑的人运气都不会差哦，希望你每天笑口常开~</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/4781135337205789117.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">哇！恭喜你中了大乐透，八百万可真不少呢！有什么特别的计划或想法吗？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">哼，你这么问是想请本小姐吃饭吗？如果对象是你的话，那也不是不可以。</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;" rowspan="3">화자2</td>
-        <td style="vertical-align : middle;text-align:center;">是呀，他还想换个地球仪哈哈哈，看来给你积累了一些快乐值了，你还想不想再听一个其他的笑话呀？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/-1325430967143158944.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">嘿嘿，你是不是也想拥有甜甜的恋爱呢？《微微一笑很倾城》是你的不二选择，男女主是校花校草类型，他们通过游戏结识，再到两人见面，全程没有一点误会，真的齁甜，想想都忍不住“姨妈笑”~</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-        <td style="vertical-align : middle;text-align:center;">小傻瓜，嗯……算是个很可爱很亲切的名字，有点“独特”哦，不过我有些好奇，你为什么会给我选这个昵称呢？</td>
-        <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-    </tbody>
-</table>
-<br>
-
-## 콘텐츠 편집
-
-<table>
-    <thead>
-    <tr><th style="text-align: center">언어</th>
-    <th style="text-align: center">원본 텍스트</th>
-    <th style="text-align: center">원본 음성</th>
-    <th style="text-align: center">목표 텍스트</th>
-    <th style="text-align: center">편집된 음성</th>
-    </tr></thead>
-    <tbody>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="2">EN</td>
-            <td style="vertical-align : middle;text-align:center;">They can't order me to stop dreaming. If you dream a thing more than once, it's sure to come true. Have faith in your dreams, and someday your rainbow will come shining through.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/2372076002032794455.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">They can't <b>require</b> me to stop <b>imagining.</b> If you envision a thing more than once, it's <b>bound</b> to come <b>about</b>. Have <b>trust</b> in your <b>visions</b>, and someday your <b>radiance</b> will come <b>beaming</b> through.</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/2372076002032794455/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;">Are you familiar with it? Slice the steak and place the strips on top, then garnish with the dried cranberries, pine nuts, and blue cheese. I wonder how people rationalise the decision?</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/3347127306902202498.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">Are you <b>acquainted</b> with it? <b>Cut the pork</b> and place the strips on top, then garnish with the dried <b>cherries, almonds,</b> and <b>feta</b> cheese. I <b>query</b> how people <b>justify</b> the <b>choice?</b></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/3347127306902202498/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;" rowspan="2">ZH</td>
-            <td style="vertical-align : middle;text-align:center;">自古以来，庸君最怕党政了，可圣君他就不怕，不但不怕，反能利用。要我说，你就让明珠索额图互相争宠，只要你心里明白，左右逢源，你就能立于不败之地。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/1297014176484007082.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;"><b>从古至今</b>，庸君最怕<b>朝纲了</b>，可<b>明</b>君他就不怕，不但不怕，反能<b>借助</b>。要我说，你就让<b>李四张三</b>互相争宠，只要你心里<b>清楚</b>，左右<b>周旋</b>，你就能<b>处</b>于不败之<b>境</b>。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/1297014176484007082/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-        <tr>
-            <td style="vertical-align : middle;text-align:center;">对，这就是我，万人敬仰的太乙真人，虽然有点婴儿肥，但也掩不住我逼人的帅气。</td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/-40165564411515767.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-            <td style="vertical-align : middle;text-align:center;">对，这就是我，<b>众人尊崇</b>的太<b>白金星</b>，虽然有点<b>娃娃脸</b>，但也<b>遮</b>不住我<b>迷人</b>的<b>魅力。</b></td>
-            <td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/-40165564411515767/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
-        </tr>
-    </tbody>
-</table>

docs/ko/start_agent.md

@@ -1,80 +0,0 @@
-# 에이전트 시작하기
-
-!!! note
-    전체 문서는 claude3.5 Sonnet에 의해 번역되었으며, 원어민인 경우 번역에 문제가 있다고 생각되면 이슈나 풀 리퀘스트를 보내주셔서 대단히 감사합니다!
-
-## 요구사항
-
-- GPU 메모리: 최소 8GB(양자화 사용 시), 16GB 이상 권장
-- 디스크 사용량: 10GB
-
-## 모델 다운로드
-
-다음 명령어로 모델을 받을 수 있습니다:
-
-```bash
-huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
-```
-
-'checkpoints' 폴더에 파일들을 넣으세요.
-
-또한 [inference](inference.md)에 설명된 대로 fish-speech 모델도 다운로드해야 합니다.
-
-checkpoints에는 2개의 폴더가 있어야 합니다.
-
-`checkpoints/fish-speech-1.4`와 `checkpoints/fish-agent-v0.1-3b`입니다.
-
-## 환경 준비
-
-이미 Fish-speech가 있다면 다음 명령어를 추가하여 바로 사용할 수 있습니다:
-```bash
-pip install cachetools
-```
-
-!!! 참고
-    컴파일을 위해 Python 3.12 미만 버전을 사용해 주세요.
-
-없다면 아래 명령어를 사용하여 환경을 구축하세요:
-
-```bash
-sudo apt-get install portaudio19-dev
-
-pip install -e .[stable]
-```
-
-## 에이전트 데모 실행
-
-fish-agent를 구축하려면 메인 폴더에서 아래 명령어를 사용하세요:
-
-```bash
-python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
-```
-
-`--compile` 인자는 Python < 3.12에서만 지원되며, 토큰 생성 속도를 크게 향상시킵니다.
-
-한 번에 컴파일되지 않습니다(기억해 두세요).
-
-그런 다음 다른 터미널을 열고 다음 명령어를 사용하세요:
-
-```bash
-python -m tools.e2e_webui
-```
-
-이렇게 하면 기기에 Gradio WebUI가 생성됩니다.
-
-모델을 처음 사용할 때는 (`--compile`이 True인 경우) 잠시 컴파일이 진행되므로 기다려 주세요.
-
-## Gradio Webui
-<p align="center">
-   <img src="../../assets/figs/agent_gradio.png" width="75%">
-</p>
-
-즐거운 시간 되세요!
-
-## 성능
-
-테스트 결과, 4060 노트북은 겨우 실행되며 매우 부하가 큰 상태로, 초당 약 8토큰 정도만 처리합니다. 4090은 컴파일 상태에서 초당 약 95토큰을 처리하며, 이것이 저희가 권장하는 사양입니다.
-
-# 에이전트 소개
-
-이 데모는 초기 알파 테스트 버전으로, 추론 속도 최적화가 필요하며 수정해야 할 버그가 많이 있습니다. 버그를 발견하거나 수정하고 싶으시다면 이슈나 풀 리퀘스트를 보내주시면 매우 감사하겠습니다.

docs/pt/finetune.md

@@ -1,128 +0,0 @@
-# Ajuste Fino
-
-É óbvio que ao abrir esta página, você não deve estar muito satisfeito com o desempenho do modelo pré-treinado com poucos exemplos. Você pode querer ajustar o modelo para melhorar seu desempenho em seu conjunto de dados.
-
-Na atual versão, a única coisa que você precisa ajustar é a parte do 'LLAMA'.
-
-## Ajuste Fino do LLAMA
-### 1. Preparando o conjunto de dados
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 30.1-32.71.lab
-│   └── 30.1-32.71.mp3
-└── SPK2
-    ├── 38.79-40.85.lab
-    └── 38.79-40.85.mp3
-```
-
-Você precisa converter seu conjunto de dados para o formato acima e colocá-lo em `data`. O arquivo de áudio pode ter as extensões `.mp3`, `.wav` ou `.flac`, e o arquivo de anotação deve ter a extensão `.lab`.
-
-!!! info
-    O arquivo de anotação `.lab` deve conter apenas a transcrição do áudio, sem a necessidade de formatação especial. Por exemplo, se o arquivo `hi.mp3` disser "Olá, tchau", o arquivo `hi.lab` conterá uma única linha de texto: "Olá, tchau".
-
-!!! warning
-    É recomendado aplicar normalização de volume ao conjunto de dados. Você pode usar o [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) para fazer isso.
-
-    ```bash
-    fap loudness-norm data-raw data --clean
-    ```
-
-
-### 2. Extração em lote de tokens semânticos
-
-Certifique-se de ter baixado os pesos do VQGAN. Se não, execute o seguinte comando:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-Em seguida, você pode executar o seguinte comando para extrair os tokens semânticos:
-
-```bash
-python tools/vqgan/extract_vq.py data \
-    --num-workers 1 --batch-size 16 \
-    --config-name "firefly_gan_vq" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-!!! note
-    Você pode ajustar `--num-workers` e `--batch-size` para aumentar a velocidade de extração, mas certifique-se de não exceder o limite de memória da sua GPU.  
-    Para o formato VITS, você pode especificar uma lista de arquivos usando `--filelist xxx.list`.
-
-Este comando criará arquivos `.npy` no diretório `data`, como mostrado abaixo:
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 21.15-26.44.npy
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 27.51-29.98.npy
-│   ├── 30.1-32.71.lab
-│   ├── 30.1-32.71.mp3
-│   └── 30.1-32.71.npy
-└── SPK2
-    ├── 38.79-40.85.lab
-    ├── 38.79-40.85.mp3
-    └── 38.79-40.85.npy
-```
-
-### 3. Empacotar o conjunto de dados em protobuf
-
-```bash
-python tools/llama/build_dataset.py \
-    --input "data" \
-    --output "data/protos" \
-    --text-extension .lab \
-    --num-workers 16
-```
-
-Após executar o comando, você deverá ver o arquivo `quantized-dataset-ft.protos` no diretório `data`.
-
-### 4. E finalmente, chegamos ao ajuste fino com LoRA
-
-Da mesma forma, certifique-se de ter baixado os pesos do `LLAMA`. Se não, execute o seguinte comando:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-E então, execute o seguinte comando para iniciar o ajuste fino:
-
-```bash
-python fish_speech/train.py --config-name text2semantic_finetune \
-    project=$project \
-    +lora@model.model.lora_config=r_8_alpha_16
-```
-
-!!! note
-    Se quiser, você pode modificar os parâmetros de treinamento, como `batch_size`, `gradient_accumulation_steps`, etc., para se ajustar à memória da sua GPU, modificando `fish_speech/configs/text2semantic_finetune.yaml`.
-
-!!! note
-    Para usuários do Windows, é recomendado usar `trainer.strategy.process_group_backend=gloo` para evitar problemas com `nccl`.
-
-Após concluir o treinamento, consulte a seção [inferência](inference.md).
-
-!!! info
-    Por padrão, o modelo aprenderá apenas os padrões de fala do orador e não o timbre. Ainda pode ser preciso usar prompts para garantir a estabilidade do timbre.
-    Se quiser que ele aprenda o timbre, aumente o número de etapas de treinamento, mas isso pode levar ao overfitting (sobreajuste).
-
-Após o treinamento, é preciso converter os pesos do LoRA em pesos regulares antes de realizar a inferência.
-
-```bash
-python tools/llama/merge_lora.py \
-    --lora-config r_8_alpha_16 \
-    --base-weight checkpoints/fish-speech-1.5 \
-    --lora-weight results/$project/checkpoints/step_000000010.ckpt \
-    --output checkpoints/fish-speech-1.5-yth-lora/
-```
-!!! note
-    É possível também tentar outros checkpoints. Sugerimos usar o checkpoint que melhor atenda aos seus requisitos, pois eles geralmente têm um desempenho melhor em dados fora da distribuição (OOD).

docs/pt/index.md

@@ -1,171 +0,0 @@
-# Introdução
-
-<div>
-<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
-<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
-<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
-<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
-</a>
-</div>
-
-!!! warning
-    Não nos responsabilizamos por qualquer uso ilegal do código-fonte. Consulte as leis locais sobre DMCA (Digital Millennium Copyright Act) e outras leis relevantes em sua região. <br/>
-    Este repositório de código e os modelos são distribuídos sob a licença CC-BY-NC-SA-4.0.
-
-<p align="center">
-   <img src="../assets/figs/diagram.png" width="75%">
-</p>
-
-## Requisitos
-
-- Memória da GPU: 4GB (para inferência), 8GB (para ajuste fino)
-- Sistema: Linux, Windows
-
-## Configuração do Windows
-
-!!! info "Aviso"
-    Recomendamos fortemente que usuários que não sejam especialistas em Windows usem a GUI para executar o projeto. [A GUI está aqui](https://github.com/AnyaCoder/fish-speech-gui).
-    
-Usuários profissionais do Windows podem considerar o uso do WSL2 ou Docker para executar a base de código.
-
-```bash
-# Crie um ambiente virtual Python 3.10, também é possível usar o virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# Instale o pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
-
-# Instale o fish-speech
-pip3 install -e .
-
-# (Ativar aceleração) Instalar triton-windows
-pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
-```
-
-## Configuração para Linux
-
-Para mais detalhes, consulte [pyproject.toml](../../pyproject.toml).
-```bash
-# Crie um ambiente virtual python 3.10, você também pode usar virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# Instale o pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-
-# Para os Usuário do Ubuntu / Debian: Instale o sox + ffmpeg
-apt install libsox-dev ffmpeg
-
-# Para os Usuário do Ubuntu / Debian: Instale o pyaudio
-apt install build-essential \
-    cmake \
-    libasound-dev \
-    portaudio19-dev \
-    libportaudio2 \
-    libportaudiocpp0
-    
-# Instale o fish-speech
-pip3 install -e .[stable]
-```
-
-## Configuração para macos
-
-Se você quiser realizar inferências no MPS, adicione a flag `--device mps`.
-Para uma comparação das velocidades de inferência, consulte [este PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772).
-
-!!! aviso
-    A opção `compile` não é oficialmente suportada em dispositivos Apple Silicon, então não há garantia de que a velocidade de inferência irá melhorar.
-
-```bash
-# create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-# install pytorch
-pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-# install fish-speech
-pip install -e .[stable]
-```
-
-## Configuração do Docker
-
-1. Instale o NVIDIA Container Toolkit:
-
-    Para usar a GPU com Docker para treinamento e inferência de modelos, você precisa instalar o NVIDIA Container Toolkit:
-
-    Para usuários Ubuntu:
-
-    ```bash
-    # Adicione o repositório remoto
-    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
-        && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
-            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
-            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-    # Instale o nvidia-container-toolkit
-    sudo apt-get update
-    sudo apt-get install -y nvidia-container-toolkit
-    # Reinicie o serviço Docker
-    sudo systemctl restart docker
-    ```
-
-    Para usuários de outras distribuições Linux, consulte o guia de instalação: [NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
-
-2. Baixe e execute a imagem fish-speech
-
-    ```shell
-    # Baixe a imagem
-    docker pull fishaudio/fish-speech:latest-dev
-    # Execute a imagem
-    docker run -it \
-        --name fish-speech \
-        --gpus all \
-        -p 7860:7860 \
-        fishaudio/fish-speech:latest-dev \
-        zsh
-    # Se precisar usar outra porta, modifique o parâmetro -p para YourPort:7860
-    ```
-
-3. Baixe as dependências do modelo
-
-    Certifique-se de estar no terminal do contêiner Docker e, em seguida, baixe os modelos necessários `vqgan` e `llama` do nosso repositório HuggingFace.
-
-    ```bash
-    huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-4. Configure as variáveis de ambiente e acesse a WebUI
-
-    No terminal do contêiner Docker, digite `export GRADIO_SERVER_NAME="0.0.0.0"` para permitir o acesso externo ao serviço gradio dentro do Docker.
-    Em seguida, no terminal do contêiner Docker, digite `python tools/run_webui.py` para iniciar o serviço WebUI.
-
-    Se estiver usando WSL ou MacOS, acesse [http://localhost:7860](http://localhost:7860) para abrir a interface WebUI.
-
-    Se estiver implantando em um servidor, substitua localhost pelo IP do seu servidor.
-
-## Histórico de Alterações
-
-- 12/03/2024: Atualização do Fish-Speech para 1.5, com suporte para mais idiomas, sendo considerado SOTA (estado da arte) no campo de código aberto.
-- 10/09/2024: Fish-Speech atualizado para a versão 1.4, aumentado o tamanho do conjunto de dados, quantizer n_groups 4 -> 8.
-- 02/07/2024: Fish-Speech atualizado para a versão 1.2, removido o Decodificador VITS e aprimorado consideravelmente a capacidade de zero-shot.
-- 10/05/2024: Fish-Speech atualizado para a versão 1.1, implementado o decodificador VITS para reduzir a WER e melhorar a similaridade de timbre.
-- 22/04/2024: Finalizada a versão 1.0 do Fish-Speech, modificados significativamente os modelos VQGAN e LLAMA.
-- 28/12/2023: Adicionado suporte para ajuste fino `lora`.
-- 27/12/2023: Adicionado suporte para `gradient checkpointing`, `causual sampling` e `flash-attn`.
-- 19/12/2023: Atualizada a interface web e a API HTTP.
-- 18/12/2023: Atualizada a documentação de ajuste fino e exemplos relacionados.
-- 17/12/2023: Atualizado o modelo `text2semantic`, suportando o modo sem fonemas.
-- 13/12/2023: Versão beta lançada, incluindo o modelo VQGAN e um modelo de linguagem baseado em LLAMA (suporte apenas a fonemas).
-
-## Agradecimentos
-
-- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
-- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
-- [GPT VITS](https://github.com/innnky/gpt-vits)
-- [MQTTS](https://github.com/b04901014/MQTTS)
-- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
-- [Transformers](https://github.com/huggingface/transformers)
-- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/pt/inference.md

@@ -1,123 +0,0 @@
-# Inferência
-
-Suporte para inferência por linha de comando, API HTTP e interface web (WebUI).
-
-!!! note
-    O processo de raciocínio, em geral, consiste em várias partes:
-
-    1. Codificar cerca de 10 segundos de voz usando VQGAN.
-    2. Inserir os tokens semânticos codificados e o texto correspondente no modelo de linguagem como um exemplo.
-    3. Dado um novo trecho de texto, fazer com que o modelo gere os tokens semânticos correspondentes.
-    4. Inserir os tokens semânticos gerados no VITS / VQGAN para decodificar e gerar a voz correspondente.
-
-## Baixar modelos
-Baixe os modelos `vqgan` e `llama` necessários do nosso repositório Hugging Face.
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-## Inferência por Linha de Comando
-### 1. Gerar prompt a partir da voz:
-
-!!! note
-    Se quiser permitir que o modelo escolha aleatoriamente um timbre de voz, pule esta etapa.
-
-!!! warning "Aviso de Versão Futura"
-    Mantivemos a interface acessível a partir do caminho original (tools/vqgan/infernce.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "paimon.wav" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-Você deverá obter um arquivo `fake.npy`.
-
-### 2. Gerar tokens semânticos a partir do texto:
-
-!!! warning "Aviso de Versão Futura"
-    Mantivemos a interface acessível a partir do caminho original (tools/llama/generate.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
-
-```bash
-python fish_speech/models/text2semantic/inference.py \
-    --text "O texto que você deseja converter" \
-    --prompt-text "Seu texto de referência" \
-    --prompt-tokens "fake.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5" \
-    --num-samples 2 \
-    --compile
-```
-
-Este comando criará um arquivo `codes_N` no diretório de trabalho, onde N é um número inteiro começando de 0.
-
-!!! note
-    Use `--compile` para fundir kernels CUDA para ter uma inferência mais rápida (~30 tokens/segundo -> ~500 tokens/segundo).
-    Mas, se não planeja usar a aceleração CUDA, comente o parâmetro `--compile`.
-
-!!! info
-    Para GPUs que não suportam bf16, pode ser necessário usar o parâmetro `--half`.
-
-### 3. Gerar vocais a partir de tokens semânticos:
-
-#### Decodificador VQGAN
-
-!!! warning "Aviso de Versão Futura"
-    Mantivemos a interface acessível a partir do caminho original (tools/vqgan/infernce.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "codes_0.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-## Inferência por API HTTP
-
-Fornecemos uma API HTTP para inferência. O seguinte comando pode ser usado para iniciar o servidor:
-
-```bash
-python -m tools.api_server \
-    --listen 0.0.0.0:8080 \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-
-> Para acelerar a inferência, adicione o parâmetro `--compile`.
-
-Depois disso, é possível visualizar e testar a API em http://127.0.0.1:8080/.
-
-Abaixo está um exemplo de envio de uma solicitação usando `tools/api_client.py`.
-
-```bash
-python -m tools.api_client \
-    --text "Texto a ser inserido" \
-    --reference_audio "Caminho para o áudio de referência" \
-    --reference_text "Conteúdo de texto do áudio de referência" \
-    --streaming True
-```
-
-O comando acima indica a síntese do áudio desejada de acordo com as informações do áudio de referência e a retorna em modo de streaming.
-
-!!! info
-    Para aprender mais sobre parâmetros disponíveis, você pode usar o comando `python -m tools.api_client -h`
-
-## Inferência por WebUI
-
-Para iniciar a WebUI de Inferência execute o seguinte comando:
-
-```bash
-python -m tools.run_webui \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-> Para acelerar a inferência, adicione o parâmetro `--compile`.
-
-!!! note
-    Você pode salvar antecipadamente o arquivo de rótulos e o arquivo de áudio de referência na pasta `references` do diretório principal (que você precisa criar), para que possa chamá-los diretamente na WebUI.
-    
-!!! note
-    É possível usar variáveis de ambiente do Gradio, como `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME`, para configurar a WebUI.
-
-Divirta-se!

docs/pt/install.md

@@ -0,0 +1,50 @@
+# Introdução
+
+<div>
+<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
+<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
+<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
+<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
+</a>
+</div>
+
+!!! warning
+    Não assumimos nenhuma responsabilidade pelo uso ilegal da base de código. Consulte as leis locais sobre DMCA (Digital Millennium Copyright Act) e outras leis relevantes em sua área. <br/>
+    Esta base de código é lançada sob a licença Apache 2.0 e todos os modelos são lançados sob a licença CC-BY-NC-SA-4.0.
+
+## Requisitos
+
+- Memória GPU: 12GB (Inferência)
+- Sistema: Linux, Windows
+
+## Configuração
+
+Primeiro, precisamos criar um ambiente conda para instalar os pacotes.
+
+```bash
+
+conda create -n fish-speech python=3.12
+conda activate fish-speech
+
+pip install sudo apt-get install portaudio19-dev # Para pyaudio
+pip install -e . # Isso baixará todos os pacotes restantes.
+
+apt install libsox-dev ffmpeg # Se necessário.
+```
+
+!!! warning
+    A opção `compile` não é suportada no Windows e macOS, se você quiser executar com compile, precisa instalar o trition por conta própria.
+
+## Agradecimentos
+
+- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
+- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
+- [GPT VITS](https://github.com/innnky/gpt-vits)
+- [MQTTS](https://github.com/b04901014/MQTTS)
+- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
+- [Transformers](https://github.com/huggingface/transformers)
+- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/pt/samples.md

@@ -1,225 +0,0 @@
-# Amostras
-
-A demonstração da versão 1.4 foi atualizada [aqui](https://speech.fish.audio/samples/)
-
-As amostras da v1.2 estão disponíveis em [Bilibili](https://www.bilibili.com/video/BV1wz421B71D/).
-
-As seguintes amostras são do modelo v1.1.
-
-## Frase em Chinês 1
-```
-人间灯火倒映湖中，她的渴望让静水泛起涟漪。若代价只是孤独，那就让这份愿望肆意流淌。
-流入她所注视的世间，也流入她如湖水般澄澈的目光。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Nahida (Genshin Impact)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Zhongli (Genshin Impact)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Furina (Genshin Impact)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Orador Aleatório 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/4_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Orador Aleatório 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/5_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## Frase em Chinês 2
-```
-你们这个是什么群啊，你们这是害人不浅啊你们这个群！谁是群主，出来！真的太过分了。你们搞这个群干什么？
-我儿子每一科的成绩都不过那个平均分呐，他现在初二，你叫我儿子怎么办啊？他现在还不到高中啊？
-你们害死我儿子了！快点出来你这个群主！再这样我去报警了啊！我跟你们说你们这一帮人啊，一天到晚啊，
-搞这些什么游戏啊，动漫啊，会害死你们的，你们没有前途我跟你说。你们这九百多个人，好好学习不好吗？
-一天到晚在上网。有什么意思啊？麻烦你重视一下你们的生活的目标啊？有一点学习目标行不行？一天到晚上网是不是人啊？
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Nahida (Genshin Impact)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/6_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Orador Aleatório</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/7_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## Frase em Chinês 3
-```
-大家好，我是 Fish Audio 开发的开源文本转语音模型。经过十五万小时的数据训练，
-我已经能够熟练掌握中文、日语和英语，我的语言处理能力接近人类水平，声音表现形式丰富多变。
-作为一个仅有亿级参数的模型，我相信社区成员能够在个人设备上轻松运行和微调，让我成为您的私人语音助手。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Orador Aleatório</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/8_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## Frase em Inglês 1
-
-```
-In the realm of advanced technology, the evolution of artificial intelligence stands as a 
-monumental achievement. This dynamic field, constantly pushing the boundaries of what 
-machines can do, has seen rapid growth and innovation. From deciphering complex data 
-patterns to driving cars autonomously, AI's applications are vast and diverse.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Orador Aleatório 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Orador Aleatório 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## Frase em Inglês 2
-```
-Hello everyone, I am an open-source text-to-speech model developed by 
-Fish Audio. After training with 150,000 hours of data, I have become proficient 
-in Chinese, Japanese, and English, and my language processing abilities 
-are close to human level. My voice is capable of a wide range of expressions. 
-As a model with only hundreds of millions of parameters, I believe community 
-members can easily run and fine-tune me on their personal devices, allowing 
-me to serve as your personal voice assistant.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Orador Aleatório</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## Frase em Japonês 1
-
-```
-先進技術の領域において、人工知能の進化は画期的な成果として立っています。常に機械ができることの限界を
-押し広げているこのダイナミックな分野は、急速な成長と革新を見せています。複雑なデータパターンの解読か
-ら自動運転車の操縦まで、AIの応用は広範囲に及びます。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Orador Aleatório 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>Orador Aleatório 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## Frase em Japonês 2
-```
-皆さん、こんにちは。私はフィッシュオーディオによって開発されたオープンソースのテ
-キストから音声への変換モデルです。15万時間のデータトレーニングを経て、
-中国語、日本語、英語を熟知しており、言語処理能力は人間に近いレベルです。
-声の表現も多彩で豊かです。数億のパラメータを持つこのモデルは、コミュニティ
-のメンバーが個人のデバイスで簡単に実行し、微調整することができると
-信じています。これにより、私を個人の音声アシスタントとして活用できます。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>Orador</th>
-        <th>Áudio de Entrada</th>
-        <th>Áudio Sintetizado</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>Orador Aleatório</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>

docs/pt/start_agent.md

@@ -1,80 +0,0 @@
-# Iniciar Agente
-
-!!! note
-    Todo o documento foi traduzido por claude3.5 Sonnet, se você for um falante nativo e achar a tradução problemática, muito obrigado por nos enviar um problema ou uma solicitação pull!
-
-## Requisitos
-
-- Memória GPU: No mínimo 8GB (com quantização), 16GB ou mais é recomendado.
-- Uso de disco: 10GB
-
-## Download do Modelo
-
-Você pode obter o modelo através de:
-
-```bash
-huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
-```
-
-Coloque-os na pasta 'checkpoints'.
-
-Você também precisará do modelo fish-speech que pode ser baixado seguindo as instruções em [inference](inference.md).
-
-Então haverá 2 pastas em checkpoints.
-
-O `checkpoints/fish-speech-1.4` e `checkpoints/fish-agent-v0.1-3b`
-
-## Preparação do Ambiente
-
-Se você já tem o Fish-speech, pode usar diretamente adicionando a seguinte instrução:
-```bash
-pip install cachetools
-```
-
-!!! nota
-    Por favor, use a versão Python abaixo de 3.12 para compilação.
-
-Se você não tem, use os comandos abaixo para construir seu ambiente:
-
-```bash
-sudo apt-get install portaudio19-dev
-
-pip install -e .[stable]
-```
-
-## Iniciar a Demo do Agente
-
-Para construir o fish-agent, use o comando abaixo na pasta principal:
-
-```bash
-python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
-```
-
-O argumento `--compile` só suporta Python < 3.12, o que aumentará muito a velocidade de geração de tokens.
-
-Não será compilado de uma vez (lembre-se).
-
-Então abra outro terminal e use o comando:
-
-```bash
-python -m tools.e2e_webui
-```
-
-Isso criará uma WebUI Gradio no dispositivo.
-
-Quando você usar o modelo pela primeira vez, ele irá compilar (se `--compile` estiver True) por um curto período, então aguarde com paciência.
-
-## Gradio Webui
-<p align="center">
-   <img src="../../assets/figs/agent_gradio.png" width="75%">
-</p>
-
-Divirta-se!
-
-## Desempenho
-
-Em nossos testes, um laptop com 4060 mal consegue rodar, ficando muito sobrecarregado, gerando apenas cerca de 8 tokens/s. A 4090 gera cerca de 95 tokens/s com compilação, que é o que recomendamos.
-
-# Sobre o Agente
-
-A demo é uma versão alpha inicial de teste, a velocidade de inferência precisa ser otimizada, e há muitos bugs aguardando correção. Se você encontrou um bug ou quer corrigi-lo, ficaremos muito felizes em receber uma issue ou um pull request.

docs/zh/finetune.md

@@ -1,139 +0,0 @@
-# 微调
-
-显然, 当你打开这个页面的时候, 你已经对预训练模型 zero-shot 的效果不算满意. 你想要微调一个模型, 使得它在你的数据集上表现更好.  
-
-在目前版本，你只需要微调'LLAMA'部分即可.
-
-## LLAMA 微调
-### 1. 准备数据集
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 30.1-32.71.lab
-│   └── 30.1-32.71.mp3
-└── SPK2
-    ├── 38.79-40.85.lab
-    └── 38.79-40.85.mp3
-```
-
-你需要将数据集转为以上格式, 并放到 `data` 下, 音频后缀可以为 `.mp3`, `.wav` 或 `.flac`, 标注文件后缀建议为 `.lab`.
-
-!!! info
-    标注文件 `.lab` 仅需包含音频的转写文本，无需遵循特殊格式要求。例如，如果 `hi.mp3` 中的内容是“你好，再见。”，那么 `hi.lab` 文件中只需包含一行文本：“你好，再见”。    
-
-!!! warning
-    建议先对数据集进行响度匹配, 你可以使用 [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) 来完成这一步骤. 
-    ```bash
-    fap loudness-norm data-raw data --clean
-    ```
-
-### 2. 批量提取语义 token
-
-确保你已经下载了 vqgan 权重, 如果没有, 请运行以下命令:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-对于中国大陆用户, 可使用 mirror 下载.
-
-```bash
-HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-随后可运行以下命令来提取语义 token:
-
-```bash
-python tools/vqgan/extract_vq.py data \
-    --num-workers 1 --batch-size 16 \
-    --config-name "firefly_gan_vq" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-!!! note
-    你可以调整 `--num-workers` 和 `--batch-size` 来提高提取速度, 但是请注意不要超过你的显存限制.  
-
-该命令会在 `data` 目录下创建 `.npy` 文件, 如下所示:
-
-```
-.
-├── SPK1
-│   ├── 21.15-26.44.lab
-│   ├── 21.15-26.44.mp3
-│   ├── 21.15-26.44.npy
-│   ├── 27.51-29.98.lab
-│   ├── 27.51-29.98.mp3
-│   ├── 27.51-29.98.npy
-│   ├── 30.1-32.71.lab
-│   ├── 30.1-32.71.mp3
-│   └── 30.1-32.71.npy
-└── SPK2
-    ├── 38.79-40.85.lab
-    ├── 38.79-40.85.mp3
-    └── 38.79-40.85.npy
-```
-
-### 3. 打包数据集为 protobuf
-
-```bash
-python tools/llama/build_dataset.py \
-    --input "data" \
-    --output "data/protos" \
-    --text-extension .lab \
-    --num-workers 16
-```
-
-命令执行完毕后, 你应该能在 `data` 目录下看到 `protos` 文件.
-
-
-### 4. 最后, 使用 LoRA 进行微调
-
-同样的, 请确保你已经下载了 `LLAMA` 权重, 如果没有, 请运行以下命令:
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-对于中国大陆用户, 可使用 mirror 下载.
-
-```bash
-HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-最后, 你可以运行以下命令来启动微调:
-
-```bash
-python fish_speech/train.py --config-name text2semantic_finetune \
-    project=$project \
-    +lora@model.model.lora_config=r_8_alpha_16
-```
-
-!!! note
-    你可以通过修改 `fish_speech/configs/text2semantic_finetune.yaml` 来修改训练参数如 `batch_size`, `gradient_accumulation_steps` 等, 来适应你的显存.
-
-!!! note
-    对于 Windows 用户, 你可以使用 `trainer.strategy.process_group_backend=gloo` 来避免 `nccl` 的问题.
-
-训练结束后, 你可以参考 [推理](inference.md) 部分来测试你的模型.
-
-!!! info
-    默认配置下, 基本只会学到说话人的发音方式, 而不包含音色, 你依然需要使用 prompt 来保证音色的稳定性.  
-    如果你想要学到音色, 请将训练步数调大, 但这有可能会导致过拟合. 
-
-训练完成后, 你需要先将 loRA 的权重转为普通权重, 然后再进行推理.
-
-```bash
-python tools/llama/merge_lora.py \
-	--lora-config r_8_alpha_16 \
-	--base-weight checkpoints/fish-speech-1.5 \
-	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
-	--output checkpoints/fish-speech-1.5-yth-lora/
-```
-
-!!! note
-    你也可以尝试其他的 checkpoint, 我们建议你使用最早的满足你要求的 checkpoint, 他们通常在 OOD 上表现更好.

docs/zh/index.md

@@ -1,180 +0,0 @@
-# 介绍
-
-<div>
-<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
-<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
-<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
-</a>
-<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
-<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
-</a>
-</div>
-
-!!! warning "警告"
-    我们不对代码库的任何非法使用承担任何责任. 请参阅您当地关于 DMCA (数字千年法案) 和其他相关法律法规. <br/>
-    此代码库与所有模型根据 CC-BY-NC-SA-4.0 许可证发布.
-
-<p align="center">
-   <img src="../assets/figs/diagram.png" width="75%">
-</p>
-
-## 要求
-
-- GPU 内存: 4GB (用于推理), 8GB (用于微调)
-- 系统: Linux, Windows
-
-## Windows 配置
-
-!!! info "注意"
-    我们强烈建议非Windows专业用户使用GUI运行该项目。[GUI在这里](https://github.com/AnyaCoder/fish-speech-gui).
-
-
-Windows 专业用户可以考虑 WSL2 或 docker 来运行代码库。
-
-```bash
-# 创建一个 python 3.10 虚拟环境, 你也可以用 virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# 安装 pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
-
-# 安装 fish-speech
-pip3 install -e .
-
-# (开启编译加速) 安装 triton-windows
-pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
-```
-
-## Linux 配置
-
-有关详细信息，请参见 [pyproject.toml](../../pyproject.toml)。
-```bash
-# 创建一个 python 3.10 虚拟环境, 你也可以用 virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-
-# 安装 pytorch
-pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-
-# (Ubuntu / Debian 用户) 安装 sox + ffmpeg
-apt install libsox-dev ffmpeg
-
-# (Ubuntu / Debian 用户) 安装 pyaudio
-apt install build-essential \
-    cmake \
-    libasound-dev \
-    portaudio19-dev \
-    libportaudio2 \
-    libportaudiocpp0
-    
-# 安装 fish-speech
-pip3 install -e .[stable]
-```
-
-## macos 配置
-
-如果您想在 MPS 上进行推理，请添加 `--device mps` 标志。
-有关推理速度的比较，请参考 [此 PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772)。
-
-!!! 警告
-    `compile` 选项在 Apple Silicon 设备上尚未正式支持，因此推理速度没有提升的保证。
-
-```bash
-# create a python 3.10 virtual environment, you can also use virtualenv
-conda create -n fish-speech python=3.10
-conda activate fish-speech
-# install pytorch
-pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
-# install fish-speech
-pip install -e .[stable]
-```
-
-## Docker 配置
-
-1. 安装 NVIDIA Container Toolkit：
-
-    Docker 如果想使用 GPU 进行模型训练和推理，需要安装 NVIDIA Container Toolkit ：
-
-    对于 Ubuntu 用户：
-
-    ```bash
-    # 添加远程仓库
-    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
-        && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
-            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
-            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-    # 安装 nvidia-container-toolkit
-    sudo apt-get update
-    sudo apt-get install -y nvidia-container-toolkit
-    # 重启 Docker 服务
-    sudo systemctl restart docker
-    ```
-
-    对于使用其他 Linux 发行版的用户，安装指南请参考：[NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)。
-
-    注：对于中国大陆的用户，您可能需要使用代理来完成相关工具的安装。
-
-2. 拉取并运行 fish-speech 镜像
-
-    ```shell
-    # 拉取镜像
-    docker pull fishaudio/fish-speech:latest-dev
-    # 运行镜像
-    docker run -it \
-        --name fish-speech \
-        --gpus all \
-        -p 7860:7860 \
-        fishaudio/fish-speech:latest-dev \
-        zsh
-    # 如果需要使用其他端口，请修改 -p 参数为 YourPort:7860
-    ```
-
-3. 下载模型依赖
-
-    确保您在 docker 容器内的终端，然后再从我们的 huggingface 仓库下载所需的 `vqgan` 和 `llama` 模型。
-
-    ```bash
-    huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-    对于中国大陆用户，可以通过镜像站下载。
-
-    ```bash
-    HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-    ```
-
-4. 配置环境变量，访问 WebUI
-
-    在 docker 容器内的终端，输入 `export GRADIO_SERVER_NAME="0.0.0.0"` ，从而让外部可以访问 docker 内的 gradio 服务。
-    接着在 docker 容器内的终端，输入 `python tools/run_webui.py` 即可开启 WebUI 服务。
-
-    如果是 WSL 或者是 MacOS ，访问 [http://localhost:7860](http://localhost:7860) 即可打开 WebUI 界面。
-
-    如果是部署在服务器上，更换 localhost 为您的服务器 ip 即可。
-
-## 更新日志
-
-- 2024/12/03: 更新了 Fish-Speech 到 1.5，增加更多支持语言，在开源领域属于SOTA.
-- 2024/09/10: 更新了 Fish-Speech 到 1.4, 增加了数据集大小， quantizer n_groups 4 -> 8.
-- 2024/07/02: 更新了 Fish-Speech 到 1.2 版本，移除 VITS Decoder，同时极大幅度提升 zero-shot 能力.
-- 2024/05/10: 更新了 Fish-Speech 到 1.1 版本，引入了 VITS Decoder 来降低口胡和提高音色相似度.
-- 2024/04/22: 完成了 Fish-Speech 1.0 版本, 大幅修改了 VQGAN 和 LLAMA 模型.
-- 2023/12/28: 添加了 `lora` 微调支持.
-- 2023/12/27: 添加了 `gradient checkpointing`, `causual sampling` 和 `flash-attn` 支持.
-- 2023/12/19: 更新了 Webui 和 HTTP API.
-- 2023/12/18: 更新了微调文档和相关例子.
-- 2023/12/17: 更新了 `text2semantic` 模型, 支持无音素模式.
-- 2023/12/13: 测试版发布, 包含 VQGAN 模型和一个基于 LLAMA 的语言模型 (只支持音素).
-
-## 致谢
-
-- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
-- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
-- [GPT VITS](https://github.com/innnky/gpt-vits)
-- [MQTTS](https://github.com/b04901014/MQTTS)
-- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
-- [Transformers](https://github.com/huggingface/transformers)
-- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/zh/inference.md

@@ -1,143 +0,0 @@
-# 推理
-
-推理支持命令行, http api, 以及 webui 三种方式.
-
-!!! note
-    总的来说, 推理分为几个部分:
-
-    1. 给定一段 ~10 秒的语音, 将它用 VQGAN 编码.
-    2. 将编码后的语义 token 和对应文本输入语言模型作为例子.
-    3. 给定一段新文本, 让模型生成对应的语义 token.
-    4. 将生成的语义 token 输入 VQGAN 解码, 生成对应的语音.
-
-## 下载模型
-从我们的 huggingface 仓库下载所需的 `vqgan` 和 `llama` 模型。
-
-```bash
-huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-对于中国大陆用户，可使用 mirror 下载。
-
-```bash
-HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
-```
-
-## 命令行推理
-### 1. 从语音生成 prompt:
-
-!!! note
-    如果你打算让模型随机选择音色, 你可以跳过这一步.
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "paimon.wav" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-你应该能得到一个 `fake.npy` 文件.
-
-### 2. 从文本生成语义 token:
-
-```bash
-python fish_speech/models/text2semantic/inference.py \
-    --text "要转换的文本" \
-    --prompt-text "你的参考文本" \
-    --prompt-tokens "fake.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5" \
-    --num-samples 2 \
-    --compile
-```
-
-该命令会在工作目录下创建 `codes_N` 文件, 其中 N 是从 0 开始的整数.
-
-!!! note
-    您可能希望使用 `--compile` 来融合 cuda 内核以实现更快的推理 (~30 个 token/秒 -> ~500 个 token/秒).  
-    对应的, 如果你不打算使用加速, 你可以注释掉 `--compile` 参数.
-
-!!! info
-    对于不支持 bf16 的 GPU, 你可能需要使用 `--half` 参数.
-
-### 3. 从语义 token 生成人声:
-
-#### VQGAN 解码
-
-```bash
-python fish_speech/models/vqgan/inference.py \
-    -i "codes_0.npy" \
-    --checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
-```
-
-## HTTP API 推理
-
-运行以下命令来启动 HTTP 服务:
-
-```bash
-python -m tools.api_server \
-    --listen 0.0.0.0:8080 \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-> 如果你想要加速推理，可以加上`--compile`参数。
-
-推荐中国大陆用户运行以下命令来启动 HTTP 服务:
-```bash
-HF_ENDPOINT=https://hf-mirror.com python -m ...(同上)
-```
-
-随后, 你可以在 `http://127.0.0.1:8080/` 中查看并测试 API.
-
-下面是使用`tools/api_client.py`发送请求的示例。
-
-```bash
-python -m tools.api_client \
-    --text "要输入的文本" \
-    --reference_audio "参考音频路径" \
-    --reference_text "参考音频的文本内容" \
-    --streaming True
-```
-
-上面的命令表示按照参考音频的信息，合成所需的音频并流式返回.
-
-下面的示例展示了， 可以一次使用**多个** `参考音频路径` 和 `参考音频的文本内容`。在命令里用空格隔开即可。
-```bash
-python -m tools.api_client \
-    --text "要输入的文本" \
-    --reference_audio "参考音频路径1" "参考音频路径2" \
-    --reference_text "参考音频的文本内容1" "参考音频的文本内容2"\
-    --streaming False \
-    --output "generated" \
-    --format "mp3"
-```
-
-上面的命令表示按照多个参考音频的信息，合成所需的`MP3`格式音频，并保存为当前目录的`generated.mp3`文件。
-
-还可以用`--reference_id`(仅能用一个)来代替`--reference_audio`和`--reference_text`, 前提是在项目根目录下创建`references/<your reference_id>`文件夹，
-里面放上任意对音频与标注文本。 目前支持的参考音频最多加起来总时长90s。
-
-!!! info
-    要了解有关可用参数的更多信息，可以使用命令`python -m tools.api_client -h`
-
-## GUI 推理 
-[下载客户端](https://github.com/AnyaCoder/fish-speech-gui/releases)
-
-## WebUI 推理
-
-你可以使用以下命令来启动 WebUI:
-
-```bash
-python -m tools.run_webui \
-    --llama-checkpoint-path "checkpoints/fish-speech-1.5" \
-    --decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
-    --decoder-config-name firefly_gan_vq
-```
-> 如果你想要加速推理，可以加上`--compile`参数。
-
-!!! note
-    你可以提前将label文件和参考音频文件保存到主目录下的 `references` 文件夹（需要自行创建），这样你可以直接在WebUI中调用它们。
-
-!!! note
-    你可以使用 Gradio 环境变量, 如 `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME` 来配置 WebUI.
-
-祝大家玩得开心!

docs/zh/install.md

@@ -0,0 +1,50 @@
+# 简介
+
+<div>
+<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
+<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
+<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
+</a>
+<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
+<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
+</a>
+</div>
+
+!!! warning
+    我们不对代码库的任何非法使用承担责任。请参考您所在地区有关 DMCA（数字千年版权法）和其他相关法律的规定。<br/>
+    此代码库在 Apache 2.0 许可证下发布，所有模型在 CC-BY-NC-SA-4.0 许可证下发布。
+
+## 系统要求
+
+- GPU 内存：12GB（推理）
+- 系统：Linux、Windows
+
+## 安装
+
+首先，我们需要创建一个 conda 环境来安装包。
+
+```bash
+
+conda create -n fish-speech python=3.12
+conda activate fish-speech
+
+pip install sudo apt-get install portaudio19-dev # 用于 pyaudio
+pip install -e . # 这将下载所有其余的包。
+
+apt install libsox-dev ffmpeg # 如果需要的话。
+```
+
+!!! warning
+    `compile` 选项在 Windows 和 macOS 上不受支持，如果您想使用 compile 运行，需要自己安装 trition。
+
+## 致谢
+
+- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
+- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
+- [GPT VITS](https://github.com/innnky/gpt-vits)
+- [MQTTS](https://github.com/b04901014/MQTTS)
+- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
+- [Transformers](https://github.com/huggingface/transformers)
+- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)

docs/zh/samples.md

@@ -1,225 +0,0 @@
-# 例子
-
-v1.4 演示已更新至[此处](https://speech.fish.audio/samples/)。
-
-v1.2 的样本可以在 [Bilibili](https://www.bilibili.com/video/BV1wz421B71D/) 观看。
-
-以下样本来自 v1.1 版本的模型。
-
-## 中文句子 1
-```
-人间灯火倒映湖中，她的渴望让静水泛起涟漪。若代价只是孤独，那就让这份愿望肆意流淌。
-流入她所注视的世间，也流入她如湖水般澄澈的目光。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>纳西妲 (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>钟离 (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>芙宁娜 (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>随机说话人 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/4_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>随机说话人 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/5_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## 中文句子 2
-```
-你们这个是什么群啊，你们这是害人不浅啊你们这个群！谁是群主，出来！真的太过分了。你们搞这个群干什么？
-我儿子每一科的成绩都不过那个平均分呐，他现在初二，你叫我儿子怎么办啊？他现在还不到高中啊？
-你们害死我儿子了！快点出来你这个群主！再这样我去报警了啊！我跟你们说你们这一帮人啊，一天到晚啊，
-搞这些什么游戏啊，动漫啊，会害死你们的，你们没有前途我跟你说。你们这九百多个人，好好学习不好吗？
-一天到晚在上网。有什么意思啊？麻烦你重视一下你们的生活的目标啊？有一点学习目标行不行？一天到晚上网是不是人啊？
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>纳西妲 (原神)</td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/6_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>随机说话人</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/7_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-
-## 中文句子 3
-```
-大家好，我是 Fish Audio 开发的开源文本转语音模型。经过十五万小时的数据训练，
-我已经能够熟练掌握中文、日语和英语，我的语言处理能力接近人类水平，声音表现形式丰富多变。
-作为一个仅有亿级参数的模型，我相信社区成员能够在个人设备上轻松运行和微调，让我成为您的私人语音助手。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>随机说话人</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/8_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 英文句子 1
-
-```
-In the realm of advanced technology, the evolution of artificial intelligence stands as a 
-monumental achievement. This dynamic field, constantly pushing the boundaries of what 
-machines can do, has seen rapid growth and innovation. From deciphering complex data 
-patterns to driving cars autonomously, AI's applications are vast and diverse.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>随机说话人 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>随机说话人 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 英文句子 2
-```
-Hello everyone, I am an open-source text-to-speech model developed by 
-Fish Audio. After training with 150,000 hours of data, I have become proficient 
-in Chinese, Japanese, and English, and my language processing abilities 
-are close to human level. My voice is capable of a wide range of expressions. 
-As a model with only hundreds of millions of parameters, I believe community 
-members can easily run and fine-tune me on their personal devices, allowing 
-me to serve as your personal voice assistant.
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>随机说话人</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 日文句子 1
-
-```
-先進技術の領域において、人工知能の進化は画期的な成果として立っています。常に機械ができることの限界を
-押し広げているこのダイナミックな分野は、急速な成長と革新を見せています。複雑なデータパターンの解読か
-ら自動運転車の操縦まで、AIの応用は広範囲に及びます。
-```
-
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>随机说话人 1</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/0_output.wav" /></td>
-    </tr>
-    <tr>
-        <td>随机说话人 2</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/1_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>
-
-## 日文句子 2
-```
-皆さん、こんにちは。私はフィッシュオーディオによって開発されたオープンソースのテ
-キストから音声への変換モデルです。15万時間のデータトレーニングを経て、
-中国語、日本語、英語を熟知しており、言語処理能力は人間に近いレベルです。
-声の表現も多彩で豊かです。数億のパラメータを持つこのモデルは、コミュニティ
-のメンバーが個人のデバイスで簡単に実行し、微調整することができると
-信じています。これにより、私を個人の音声アシスタントとして活用できます。
-```
-
-<table>
-    <thead>
-    <tr>
-        <th>说话人</th>
-        <th>输入音频</th>
-        <th>合成音频</th>
-    </tr>
-    </thead>
-    <tbody>
-    <tr>
-        <td>随机说话人</td>
-        <td> - </td>
-        <td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/2_output.wav" /></td>
-    </tr>
-    </tbody>
-</table>

docs/zh/start_agent.md

@@ -1,83 +0,0 @@
-# 启动 Agent
-
-## 要求
-
-- GPU 显存: 至少 8GB（在量化的条件下），推荐 16GB 及以上
-- 硬盘使用量: 10GB
-
-## 下载模型
-
-你可以执行下面的语句来获取模型:
-
-```bash
-huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
-```
-
-如果你处于国内网络，首先执行:
-
-```bash
-export HF_ENDPOINT=https://hf-mirror.com
-```
-
-把他们放进名为 'checkpoints' 的文件夹内。
-
-你同样需要 fish-speech 的模型，关于如何获取 fish-speech 模型请查看[inference](inference.md)。
-
-完成后你的 checkpoints 文件夹中会有两个子文件夹：`checkpoints/fish-speech-1.4` 和 `checkpoints/fish-agent-v0.1-3b`。
-
-## Environment Prepare
-
-如果你已经有了 Fish-Speech 环境，你可以在安装下面的包的前提下直接使用：
-
-```bash
-pip install cachetools
-```
-
-!!! note
-请使用小于 3.12 的 python 版本使 compile 可用
-
-如果你没有 Fish-Speech 环境，请执行下面的语句来构造你的环境：
-
-```bash
-sudo apt-get install portaudio19-dev
-
-pip install -e .[stable]
-```
-
-## 链接 Agent.
-
-你需要使用以下指令来构建 fish-agent
-
-```bash
-python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
-```
-
-`--compile`只能在小于 3.12 版本的 Python 使用，这个功能可以极大程度上提高生成速度。
-
-你需要哦注意 compile 需要进行一段时间.
-
-然后启动另一个终端并执行:
-
-```bash
-python -m tools.e2e_webui
-```
-
-这会在设备上创建一个 Gradio WebUI。
-
-每当进行第一轮对话的时候，模型需要 compile 一段时间，请耐心等待
-
-## Gradio Webui
-
-<p align="center">
-   <img src="../../assets/figs/agent_gradio.png" width="75%">
-</p>
-
-玩得开心！
-
-## Performance
-
-在我们的测试环境下， 4060 laptop GPU 只能刚刚运行该模型，只有大概 8 tokens/s。 4090 GPU 可以在编译后达到 95 tokens/s，我们推荐使用至少 4080 以上级别的 GPU 来达到较好体验。
-
-# About Agent
-
-该模型仍处于测试阶段。如果你发现了问题，请给我们提 issue 或者 pull request，我们非常感谢。

fish_speech/callbacks/__init__.py

@@ -1,3 +0,0 @@
-from .grad_norm import GradNormMonitor
-
-__all__ = ["GradNormMonitor"]

fish_speech/callbacks/grad_norm.py

@@ -1,113 +0,0 @@
-from typing import Optional, Union
-
-import lightning.pytorch as pl
-import torch
-from lightning import LightningModule, Trainer
-from lightning.pytorch.callbacks import Callback
-from torch import Tensor, nn
-from torch.utils._foreach_utils import (
-    _group_tensors_by_device_and_dtype,
-    _has_foreach_support,
-)
-
-
-@torch.no_grad()
-def grad_norm(
-    parameters: Union[Tensor, list[Tensor]],
-    norm_type: float = 2.0,
-) -> float:
-    """
-    Returns the norm of the gradients of the given parameters.
-
-    Args:
-        parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
-            single Tensor that will have gradients normalized
-        norm_type (float): type of the used p-norm.
-
-    Returns:
-        Total norm of the parameter gradients (viewed as a single vector).
-    """  # noqa: E501
-
-    if isinstance(parameters, Tensor):
-        parameters = [parameters]
-
-    grads = [p.grad for p in parameters if p.grad is not None]
-    if len(grads) == 0:
-        return None
-
-    first_device = grads[0].device
-    grouped_grads: dict[
-        tuple[torch.device, torch.dtype], list[list[Tensor]]
-    ] = _group_tensors_by_device_and_dtype(
-        [[g.detach() for g in grads]]
-    )  # type: ignore[assignment]
-
-    norms = []
-    for (device, _), ([grads], _) in grouped_grads.items():
-        if _has_foreach_support(grads, device=device):
-            norms.extend(torch._foreach_norm(grads, norm_type))
-        else:
-            norms.extend([torch.norm(g, norm_type) for g in grads])
-
-    return torch.norm(torch.stack([norm.to(first_device) for norm in norms]), norm_type)
-
-
-class GradNormMonitor(Callback):
-    """
-    Callback that computes the gradient norm of the model parameters.
-    """
-
-    def __init__(
-        self,
-        norm_type: float = 2.0,
-        logging_interval: str = "step",
-        sub_module: Optional[Union[str, list[str]]] = None,
-    ) -> None:
-        """
-        Args:
-            norm_type (float): type of the used p-norm.
-            logging_interval (str): "step" or "epoch".
-        """
-        super().__init__()
-
-        self.norm_type = norm_type
-        self.logging_interval = logging_interval
-        self.sub_module = sub_module
-
-    def on_after_backward(self, trainer: Trainer, model: LightningModule) -> None:
-        """
-        Computes the gradient norm of the model parameters and logs it to the logger.
-
-        Args:
-            trainer (Trainer): The trainer object
-            model (LightningModule): The current lightningModule
-        """
-
-        lightning_model = model
-
-        if self.sub_module is None:
-            return self.log_sub_module_grad_norm(lightning_model, model, "")
-
-        sub_modules = self.sub_module
-        if isinstance(sub_modules, str):
-            sub_modules = [sub_modules]
-
-        for sub_module in sub_modules:
-            self.log_sub_module_grad_norm(
-                lightning_model, getattr(model, sub_module), f"/{sub_module}"
-            )
-
-    def log_sub_module_grad_norm(
-        self, lightning_model: LightningModule, model: nn.Module, path: str
-    ) -> None:
-        grad_norm_val = grad_norm(model.parameters(), self.norm_type)
-        if grad_norm_val is None:
-            return
-
-        on_step = self.logging_interval == "step"
-        lightning_model.log(
-            f"train{path}/grad_norm",
-            grad_norm_val,
-            on_step=on_step,
-            on_epoch=not on_step,
-        )

fish_speech/configs/firefly_gan_vq.yaml

@@ -1,33 +0,0 @@
-_target_: fish_speech.models.vqgan.modules.firefly.FireflyArchitecture
-spec_transform:
-  _target_: fish_speech.utils.spectrogram.LogMelSpectrogram
-  sample_rate: 44100
-  n_mels: 160
-  n_fft: 2048
-  hop_length: 512
-  win_length: 2048
-backbone:
-  _target_: fish_speech.models.vqgan.modules.firefly.ConvNeXtEncoder
-  input_channels: 160
-  depths: [3, 3, 9, 3]
-  dims: [128, 256, 384, 512]
-  drop_path_rate: 0.2
-  kernel_size: 7
-head:
-  _target_: fish_speech.models.vqgan.modules.firefly.HiFiGANGenerator
-  hop_length: 512
-  upsample_rates: [8, 8, 2, 2, 2]  # aka. strides
-  upsample_kernel_sizes: [16, 16, 4, 4, 4]
-  resblock_kernel_sizes: [3, 7, 11]
-  resblock_dilation_sizes: [[1, 3, 5], [1, 3, 5], [1, 3, 5]]
-  num_mels: 512
-  upsample_initial_channel: 512
-  pre_conv_kernel_size: 13
-  post_conv_kernel_size: 13
-quantizer:
-  _target_: fish_speech.models.vqgan.modules.fsq.DownsampleFiniteScalarQuantize
-  input_dim: 512
-  n_groups: 8
-  n_codebooks: 1
-  levels: [8, 5, 5, 5]
-  downsample_factor: [2, 2]

fish_speech/configs/modded_dac_vq.yaml

@@ -0,0 +1,50 @@
+_target_: fish_speech.models.dac.modded_dac.DAC
+# Model setup
+sample_rate: 44100
+encoder_dim: 64
+encoder_rates: [2, 4, 8, 8]
+decoder_dim: 1536
+decoder_rates: [8, 8, 4, 2]
+encoder_transformer_layers: [0, 0, 0, 4]
+decoder_transformer_layers: [4, 0, 0, 0]
+transformer_general_config:
+  _target_: fish_speech.models.dac.modded_dac.ModelArgs
+  _partial_: true
+  block_size: 16384
+  n_local_heads: -1
+  head_dim: 64
+  rope_base: 10000
+  norm_eps: 1e-5
+  dropout_rate: 0.1
+  attn_dropout_rate: 0.1
+  channels_first: true
+# Quantization
+quantizer:
+  _target_: fish_speech.models.dac.rvq.DownsampleResidualVectorQuantize
+  input_dim: 1024
+  n_codebooks: 9
+  codebook_size: 1024
+  codebook_dim: 8
+  quantizer_dropout: 0.5
+  downsample_factor: [2, 2]
+  post_module: &transformer_module
+    _target_: fish_speech.models.dac.modded_dac.WindowLimitedTransformer
+    causal: true
+    window_size: 128  # empirically this does not seem to matter
+    input_dim: 1024
+    config: &transformer_config
+      _target_: fish_speech.models.dac.modded_dac.ModelArgs
+      block_size: 4096
+      n_layer: 8
+      n_head: 16
+      dim: 1024
+      intermediate_size: 3072
+      n_local_heads: -1
+      head_dim: 64
+      rope_base: 10000
+      norm_eps: 1e-5
+      dropout_rate: 0.1
+      attn_dropout_rate: 0.1
+      channels_first: true
+  pre_module: *transformer_module
+  semantic_codebook_size: 4096

fish_speech/datasets/concat_repeat.py

@@ -1,53 +0,0 @@
-import bisect
-import random
-from typing import Iterable
-
-from torch.utils.data import Dataset, IterableDataset
-
-
-class ConcatRepeatDataset(Dataset):
-    datasets: list[Dataset]
-    cumulative_sizes: list[int]
-    repeats: list[int]
-
-    @staticmethod
-    def cumsum(sequence, repeats):
-        r, s = [], 0
-        for dataset, repeat in zip(sequence, repeats):
-            l = len(dataset) * repeat
-            r.append(l + s)
-            s += l
-        return r
-
-    def __init__(self, datasets: Iterable[Dataset], repeats: list[int]):
-        super().__init__()
-
-        self.datasets = list(datasets)
-        self.repeats = repeats
-
-        assert len(self.datasets) > 0, "datasets should not be an empty iterable"
-        assert len(self.datasets) == len(
-            repeats
-        ), "datasets and repeats should have the same length"
-
-        for d in self.datasets:
-            assert not isinstance(
-                d, IterableDataset
-            ), "ConcatRepeatDataset does not support IterableDataset"
-
-        self.cumulative_sizes = self.cumsum(self.datasets, self.repeats)
-
-    def __len__(self):
-        return self.cumulative_sizes[-1]
-
-    def __getitem__(self, idx):
-        dataset_idx = bisect.bisect_right(self.cumulative_sizes, idx)
-
-        if dataset_idx == 0:
-            sample_idx = idx
-        else:
-            sample_idx = idx - self.cumulative_sizes[dataset_idx - 1]
-
-        dataset = self.datasets[dataset_idx]
-
-        return dataset[sample_idx % len(dataset)]

fish_speech/datasets/protos/text-data.proto

@@ -1,24 +0,0 @@
-syntax = "proto3";
-
-package text_data;
-
-message Semantics {
-    repeated uint32 values = 1;
-}
-
-message Sentence {
-    repeated string texts = 1;
-    repeated Semantics semantics = 3;
-}
-
-message TextData {
-    string source = 1;
-    string name = 2;
-    repeated Sentence sentences = 4;
-}
-
-message SampledData {
-    string source = 1;
-    string name = 2;
-    repeated Sentence samples = 3;
-}

fish_speech/datasets/protos/text_data_pb2.py

@@ -1,33 +0,0 @@
-# -*- coding: utf-8 -*-
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-# source: text-data.proto
-# Protobuf Python Version: 4.25.1
-"""Generated protocol buffer code."""
-from google.protobuf import descriptor as _descriptor
-from google.protobuf import descriptor_pool as _descriptor_pool
-from google.protobuf import symbol_database as _symbol_database
-from google.protobuf.internal import builder as _builder
-
-# @@protoc_insertion_point(imports)
-
-_sym_db = _symbol_database.Default()
-
-
-DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-    b'\n\x0ftext-data.proto\x12\ttext_data"\x1b\n\tSemantics\x12\x0e\n\x06values\x18\x01 \x03(\r"B\n\x08Sentence\x12\r\n\x05texts\x18\x01 \x03(\t\x12\'\n\tsemantics\x18\x03 \x03(\x0b\x32\x14.text_data.Semantics"P\n\x08TextData\x12\x0e\n\x06source\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12&\n\tsentences\x18\x04 \x03(\x0b\x32\x13.text_data.Sentence"Q\n\x0bSampledData\x12\x0e\n\x06source\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12$\n\x07samples\x18\x03 \x03(\x0b\x32\x13.text_data.Sentenceb\x06proto3'
-)
-
-_globals = globals()
-_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
-_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, "text_data_pb2", _globals)
-if _descriptor._USE_C_DESCRIPTORS == False:
-    DESCRIPTOR._options = None
-    _globals["_SEMANTICS"]._serialized_start = 30
-    _globals["_SEMANTICS"]._serialized_end = 57
-    _globals["_SENTENCE"]._serialized_start = 59
-    _globals["_SENTENCE"]._serialized_end = 125
-    _globals["_TEXTDATA"]._serialized_start = 127
-    _globals["_TEXTDATA"]._serialized_end = 207
-    _globals["_SAMPLEDDATA"]._serialized_start = 209
-    _globals["_SAMPLEDDATA"]._serialized_end = 290
-# @@protoc_insertion_point(module_scope)

fish_speech/datasets/protos/text_data_stream.py

@@ -1,36 +0,0 @@
-import struct
-
-from .text_data_pb2 import TextData
-
-
-def read_pb_stream(f):
-    while True:
-        buf = f.read(4)
-        if len(buf) == 0:
-            break
-        size = struct.unpack("I", buf)[0]
-        buf = f.read(size)
-        text_data = TextData()
-        text_data.ParseFromString(buf)
-        yield text_data
-
-
-def write_pb_stream(f, text_data):
-    buf = text_data.SerializeToString()
-    f.write(struct.pack("I", len(buf)))
-    f.write(buf)
-
-
-def pack_pb_stream(text_data):
-    buf = text_data.SerializeToString()
-    return struct.pack("I", len(buf)) + buf
-
-
-def split_pb_stream(f):
-    while True:
-        head = f.read(4)
-        if len(head) == 0:
-            break
-        size = struct.unpack("I", head)[0]
-        buf = f.read(size)
-        yield head + buf

fish_speech/datasets/semantic.py

@@ -1,639 +0,0 @@
-import random
-from dataclasses import dataclass
-from itertools import chain
-from pathlib import Path
-from random import Random
-from typing import Optional, Union
-
-import numpy as np
-import pyarrow.parquet as pq
-import torch
-import torch.nn.functional as F
-from datasets.download.streaming_download_manager import xopen
-from huggingface_hub import HfApi
-from lightning import LightningDataModule
-from torch.distributed import get_rank, get_world_size, is_initialized
-from torch.utils.data import DataLoader, Dataset, IterableDataset, get_worker_info
-
-from fish_speech.conversation import (
-    CODEBOOK_PAD_TOKEN_ID,
-    Conversation,
-    Message,
-    TextPart,
-    VQPart,
-)
-from fish_speech.datasets.protos.text_data_pb2 import SampledData
-from fish_speech.datasets.protos.text_data_stream import read_pb_stream
-from fish_speech.text.clean import clean_text
-from fish_speech.tokenizer import FishTokenizer
-from fish_speech.utils import RankedLogger
-from fish_speech.utils.braceexpand import braceexpand
-
-log = RankedLogger(__name__, rank_zero_only=True)
-
-
-def split_by_rank_worker(files):
-    # We need to know the total number of devices
-    # to split the data properly
-
-    total_devices = 1
-    if is_initialized():
-        total_devices = get_world_size()
-
-    worker_info = get_worker_info()
-    if worker_info is not None:
-        total_devices *= worker_info.num_workers
-
-    if len(files) < total_devices:
-        # Repeat the files N times to match the number of devices
-        files = files * (total_devices // len(files) + 1)
-
-    # DDP
-    if is_initialized():
-        files = files[get_rank() :: get_world_size()]
-
-    # Split by worker
-    if worker_info is not None:
-        files = files[worker_info.id :: worker_info.num_workers]
-
-    return files
-
-
-class AutoTextSemanticInstructionIterableDataset(IterableDataset):
-    """
-    Auto Augment Dataset by Speaker
-
-    1. Random concatenate multiple sentences from the same speaker to form a longer sentence
-    2. Automatically normalize the text
-
-    For interactive mode, we use the following format (multiple sequences):
-    <s> [INST] [SPK: speaker] text [/INST] ... [INST] text [/INST] </s>
-
-    For non-interactive mode, we use the following format (one long sequence):
-    <s> [INST] text [/INST] ... </s>
-    """
-
-    def __init__(
-        self,
-        proto_files: list[str],
-        seed: int = 42,
-        interactive_prob: float = 0.5,
-        max_length: int = 1024,
-        tokenizer: FishTokenizer = None,
-        use_speaker: bool | float = True,
-        causal: bool = True,
-        num_codebooks: Optional[int] = None,
-        skip_text_prob: float = 0.0,
-    ):
-        """
-        Args:
-            proto_files: proto buf files if using local data
-            seed: random seed
-            interactive_prob: probability to use interactive mode
-            max_length: max length of the text
-            tokenizer: tokenizer
-            use_speaker: include speaker information in the prompt
-            causal: use causal sampling when using local data, disable will lead to random sampling
-            num_codebooks: number of codebooks, if None, it will be automatically detected
-            skip_text_prob: probability to skip the text (audio only), this only applies to interactive mode
-        """
-
-        super().__init__()
-
-        assert 0 <= interactive_prob <= 1, "interactive_prob must be in [0, 1]"
-
-        self.seed = seed
-        self.max_length = max_length
-        self.tokenizer = tokenizer
-        self.interactive_prob = interactive_prob
-        self.use_speaker = use_speaker
-        self.proto_files = proto_files
-        self.causal = causal
-        self.num_codebooks = num_codebooks
-        self.skip_text_prob = skip_text_prob
-
-        self.groups = None
-
-    def __iter__(self):
-        while True:
-            yield self.augment()
-
-    def init_mock_data_server(self):
-        if self.groups is not None:
-            return
-
-        # Expand the proto files
-        expanded_proto_files = []
-        for filename in self.proto_files:
-            for i in braceexpand(filename):
-                i = Path(i)
-                if i.is_file():
-                    expanded_proto_files.append(i)
-                elif i.is_dir():
-                    expanded_proto_files.extend(i.rglob("*.proto"))
-                    expanded_proto_files.extend(i.rglob("*.protos"))
-                else:
-                    raise ValueError(f"{i} is not a file or directory")
-
-        expanded_proto_files = sorted(expanded_proto_files)
-        Random(self.seed).shuffle(expanded_proto_files)
-
-        self.groups = []
-        shard_proto_files = split_by_rank_worker(expanded_proto_files)
-        log.info(
-            f"Reading {len(shard_proto_files)} / {len(expanded_proto_files)} files"
-        )
-
-        count = 0
-        for filename in shard_proto_files:
-            with open(filename, "rb") as f:
-                for text_data in read_pb_stream(f):
-                    self.groups.append(text_data)
-                    count += 1
-
-        log.info(f"Read total {count} groups of data")
-
-        # Shuffle the lines
-        Random(self.seed).shuffle(self.groups)
-        self.group_weights = [len(i.sentences) for i in self.groups]
-
-    def sample_data(self):
-        if self.groups is None:
-            self.init_mock_data_server()
-
-        # Shuffle unique lines, estimate that each sample is at least 20 tokens
-        num_samples = self.max_length // 20
-
-        # choice group based on their number of samples
-        group = random.choices(self.groups, weights=self.group_weights, k=1)[0]
-
-        if self.causal:
-            # Sample in order
-            if num_samples >= len(group.sentences):
-                samples = group.sentences
-            else:
-                begin = random.randint(0, len(group.sentences) - num_samples)
-                samples = group.sentences[begin : begin + num_samples]
-        else:
-            samples = random.choices(
-                group.sentences, k=min(num_samples, len(group.sentences))
-            )
-
-        return SampledData(
-            source=group.source,
-            name=group.name,
-            samples=samples,
-        )
-
-    def pack_sentences(
-        self,
-        sentences: list[str],
-        semantics: list,
-        # speaker: Optional[str] = None,
-        skip_text: bool = False,
-    ):
-        # if speaker is None:
-        #     speaker = "assistant"
-
-        messages = [
-            Message(
-                role="system",
-                parts=[TextPart(text="Speak out the provided text.")],
-                # add_im_end=False,
-                # cal_loss=True,
-            )
-        ]
-
-        cated_sentences = " ".join(sentences)
-        if skip_text:
-            cated_sentences = "<|skip_text|>"
-
-        messages.append(
-            Message(
-                role="user",
-                parts=[TextPart(text=cated_sentences)],
-                # cal_loss=True,
-            )
-        )
-
-        vq_codes = [x.values for x in semantics[0]]
-        vq_codes_tensor = torch.tensor(vq_codes).to(torch.int32)
-        vqpart = VQPart(codes=vq_codes_tensor)
-        messages.append(
-            Message(
-                role="assistant",
-                parts=[TextPart(text="<|voice|>"), vqpart],
-                cal_loss=True,
-            )
-        )
-
-        num_codebooks = (
-            len(semantics[0]) if self.num_codebooks is None else self.num_codebooks
-        )
-
-        conversation = Conversation(messages=messages)
-        # conversation.visualize(tokenizer=self.tokenizer)
-        encoded = conversation.encode(
-            tokenizer=self.tokenizer,
-        )
-
-        tokens_raw = encoded.tokens
-        tokens = torch.zeros((num_codebooks + 1, len(tokens_raw)), dtype=torch.int)
-        tokens[0] = tokens_raw
-
-        vq_parts = encoded.vq_parts
-        vq_parts = [part.to(tokens.device) for part in vq_parts]
-        vq_parts = torch.cat(vq_parts, dim=1)
-        tokens[1:, encoded.vq_mask_tokens] = vq_parts
-
-        labels_raw = encoded.labels
-        labels = torch.full((num_codebooks + 1, len(labels_raw)), -100, dtype=torch.int)
-        labels[0, :] = labels_raw
-        labels[1:, encoded.vq_mask_labels] = vq_parts
-        labels[1:, -1:] = CODEBOOK_PAD_TOKEN_ID
-
-        tokens = tokens.long()
-        labels = labels.long()
-
-        # Verify the padding is correct, and the last token is eos
-        assert (tokens[1:, ~(encoded.vq_mask_tokens)] == CODEBOOK_PAD_TOKEN_ID).all()
-        assert (labels[1:, -1:] == CODEBOOK_PAD_TOKEN_ID).all()
-
-        return tokens, labels
-
-    def augment(self):
-        response = self.sample_data()
-        if len(response.samples) == 0:
-            # Invalid group
-            return None
-
-        samples = list(response.samples)
-        all_tokens, all_labels = [], []
-
-        while len(samples) > 0:
-            sentence = samples.pop(0)
-            text = clean_text(random.choice(sentence.texts))
-
-            tokens, labels = self.pack_sentences(
-                sentences=[text],
-                semantics=[sentence.semantics],
-                # speaker=response.name if use_speaker else None,
-                skip_text=random.random() < self.skip_text_prob,
-            )
-
-            all_tokens.append(tokens)
-            all_labels.append(labels)
-
-        tokens = torch.cat(all_tokens, dim=1)
-        labels = torch.cat(all_labels, dim=1)
-
-        # Verify that the length is correct
-        assert tokens.size(1) == labels.size(1), f"{tokens.size(1)} != {labels.size(1)}"
-
-        data = {"tokens": tokens, "labels": labels}
-
-        return data
-
-
-class AutoTextSemanticInstructionDataset(Dataset):
-    """
-    Auto Augment Dataset by Speaker
-
-    1. Random concatenate multiple sentences from the same speaker to form a longer sentence
-    2. Automatically normalize the text
-
-    For interactive mode, we use the following format (multiple sequences):
-    <s> [INST] [SPK: speaker] text [/INST] ... [INST] text [/INST] </s>
-
-    For non-interactive mode, we use the following format (one long sequence):
-    <s> [INST] text [/INST] ... </s>
-    """
-
-    def __init__(
-        self,
-        proto_files: list[str],
-        seed: int = 42,
-        interactive_prob: float = 0.5,
-        max_length: int = 1024,
-        tokenizer: FishTokenizer = None,
-        use_speaker: bool | float = True,
-        causal: bool = True,
-        num_codebooks: Optional[int] = None,
-        skip_text_prob: float = 0.0,
-    ):
-        """
-        Args:
-            proto_files: proto buf files if using local data
-            seed: random seed
-            interactive_prob: probability to use interactive mode
-            max_length: max length of the text
-            tokenizer: tokenizer
-            use_speaker: include speaker information in the prompt
-            causal: use causal sampling when using local data, disable will lead to random sampling
-            num_codebooks: number of codebooks, if None, it will be automatically detected
-            skip_text_prob: probability to skip the text (audio only), this only applies to interactive mode
-        """
-        super().__init__()
-
-        assert 0 <= interactive_prob <= 1, "interactive_prob must be in [0, 1]"
-
-        self.seed = seed
-        self.max_length = max_length
-        self.tokenizer = tokenizer
-        self.interactive_prob = interactive_prob
-        self.use_speaker = use_speaker
-        self.proto_files = proto_files
-        self.causal = causal
-        self.num_codebooks = num_codebooks
-        self.skip_text_prob = skip_text_prob
-
-        self.data = []
-        self._init_data()
-
-    def _init_data(self):
-        expanded_proto_files = []
-        for filename in self.proto_files:
-            for i in braceexpand(filename):
-                i = Path(i)
-                if i.is_file():
-                    expanded_proto_files.append(i)
-                elif i.is_dir():
-                    expanded_proto_files.extend(i.rglob("*.proto"))
-                    expanded_proto_files.extend(i.rglob("*.protos"))
-                else:
-                    raise ValueError(f"{i} is not a file or directory")
-
-        expanded_proto_files = sorted(expanded_proto_files)
-        Random(self.seed).shuffle(expanded_proto_files)
-
-        groups = []
-        shard_proto_files = split_by_rank_worker(expanded_proto_files)
-        log.info(
-            f"Reading {len(shard_proto_files)} / {len(expanded_proto_files)} files"
-        )
-
-        count = 0
-        for filename in shard_proto_files:
-            with open(filename, "rb") as f:
-                for text_data in read_pb_stream(f):
-                    groups.append(text_data)
-                    count += 1
-
-        log.info(f"Read total {count} groups of data")
-
-        for group in groups:
-            if len(group.sentences) == 0:
-                continue
-
-            samples = list(group.sentences)
-            for sentence in samples:
-                text = clean_text(random.choice(sentence.texts))
-
-                tokens, labels = self.pack_sentences(
-                    sentences=[text],
-                    semantics=[sentence.semantics],
-                    skip_text=random.random() < self.skip_text_prob,
-                )
-
-                self.data.append({"tokens": tokens, "labels": labels})
-
-        random.Random(self.seed).shuffle(self.data)
-
-    def __len__(self):
-        return len(self.data)
-
-    def __getitem__(self, idx):
-        return self.data[idx]
-
-    def pack_sentences(
-        self,
-        sentences: list[str],
-        semantics: list,
-        skip_text: bool = False,
-    ):
-        messages = [
-            Message(
-                role="system",
-                parts=[TextPart(text="Speak out the provided text.")],
-            )
-        ]
-
-        cated_sentences = " ".join(sentences)
-        if skip_text:
-            cated_sentences = "<|skip_text|>"
-
-        messages.append(
-            Message(
-                role="user",
-                parts=[TextPart(text=cated_sentences)],
-            )
-        )
-
-        vq_codes = [x.values for x in semantics[0]]
-        vq_codes_tensor = torch.tensor(vq_codes).to(torch.int32)
-        vqpart = VQPart(codes=vq_codes_tensor)
-        messages.append(
-            Message(
-                role="assistant",
-                parts=[TextPart(text="<|voice|>"), vqpart],
-                cal_loss=True,
-            )
-        )
-
-        num_codebooks = (
-            len(semantics[0]) if self.num_codebooks is None else self.num_codebooks
-        )
-
-        conversation = Conversation(messages=messages)
-        encoded = conversation.encode(
-            tokenizer=self.tokenizer,
-        )
-
-        tokens_raw = encoded.tokens
-        tokens = torch.zeros((num_codebooks + 1, len(tokens_raw)), dtype=torch.int)
-        tokens[0] = tokens_raw
-
-        vq_parts = encoded.vq_parts
-        vq_parts = [part.to(tokens.device) for part in vq_parts]
-        vq_parts = torch.cat(vq_parts, dim=1)
-        tokens[1:, encoded.vq_mask_tokens] = vq_parts
-
-        labels_raw = encoded.labels
-        labels = torch.full((num_codebooks + 1, len(labels_raw)), -100, dtype=torch.int)
-        labels[0, :] = labels_raw
-        labels[1:, encoded.vq_mask_labels] = vq_parts
-        labels[1:, -1:] = CODEBOOK_PAD_TOKEN_ID
-
-        tokens = tokens.long()
-        labels = labels.long()
-
-        assert (tokens[1:, ~(encoded.vq_mask_tokens)] == CODEBOOK_PAD_TOKEN_ID).all()
-        assert (labels[1:, -1:] == CODEBOOK_PAD_TOKEN_ID).all()
-
-        return tokens, labels
-
-
-class InterleaveDataset(IterableDataset):
-    def __init__(
-        self,
-        datasets: list[IterableDataset],
-        probabilities: list[float],
-        seed: int = 42,
-    ):
-        super().__init__()
-
-        self.datasets = datasets
-        self.probabilities = probabilities
-        self.seed = seed
-
-    def __iter__(self):
-        rng = np.random.default_rng(self.seed)
-        dataset_iterators = [iter(dataset) for dataset in self.datasets]
-
-        while True:
-            # Random choice one
-            dataset_idx = rng.choice(len(self.datasets), p=self.probabilities)
-            dataset_iterator = dataset_iterators[dataset_idx]
-
-            try:
-                yield next(dataset_iterator)
-            except StopIteration:
-                # Exhausted, create a new iterator
-                dataset_iterators[dataset_idx] = iter(self.datasets[dataset_idx])
-                yield next(dataset_iterators[dataset_idx])
-
-
-@dataclass
-class TextDataCollator:
-    tokenizer: FishTokenizer
-    max_length: int = 1024
-
-    def __call__(self, examples):
-        if "negative_tokens" in examples:
-            positive_examples = []
-            negative_examples = []
-
-            for i in examples:
-                positive_examples.append(
-                    {
-                        "tokens": i["tokens"],
-                        "labels": i["labels"],
-                    }
-                )
-                negative_examples.append(
-                    {
-                        "tokens": i["negative_tokens"],
-                        "labels": i["negative_labels"],
-                    }
-                )
-
-            examples = positive_examples + negative_examples
-
-        return self.batchify(examples)
-
-    def batchify(self, examples, tokens_key="tokens", labels_key="labels"):
-        tokens, attention_masks, labels = [], [], []
-
-        # Calculate the max length
-        max_tokens_length = 0
-        for example in examples:
-            max_tokens_length = max(max_tokens_length, example[tokens_key].size(1))
-        max_tokens_length = min(max_tokens_length, self.max_length)
-
-        for example in examples:
-            _tokens = example[tokens_key][:, :max_tokens_length]
-            _labels = example[labels_key][:, :max_tokens_length]
-            _attention_mask = torch.ones((max_tokens_length,), dtype=torch.bool)
-            tokens_length = _tokens.size(1)
-            _attention_mask[:tokens_length] = False
-
-            assert tokens_length == _labels.size(
-                1
-            ), f"{tokens_length} != {_labels.size(1)}"
-
-            if tokens_length < max_tokens_length:
-                _tokens = F.pad(
-                    _tokens,
-                    (0, max_tokens_length - tokens_length),
-                    value=self.tokenizer.get_token_id("<|end_of_text|>"),
-                )
-                _tokens[1:, tokens_length:] = CODEBOOK_PAD_TOKEN_ID
-                _labels = F.pad(
-                    _labels, (0, max_tokens_length - _labels.size(1)), value=-100
-                )
-
-            tokens.append(_tokens)
-            attention_masks.append(_attention_mask)
-            labels.append(_labels)
-
-        tokens = torch.stack(tokens, dim=0)
-        attention_masks = torch.stack(attention_masks, dim=0)
-        labels = torch.stack(labels, dim=0)
-
-        return {
-            "inputs": tokens,
-            "attention_masks": attention_masks,
-            "labels": labels,
-        }
-
-
-class SemanticDataModule(LightningDataModule):
-    def __init__(
-        self,
-        train_dataset: Union[
-            AutoTextSemanticInstructionDataset,
-            AutoTextSemanticInstructionIterableDataset,
-            InterleaveDataset,
-        ],
-        val_dataset: Union[
-            AutoTextSemanticInstructionDataset,
-            AutoTextSemanticInstructionIterableDataset,
-            InterleaveDataset,
-        ],
-        batch_size: int = 32,
-        tokenizer: FishTokenizer = None,
-        max_length: int = 1024,
-        num_workers: int = 4,
-    ):
-        super().__init__()
-
-        self.train_dataset = train_dataset
-        self.val_dataset = val_dataset
-        self.batch_size = batch_size
-        self.tokenizer = tokenizer
-        self.max_length = max_length
-        self.num_workers = num_workers
-
-    def train_dataloader(self):
-        return DataLoader(
-            self.train_dataset,
-            batch_size=self.batch_size,
-            collate_fn=TextDataCollator(self.tokenizer, self.max_length),
-            num_workers=self.num_workers,
-            persistent_workers=True,
-        )
-
-    def val_dataloader(self):
-        return DataLoader(
-            self.val_dataset,
-            batch_size=self.batch_size,
-            collate_fn=TextDataCollator(self.tokenizer, self.max_length),
-            num_workers=self.num_workers,
-            persistent_workers=True,
-        )
-
-
-if __name__ == "__main__":
-    from tqdm import tqdm
-
-    ds = AutoTextSemanticInstructionDataset(
-        ["data/protos"],
-        tokenizer=FishTokenizer("checkpoints/fish-speech-1.5/tokenizer.tiktoken"),
-        use_speaker=False,
-        interactive_prob=1.0,
-        skip_text_prob=0.5,
-    )
-
-    for i in range(100):
-        # Please uncomment line 235 to visualize the tokenized message
-        print(ds[i])

fish_speech/datasets/vqgan.py

@@ -1,147 +0,0 @@
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Optional
-
-import librosa
-import numpy as np
-import torch
-from lightning import LightningDataModule
-from torch.utils.data import DataLoader, Dataset
-
-from fish_speech.utils import RankedLogger
-
-logger = RankedLogger(__name__, rank_zero_only=False)
-
-
-class VQGANDataset(Dataset):
-    def __init__(
-        self,
-        filelist: str,
-        sample_rate: int = 32000,
-        hop_length: int = 640,
-        slice_frames: Optional[int] = None,
-    ):
-        super().__init__()
-
-        filelist = Path(filelist)
-        root = filelist.parent
-
-        self.files = [
-            root / line.strip()
-            for line in filelist.read_text(encoding="utf-8").splitlines()
-            if line.strip()
-        ]
-        self.sample_rate = sample_rate
-        self.hop_length = hop_length
-        self.slice_frames = slice_frames
-
-    def __len__(self):
-        return len(self.files)
-
-    def get_item(self, idx):
-        file = self.files[idx]
-
-        audio, _ = librosa.load(file, sr=self.sample_rate, mono=True)
-
-        # Slice audio and features
-        if (
-            self.slice_frames is not None
-            and audio.shape[0] > self.slice_frames * self.hop_length
-        ):
-            start = np.random.randint(
-                0, audio.shape[0] - self.slice_frames * self.hop_length
-            )
-            audio = audio[start : start + self.slice_frames * self.hop_length]
-
-        if len(audio) == 0:
-            return None
-
-        max_value = np.abs(audio).max()
-        if max_value > 1.0:
-            audio = audio / max_value
-
-        return {
-            "audio": torch.from_numpy(audio),
-        }
-
-    def __getitem__(self, idx):
-        try:
-            return self.get_item(idx)
-        except Exception as e:
-            import traceback
-
-            traceback.print_exc()
-            logger.error(f"Error loading {self.files[idx]}: {e}")
-            return None
-
-
-@dataclass
-class VQGANCollator:
-    def __call__(self, batch):
-        batch = [x for x in batch if x is not None]
-
-        audio_lengths = torch.tensor([len(x["audio"]) for x in batch])
-        audio_maxlen = audio_lengths.max()
-
-        # Rounds up to nearest multiple of 2 (audio_lengths)
-        audios = []
-        for x in batch:
-            audios.append(
-                torch.nn.functional.pad(x["audio"], (0, audio_maxlen - len(x["audio"])))
-            )
-
-        return {
-            "audios": torch.stack(audios),
-            "audio_lengths": audio_lengths,
-        }
-
-
-class VQGANDataModule(LightningDataModule):
-    def __init__(
-        self,
-        train_dataset: VQGANDataset,
-        val_dataset: VQGANDataset,
-        batch_size: int = 32,
-        num_workers: int = 4,
-        val_batch_size: Optional[int] = None,
-    ):
-        super().__init__()
-
-        self.train_dataset = train_dataset
-        self.val_dataset = val_dataset
-        self.batch_size = batch_size
-        self.val_batch_size = val_batch_size or batch_size
-        self.num_workers = num_workers
-
-    def train_dataloader(self):
-        return DataLoader(
-            self.train_dataset,
-            batch_size=self.batch_size,
-            collate_fn=VQGANCollator(),
-            num_workers=self.num_workers,
-            shuffle=True,
-            persistent_workers=True,
-        )
-
-    def val_dataloader(self):
-        return DataLoader(
-            self.val_dataset,
-            batch_size=self.val_batch_size,
-            collate_fn=VQGANCollator(),
-            num_workers=self.num_workers,
-            persistent_workers=True,
-        )
-
-
-if __name__ == "__main__":
-    dataset = VQGANDataset("data/LibriTTS_R/vq_train_filelist.txt")
-    dataloader = DataLoader(
-        dataset, batch_size=4, shuffle=False, collate_fn=VQGANCollator()
-    )
-
-    for batch in dataloader:
-        print(batch["audios"].shape)
-        print(batch["features"].shape)
-        print(batch["audio_lengths"])
-        print(batch["feature_lengths"])
-        break

fish_speech/inference_engine/__init__.py

@@ -9,12 +9,12 @@ from loguru import logger
 from fish_speech.inference_engine.reference_loader import ReferenceLoader
 from fish_speech.inference_engine.utils import InferenceResult, wav_chunk_header
 from fish_speech.inference_engine.vq_manager import VQManager
+from fish_speech.models.dac.modded_dac import DAC
 from fish_speech.models.text2semantic.inference import (
     GenerateRequest,
     GenerateResponse,
     WrappedGenerateResponse,
 )
-from fish_speech.models.vqgan.modules.firefly import FireflyArchitecture
 from fish_speech.utils import autocast_exclude_mps, set_seed
 from fish_speech.utils.schema import ServeTTSRequest
 
@@ -24,7 +24,7 @@ class TTSInferenceEngine(ReferenceLoader, VQManager):
     def __init__(
         self,
         llama_queue: queue.Queue,
-        decoder_model: FireflyArchitecture,
+        decoder_model: DAC,
         precision: torch.dtype,
         compile: bool,
     ) -> None:

fish_speech/inference_engine/reference_loader.py

@@ -7,7 +7,7 @@ import torch
 import torchaudio
 from loguru import logger
 
-from fish_speech.models.vqgan.modules.firefly import FireflyArchitecture
+from fish_speech.models.dac.modded_dac import DAC
 from fish_speech.utils.file import (
     AUDIO_EXTENSIONS,
     audio_to_bytes,
@@ -28,7 +28,7 @@ class ReferenceLoader:
         self.ref_by_hash: dict = {}
 
         # Make Pylance happy (attribut/method not defined...)
-        self.decoder_model: FireflyArchitecture
+        self.decoder_model: DAC
         self.encode_reference: Callable
 
         # Define the torchaudio backend

fish_speech/inference_engine/vq_manager.py

@@ -3,14 +3,14 @@ from typing import Callable
 import torch
 from loguru import logger
 
-from fish_speech.models.vqgan.modules.firefly import FireflyArchitecture
+from fish_speech.models.dac.modded_dac import DAC
 
 
 class VQManager:
 
     def __init__(self):
         # Make Pylance happy (attribut/method not defined...)
-        self.decoder_model: FireflyArchitecture
+        self.decoder_model: DAC
         self.load_audio: Callable
 
     def decode_vq_tokens(self, codes):
@@ -19,7 +19,7 @@ class VQManager:
         )
         logger.info(f"VQ features: {codes.shape}")
 
-        if isinstance(self.decoder_model, FireflyArchitecture):
+        if isinstance(self.decoder_model, DAC):
             return self.decoder_model.decode(
                 indices=codes[None],
                 feature_lengths=feature_lengths,
@@ -45,7 +45,7 @@ class VQManager:
             )
 
             # VQ Encoder
-            if isinstance(self.decoder_model, FireflyArchitecture):
+            if isinstance(self.decoder_model, DAC):
                 prompt_tokens = self.decoder_model.encode(audios, audio_lengths)[0][0]
                 logger.info(f"Encoded prompt: {prompt_tokens.shape}")
             else:

fish_speech/models/vqgan/inference.py → fish_speech/models/dac/inference.py

@@ -78,18 +78,16 @@ def main(input_path, output_path, config_name, checkpoint_path, device):
         audio, sr = torchaudio.load(str(input_path))
         if audio.shape[0] > 1:
             audio = audio.mean(0, keepdim=True)
-        audio = torchaudio.functional.resample(
-            audio, sr, model.spec_transform.sample_rate
-        )
+        audio = torchaudio.functional.resample(audio, sr, model.sample_rate)
 
         audios = audio[None].to(device)
         logger.info(
-            f"Loaded audio with {audios.shape[2] / model.spec_transform.sample_rate:.2f} seconds"
+            f"Loaded audio with {audios.shape[2] / model.sample_rate:.2f} seconds"
         )
 
         # VQ Encoder
         audio_lengths = torch.tensor([audios.shape[2]], device=device, dtype=torch.long)
-        indices = model.encode(audios, audio_lengths)[0][0]
+        indices, indices_lens = model.encode(audios, audio_lengths)
 
         logger.info(f"Generated indices of shape {indices.shape}")
 
@@ -100,15 +98,13 @@ def main(input_path, output_path, config_name, checkpoint_path, device):
         indices = np.load(input_path)
         indices = torch.from_numpy(indices).to(device).long()
         assert indices.ndim == 2, f"Expected 2D indices, got {indices.ndim}"
+        indices_lens = torch.tensor([indices.shape[1]], device=device, dtype=torch.long)
     else:
         raise ValueError(f"Unknown input type: {input_path}")
 
     # Restore
-    feature_lengths = torch.tensor([indices.shape[1]], device=device)
-    fake_audios, _ = model.decode(
-        indices=indices[None], feature_lengths=feature_lengths
-    )
-    audio_time = fake_audios.shape[-1] / model.spec_transform.sample_rate
+    fake_audios, audio_lengths = model.decode(indices, indices_lens)
+    audio_time = fake_audios.shape[-1] / model.sample_rate
 
     logger.info(
         f"Generated audio of shape {fake_audios.shape}, equivalent to {audio_time:.2f} seconds from {indices.shape[1]} features, features/second: {indices.shape[1] / audio_time:.2f}"
@@ -116,7 +112,7 @@ def main(input_path, output_path, config_name, checkpoint_path, device):
 
     # Save audio
     fake_audio = fake_audios[0, 0].float().cpu().numpy()
-    sf.write(output_path, fake_audio, model.spec_transform.sample_rate)
+    sf.write(output_path, fake_audio, model.sample_rate)
     logger.info(f"Saved audio to {output_path}")
 
 

fish_speech/models/dac/modded_dac.py

@@ -0,0 +1,1021 @@
+import math
+import typing as tp
+from dataclasses import dataclass
+from typing import List, Optional, Union
+
+import hydra
+import librosa
+import numpy as np
+import soundfile as sf
+import torch
+from audiotools import AudioSignal
+from audiotools.ml import BaseModel
+from dac.model.base import CodecMixin
+from dac.nn.layers import Snake1d, WNConv1d, WNConvTranspose1d
+from omegaconf import OmegaConf
+from torch import Tensor, nn
+from torch.nn import functional as F
+from torch.nn.utils.parametrizations import weight_norm
+from torch.nn.utils.parametrize import remove_parametrizations
+
+
+@dataclass
+class VQResult:
+    z: torch.Tensor
+    codes: torch.Tensor
+    latents: torch.Tensor
+    codebook_loss: torch.Tensor
+    commitment_loss: torch.Tensor
+    semantic_distill_z: torch.Tensor | None = None
+
+
+def find_multiple(n: int, k: int) -> int:
+    if n % k == 0:
+        return n
+    return n + k - (n % k)
+
+
+@dataclass
+class ModelArgs:
+    block_size: int = 2048
+    n_layer: int = 8
+    n_head: int = 8
+    dim: int = 512
+    intermediate_size: int = 1536
+    n_local_heads: int = -1
+    head_dim: int = 64
+    rope_base: float = 10000
+    norm_eps: float = 1e-5
+    dropout_rate: float = 0.1
+    attn_dropout_rate: float = 0.1
+    channels_first: bool = True  # to be compatible with conv1d input/output
+    pos_embed_type: str = "rope"  # can be "rope" or "conformer"
+    max_relative_position: int = 128  # for conformer-style relative position embedding
+
+    def __post_init__(self):
+        if self.n_local_heads == -1:
+            self.n_local_heads = self.n_head
+        if self.intermediate_size is None:
+            hidden_dim = 4 * self.dim
+            n_hidden = int(2 * hidden_dim / 3)
+            self.intermediate_size = find_multiple(n_hidden, 256)
+        assert self.pos_embed_type in [
+            "rope",
+            "conformer",
+        ], "pos_embed_type must be either 'rope' or 'conformer'"
+
+
+class KVCache(nn.Module):
+    def __init__(
+        self, max_batch_size, max_seq_length, n_heads, head_dim, dtype=torch.bfloat16
+    ):
+        super().__init__()
+        cache_shape = (max_batch_size, n_heads, max_seq_length, head_dim)
+        self.register_buffer("k_cache", torch.zeros(cache_shape, dtype=dtype))
+        self.register_buffer("v_cache", torch.zeros(cache_shape, dtype=dtype))
+
+    def update(self, input_pos, k_val, v_val):
+        # input_pos: [S], k_val: [B, H, S, D]
+        assert input_pos.shape[0] == k_val.shape[2]
+
+        k_out = self.k_cache
+        v_out = self.v_cache
+        k_out[:, :, input_pos] = k_val
+        v_out[:, :, input_pos] = v_val
+
+        return (
+            k_out[:, :, : input_pos.max() + 1, :],
+            v_out[:, :, : input_pos.max() + 1, :],
+        )
+
+    def clear_cache(self, prompt_len):
+        self.k_cache[:, :, prompt_len:, :].fill_(0)
+        self.v_cache[:, :, prompt_len:, :].fill_(0)
+
+
+class Transformer(nn.Module):
+    def __init__(self, config: ModelArgs) -> None:
+        super().__init__()
+        self.config = config
+
+        self.layers = nn.ModuleList(
+            TransformerBlock(config) for _ in range(config.n_layer)
+        )
+        self.norm = RMSNorm(config.dim, eps=config.norm_eps)
+
+        # Only compute RoPE frequencies if using RoPE
+        if config.pos_embed_type == "rope":
+            freqs_cis = precompute_freqs_cis(
+                self.config.block_size, self.config.head_dim, self.config.rope_base
+            )
+            self.register_buffer("freqs_cis", freqs_cis)
+        else:
+            self.register_buffer("freqs_cis", None)
+
+        causal_mask = torch.tril(
+            torch.ones(self.config.block_size, self.config.block_size, dtype=torch.bool)
+        )
+        self.register_buffer("causal_mask", causal_mask)
+
+        self.max_batch_size = -1
+        self.max_seq_length = -1
+        self.use_kv_cache = False
+
+    def setup_caches(self, max_batch_size, max_seq_length):
+        """
+        This method will only be called during inference when using KV cache.
+        """
+        head_dim = self.config.dim // self.config.n_head
+        max_seq_length = find_multiple(max_seq_length, 8)
+        self.max_seq_length = max_seq_length
+        self.max_batch_size = max_batch_size
+        dtype = self.norm.weight.dtype
+        device = self.norm.weight.device
+
+        for b in self.layers:
+            b.attention.kv_cache = KVCache(
+                max_batch_size,
+                max_seq_length,
+                self.config.n_local_heads,
+                head_dim,
+                dtype,
+            ).to(device)
+
+        self.use_kv_cache = True
+
+    def forward(
+        self,
+        x: Tensor,
+        input_pos: Optional[Tensor] = None,
+        mask: Optional[Tensor] = None,
+    ) -> Tensor:
+        if self.config.pos_embed_type == "rope":
+            assert (
+                self.freqs_cis is not None
+            ), "RoPE frequencies must be initialized for RoPE positional embedding"
+            freqs_cis = self.freqs_cis[input_pos]
+        else:
+            freqs_cis = None
+
+        if mask is None:  # in case of non-causal model
+            if not self.training and self.use_kv_cache:
+                mask = self.causal_mask[None, None, input_pos]
+                mask = mask[..., : input_pos.max() + 1]
+            else:
+                mask = self.causal_mask[None, None, input_pos]
+                mask = mask[..., input_pos]
+
+        for i, layer in enumerate(self.layers):
+            x = layer(x, input_pos, freqs_cis, mask)
+        x = self.norm(x)
+        return x
+
+
+class TransformerBlock(nn.Module):
+    def __init__(self, config: ModelArgs) -> None:
+        super().__init__()
+        self.attention = Attention(config)
+        self.feed_forward = FeedForward(config)
+        self.ffn_norm = RMSNorm(config.dim, eps=config.norm_eps)
+        self.attention_norm = RMSNorm(config.dim, eps=config.norm_eps)
+        self.attention_layer_scale = LayerScale(config.dim, inplace=True)
+        self.ffn_layer_scale = LayerScale(config.dim, inplace=True)
+
+    def forward(
+        self,
+        x: Tensor,
+        input_pos: Tensor,
+        freqs_cis: Tensor,
+        mask: Tensor,
+    ) -> Tensor:
+        h = x + self.attention_layer_scale(
+            self.attention(self.attention_norm(x), freqs_cis, mask, input_pos)
+        )
+        out = h + self.ffn_layer_scale(self.feed_forward(self.ffn_norm(h)))
+        return out
+
+
+class Attention(nn.Module):
+    def __init__(self, config: ModelArgs):
+        super().__init__()
+        assert config.dim % config.n_head == 0
+
+        total_head_dim = (config.n_head + 2 * config.n_local_heads) * config.head_dim
+        # key, query, value projections for all heads, but in a batch
+        self.wqkv = nn.Linear(config.dim, total_head_dim, bias=False)
+        self.wo = nn.Linear(config.head_dim * config.n_head, config.dim, bias=False)
+        self.kv_cache = None
+
+        self.n_head = config.n_head
+        self.head_dim = config.head_dim
+        self.n_local_heads = config.n_local_heads
+        self.dim = config.dim
+        self.attn_dropout_rate = config.attn_dropout_rate
+        self.pos_embed_type = config.pos_embed_type
+
+        # Add relative position embedding for conformer-style
+        if self.pos_embed_type == "conformer":
+            self.max_relative_position = config.max_relative_position
+            num_pos_embeddings = 2 * config.max_relative_position + 1
+            self.rel_pos_embeddings = nn.Parameter(
+                torch.zeros(num_pos_embeddings, self.head_dim)
+            )
+            nn.init.normal_(self.rel_pos_embeddings, mean=0.0, std=0.02)
+
+    def _compute_conformer_pos_scores(self, q: Tensor, seqlen: int) -> Tensor:
+        # q: [B, H, S, D]
+        # Returns: [B, H, S, S]
+        positions = torch.arange(seqlen, device=q.device)
+        relative_positions = positions.unsqueeze(1) - positions.unsqueeze(0)  # [S, S]
+        relative_positions = torch.clamp(
+            relative_positions + self.max_relative_position,
+            0,
+            2 * self.max_relative_position,
+        )
+        rel_embeddings = self.rel_pos_embeddings[relative_positions]  # [S, S, D]
+
+        # Compute attention scores with relative position embeddings
+        q = q.transpose(1, 2)  # [B, S, H, D]
+        rel_logits = torch.matmul(q, rel_embeddings.transpose(-2, -1))  # [B, S, H, S]
+        rel_logits = rel_logits.transpose(1, 2)  # [B, H, S, S]
+        return rel_logits
+
+    def forward(
+        self,
+        x: Tensor,
+        freqs_cis: Tensor,
+        mask: Tensor,
+        input_pos: Optional[Tensor] = None,
+    ) -> Tensor:
+        bsz, seqlen, _ = x.shape
+
+        kv_size = self.n_local_heads * self.head_dim
+        q, k, v = self.wqkv(x).split([kv_size, kv_size, kv_size], dim=-1)
+        context_seqlen = seqlen
+
+        q = q.view(bsz, seqlen, self.n_head, self.head_dim)
+        k = k.view(bsz, context_seqlen, self.n_local_heads, self.head_dim)
+        v = v.view(bsz, context_seqlen, self.n_local_heads, self.head_dim)
+
+        if self.pos_embed_type == "rope":
+            q = apply_rotary_emb(q, freqs_cis)
+            k = apply_rotary_emb(k, freqs_cis)
+
+        q, k, v = map(lambda x: x.transpose(1, 2), (q, k, v))
+
+        if self.kv_cache is not None:
+            k, v = self.kv_cache.update(input_pos, k, v)
+
+        k = k.repeat_interleave(self.n_head // self.n_local_heads, dim=1)
+        v = v.repeat_interleave(self.n_head // self.n_local_heads, dim=1)
+
+        if self.pos_embed_type == "conformer":
+            # Compute attention scores
+            scale = 1.0 / math.sqrt(self.head_dim)
+            scores = torch.matmul(q, k.transpose(-2, -1)) * scale
+
+            # Add relative position embeddings for conformer-style
+            rel_scores = self._compute_conformer_pos_scores(q, seqlen)
+            scores = scores + rel_scores
+
+            # Apply attention
+            if mask is not None:
+                scores = scores.masked_fill(~mask, float("-inf"))
+
+            attn = F.softmax(scores, dim=-1)
+            if self.attn_dropout_rate > 0 and self.training:
+                attn = F.dropout(attn, p=self.attn_dropout_rate)
+
+            y = torch.matmul(attn, v)
+        else:
+            y = F.scaled_dot_product_attention(
+                q,
+                k,
+                v,
+                dropout_p=self.attn_dropout_rate if self.training else 0.0,
+                attn_mask=mask,
+            )
+            # is_causal=True)
+        y = (
+            y.transpose(1, 2)
+            .contiguous()
+            .view(bsz, seqlen, self.head_dim * self.n_head)
+        )
+        y = self.wo(y)
+        return y
+
+
+class FeedForward(nn.Module):
+    def __init__(self, config: ModelArgs) -> None:
+        super().__init__()
+        self.w1 = nn.Linear(config.dim, config.intermediate_size, bias=False)
+        self.w3 = nn.Linear(config.dim, config.intermediate_size, bias=False)
+        self.w2 = nn.Linear(config.intermediate_size, config.dim, bias=False)
+        self.dropout = nn.Dropout(config.dropout_rate)
+
+    def forward(self, x: Tensor) -> Tensor:
+        return self.w2(self.dropout(F.silu(self.w1(x)) * self.w3(x)))
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-5):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def _norm(self, x):
+        return x * torch.rsqrt(torch.mean(x * x, dim=-1, keepdim=True) + self.eps)
+
+    def forward(self, x: Tensor) -> Tensor:
+        output = self._norm(x.float()).type_as(x)
+        return output * self.weight
+
+
+class LayerScale(nn.Module):
+    def __init__(
+        self,
+        dim: int,
+        init_values: Union[float, Tensor] = 1e-2,
+        inplace: bool = False,
+    ) -> None:
+        super().__init__()
+        self.inplace = inplace
+        self.gamma = nn.Parameter(init_values * torch.ones(dim))
+
+    def forward(self, x: Tensor) -> Tensor:
+        return x.mul_(self.gamma) if self.inplace else x * self.gamma
+
+
+class WindowLimitedTransformer(Transformer):
+    """
+    Transformer with window limited attention, causal.
+    """
+
+    def __init__(
+        self,
+        config: ModelArgs,
+        input_dim: int = 512,
+        window_size: Optional[int] = None,
+        causal: bool = True,
+        look_ahead_conv: nn.Module = None,
+    ):
+        super().__init__(config)
+        self.window_size = window_size
+        self.causal = causal
+        self.channels_first = config.channels_first
+        self.look_ahead_conv = (
+            look_ahead_conv if look_ahead_conv is not None else nn.Identity()
+        )
+        self.input_proj = (
+            nn.Linear(input_dim, config.dim)
+            if input_dim != config.dim
+            else nn.Identity()
+        )
+        self.output_proj = (
+            nn.Linear(config.dim, input_dim)
+            if input_dim != config.dim
+            else nn.Identity()
+        )
+
+    def make_window_limited_mask(
+        self,
+        max_length: int,
+        x_lens: Optional[Tensor] = None,
+    ) -> Tensor:
+        """
+        Make mask to form window limited attention.
+        """
+        if self.causal:
+            mask = torch.tril(torch.ones(max_length, max_length))
+            row_indices = torch.arange(max_length).view(-1, 1)
+            window_size = self.window_size or max_length
+            valid_range = (row_indices - window_size + 1).clamp(min=0)
+            column_indices = torch.arange(max_length)
+            mask = (column_indices >= valid_range) & mask.bool()
+        else:
+            raise NotImplementedError
+        mask = mask.bool()[None, None]
+        return mask
+
+    def make_mask(
+        self,
+        max_length: int,
+        x_lens: Optional[Tensor] = None,
+    ) -> Tensor:
+        """
+        Make ordinary mask if window size is not specified.
+        """
+        if self.causal:
+            mask = torch.tril(torch.ones(max_length, max_length))
+        else:
+            mask = torch.ones(max_length, max_length)
+            mask = mask.bool()[None, None]
+            for i, x_len in enumerate(x_lens):
+                mask[:x_len, i] = 0
+        mask = mask.bool()[None, None]
+        return mask
+
+    def forward(
+        self,
+        x: Tensor,
+        x_lens: Optional[Tensor] = None,
+    ) -> Tensor:
+        if self.channels_first:
+            x = x.transpose(1, 2)
+        x = self.input_proj(x)  # (B, T, D)
+        x = self.look_ahead_conv(x)
+        input_pos = torch.arange(x.shape[1], device=x.device)
+        # construct mask to form window limited attention
+        max_length = x.shape[1]
+        if self.window_size is not None:
+            mask = self.make_window_limited_mask(max_length, x_lens)
+        else:
+            mask = self.make_mask(max_length, x_lens)
+        mask = mask.to(x.device)
+        x = super().forward(x, input_pos, mask)
+        x = self.output_proj(x)  # (B, T, D)
+        if self.channels_first:
+            x = x.transpose(1, 2)
+        return x
+
+
+def precompute_freqs_cis(
+    seq_len: int, n_elem: int, base: int = 10000, dtype: torch.dtype = torch.bfloat16
+) -> Tensor:
+    freqs = 1.0 / (
+        base ** (torch.arange(0, n_elem, 2)[: (n_elem // 2)].float() / n_elem)
+    )
+    t = torch.arange(seq_len, device=freqs.device)
+    freqs = torch.outer(t, freqs)
+    freqs_cis = torch.polar(torch.ones_like(freqs), freqs)
+    cache = torch.stack([freqs_cis.real, freqs_cis.imag], dim=-1)
+    return cache.to(dtype=dtype)
+
+
+def apply_rotary_emb(x: Tensor, freqs_cis: Tensor) -> Tensor:
+    xshaped = x.float().reshape(*x.shape[:-1], -1, 2)
+    freqs_cis = freqs_cis.view(1, xshaped.size(1), 1, xshaped.size(3), 2)
+    x_out2 = torch.stack(
+        [
+            xshaped[..., 0] * freqs_cis[..., 0] - xshaped[..., 1] * freqs_cis[..., 1],
+            xshaped[..., 1] * freqs_cis[..., 0] + xshaped[..., 0] * freqs_cis[..., 1],
+        ],
+        -1,
+    )
+
+    x_out2 = x_out2.flatten(3)
+    return x_out2.type_as(x)
+
+
+def init_weights(m):
+    if isinstance(m, nn.Conv1d):
+        nn.init.trunc_normal_(m.weight, std=0.02)
+        nn.init.constant_(m.bias, 0)
+
+
+def unpad1d(x: torch.Tensor, paddings: tp.Tuple[int, int]):
+    """Remove padding from x, handling properly zero padding. Only for 1d!"""
+    padding_left, padding_right = paddings
+    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
+    assert (padding_left + padding_right) <= x.shape[-1]
+    end = x.shape[-1] - padding_right
+    return x[..., padding_left:end]
+
+
+def get_extra_padding_for_conv1d(
+    x: torch.Tensor, kernel_size: int, stride: int, padding_total: int = 0
+) -> int:
+    """See `pad_for_conv1d`."""
+    length = x.shape[-1]
+    n_frames = (length - kernel_size + padding_total) / stride + 1
+    ideal_length = (math.ceil(n_frames) - 1) * stride + (kernel_size - padding_total)
+    return ideal_length - length
+
+
+def pad1d(
+    x: torch.Tensor,
+    paddings: tp.Tuple[int, int],
+    mode: str = "zeros",
+    value: float = 0.0,
+):
+    """Tiny wrapper around F.pad, just to allow for reflect padding on small input.
+    If this is the case, we insert extra 0 padding to the right
+    before the reflection happen.
+    """
+    length = x.shape[-1]
+    padding_left, padding_right = paddings
+    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
+    if mode == "reflect":
+        max_pad = max(padding_left, padding_right)
+        extra_pad = 0
+        if length <= max_pad:
+            extra_pad = max_pad - length + 1
+            x = F.pad(x, (0, extra_pad))
+        padded = F.pad(x, paddings, mode, value)
+        end = padded.shape[-1] - extra_pad
+        return padded[..., :end]
+    else:
+        return F.pad(x, paddings, mode, value)
+
+
+class CausalConvNet(nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        out_channels,
+        kernel_size,
+        dilation=1,
+        stride=1,
+        groups=1,
+        padding=None,
+    ):
+        super(CausalConvNet, self).__init__()
+        self.conv = nn.Conv1d(
+            in_channels,
+            out_channels,
+            kernel_size,
+            stride=stride,
+            dilation=dilation,
+            groups=groups,
+        )
+        self.stride = stride
+        self.kernel_size = (kernel_size - 1) * dilation + 1
+        self.dilation = dilation
+        self.padding = self.kernel_size - self.stride
+
+    def forward(self, x):
+        pad = self.padding
+        extra_padding = get_extra_padding_for_conv1d(
+            x, self.kernel_size, self.stride, pad
+        )
+        x = pad1d(x, (pad, extra_padding), mode="constant", value=0)
+        return self.conv(x).contiguous()
+
+    def weight_norm(self, name="weight", dim=0):
+        self.conv = weight_norm(self.conv, name=name, dim=dim)
+        return self
+
+    def remove_weight_norm(self):
+        self.conv = remove_parametrizations(self.conv)
+        return self
+
+
+class CausalTransConvNet(nn.Module):
+    def __init__(
+        self, in_channels, out_channels, kernel_size, dilation=1, stride=1, padding=None
+    ):
+        super(CausalTransConvNet, self).__init__()
+        self.conv = nn.ConvTranspose1d(
+            in_channels, out_channels, kernel_size, stride=stride, dilation=dilation
+        )
+        self.stride = stride
+        self.kernel_size = kernel_size
+
+    def forward(self, x):
+        x = self.conv(x)
+        pad = self.kernel_size - self.stride
+        padding_right = math.ceil(pad)
+        padding_left = pad - padding_right
+        x = unpad1d(x, (padding_left, padding_right))
+        return x.contiguous()
+
+    def weight_norm(self, name="weight", dim=0):
+        self.conv = weight_norm(self.conv, name=name, dim=dim)
+        return self
+
+    def remove_weight_norm(self):
+        self.conv = remove_parametrizations(self.conv)
+        return self
+
+
+def CausalWNConv1d(*args, **kwargs):
+    return CausalConvNet(*args, **kwargs).weight_norm()
+
+
+def CausalWNConvTranspose1d(*args, **kwargs):
+    return CausalTransConvNet(*args, **kwargs).weight_norm()
+
+
+class ResidualUnit(nn.Module):
+    def __init__(self, dim: int = 16, dilation: int = 1, causal: bool = False):
+        super().__init__()
+        conv_class = CausalWNConv1d if causal else WNConv1d
+        pad = ((7 - 1) * dilation) // 2
+        self.block = nn.Sequential(
+            Snake1d(dim),
+            conv_class(dim, dim, kernel_size=7, dilation=dilation, padding=pad),
+            Snake1d(dim),
+            conv_class(dim, dim, kernel_size=1),
+        )
+        self.causal = causal
+
+    def forward(self, x):
+        y = self.block(x)
+        pad = x.shape[-1] - y.shape[-1]
+        if pad > 0:
+            if self.causal:
+                x = x[..., :-pad]
+            else:
+                x = x[..., pad // 2 : -pad // 2]
+        return x + y
+
+
+class EncoderBlock(nn.Module):
+    def __init__(
+        self,
+        dim: int = 16,
+        stride: int = 1,
+        causal: bool = False,
+        n_t_layer: int = 0,
+        transformer_general_config=None,
+    ):
+        super().__init__()
+        conv_class = CausalWNConv1d if causal else WNConv1d
+        transformer_module = (
+            nn.Identity()
+            if n_t_layer == 0
+            else (
+                WindowLimitedTransformer(
+                    causal=causal,
+                    input_dim=dim,
+                    window_size=512,
+                    config=transformer_general_config(
+                        n_layer=n_t_layer,
+                        n_head=dim // 64,
+                        dim=dim,
+                        intermediate_size=dim * 3,
+                    ),
+                )
+            )
+        )
+        self.block = nn.Sequential(
+            ResidualUnit(dim // 2, dilation=1, causal=causal),
+            ResidualUnit(dim // 2, dilation=3, causal=causal),
+            ResidualUnit(dim // 2, dilation=9, causal=causal),
+            Snake1d(dim // 2),
+            conv_class(
+                dim // 2,
+                dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+            ),
+            transformer_module,
+        )
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class Encoder(nn.Module):
+    def __init__(
+        self,
+        d_model: int = 64,
+        strides: list = [2, 4, 8, 8],
+        d_latent: int = 64,
+        n_transformer_layers: list = [0, 0, 4, 4],
+        transformer_general_config: ModelArgs = None,
+        causal: bool = False,
+    ):
+        super().__init__()
+        conv_class = CausalWNConv1d if causal else WNConv1d
+        # Create first convolution
+        self.block = [conv_class(1, d_model, kernel_size=7, padding=3)]
+
+        # Create EncoderBlocks that double channels as they downsample by `stride`
+        for stride, n_t_layer in zip(strides, n_transformer_layers):
+            d_model *= 2
+            self.block += [
+                EncoderBlock(
+                    d_model,
+                    stride=stride,
+                    causal=causal,
+                    n_t_layer=n_t_layer,
+                    transformer_general_config=transformer_general_config,
+                )
+            ]
+
+        # Create last convolution
+        self.block += [
+            Snake1d(d_model),
+            conv_class(d_model, d_latent, kernel_size=3, padding=1),
+        ]
+
+        # Wrap black into nn.Sequential
+        self.block = nn.Sequential(*self.block)
+        self.enc_dim = d_model
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class DecoderBlock(nn.Module):
+    def __init__(
+        self,
+        input_dim: int = 16,
+        output_dim: int = 8,
+        stride: int = 1,
+        causal: bool = False,
+        n_t_layer: int = 0,
+        transformer_general_config=None,
+    ):
+        super().__init__()
+        conv_trans_class = CausalWNConvTranspose1d if causal else WNConvTranspose1d
+        transformer_module = (
+            nn.Identity()
+            if n_t_layer == 0
+            else (
+                WindowLimitedTransformer(
+                    causal=causal,
+                    input_dim=input_dim,
+                    window_size=None,
+                    config=transformer_general_config(
+                        n_layer=n_t_layer,
+                        n_head=input_dim // 64,
+                        dim=input_dim,
+                        intermediate_size=input_dim * 3,
+                    ),
+                )
+            )
+        )
+        self.block = nn.Sequential(
+            # transformer_module,
+            Snake1d(input_dim),
+            conv_trans_class(
+                input_dim,
+                output_dim,
+                kernel_size=2 * stride,
+                stride=stride,
+                padding=math.ceil(stride / 2),
+            ),
+            ResidualUnit(output_dim, dilation=1, causal=causal),
+            ResidualUnit(output_dim, dilation=3, causal=causal),
+            ResidualUnit(output_dim, dilation=9, causal=causal),
+        )
+
+    def forward(self, x):
+        return self.block(x)
+
+
+class Decoder(nn.Module):
+    def __init__(
+        self,
+        input_channel,
+        channels,
+        rates,
+        d_out: int = 1,
+        causal: bool = False,
+        n_transformer_layers: list = [0, 0, 0, 0],
+        transformer_general_config=None,
+    ):
+        super().__init__()
+        conv_class = CausalWNConv1d if causal else WNConv1d
+        # Add first conv layer
+        layers = [conv_class(input_channel, channels, kernel_size=7, padding=3)]
+
+        # Add upsampling + MRF blocks
+        for i, (stride, n_t_layer) in enumerate(zip(rates, n_transformer_layers)):
+            input_dim = channels // 2**i
+            output_dim = channels // 2 ** (i + 1)
+            layers += [
+                DecoderBlock(
+                    input_dim,
+                    output_dim,
+                    stride,
+                    causal=causal,
+                    n_t_layer=n_t_layer,
+                    transformer_general_config=transformer_general_config,
+                )
+            ]
+
+        # Add final conv layer
+        layers += [
+            Snake1d(output_dim),
+            conv_class(output_dim, d_out, kernel_size=7, padding=3),
+            nn.Tanh(),
+        ]
+
+        self.model = nn.Sequential(*layers)
+
+    def forward(self, x):
+        return self.model(x)
+
+
+class DAC(BaseModel, CodecMixin):
+    def __init__(
+        self,
+        encoder_dim: int = 64,
+        encoder_rates: List[int] = [2, 4, 8, 8],
+        latent_dim: int = None,
+        decoder_dim: int = 1536,
+        decoder_rates: List[int] = [8, 8, 4, 2],
+        quantizer: torch.nn.Module = None,
+        sample_rate: int = 44100,
+        causal: bool = True,
+        encoder_transformer_layers: List[int] = [0, 0, 0, 0],
+        decoder_transformer_layers: List[int] = [0, 0, 0, 0],
+        transformer_general_config=None,
+    ):
+        super().__init__()
+
+        self.encoder_dim = encoder_dim
+        self.encoder_rates = encoder_rates
+        self.decoder_dim = decoder_dim
+        self.decoder_rates = decoder_rates
+        self.sample_rate = sample_rate
+
+        if latent_dim is None:
+            latent_dim = encoder_dim * (2 ** len(encoder_rates))
+
+        self.latent_dim = latent_dim
+
+        self.hop_length = np.prod(encoder_rates)
+        self.encoder = Encoder(
+            encoder_dim,
+            encoder_rates,
+            latent_dim,
+            causal=causal,
+            n_transformer_layers=encoder_transformer_layers,
+            transformer_general_config=transformer_general_config,
+        )
+
+        self.quantizer = quantizer
+
+        self.decoder = Decoder(
+            latent_dim,
+            decoder_dim,
+            decoder_rates,
+            causal=causal,
+            n_transformer_layers=decoder_transformer_layers,
+            transformer_general_config=transformer_general_config,
+        )
+        self.sample_rate = sample_rate
+        self.apply(init_weights)
+
+        self.delay = self.get_delay()
+
+        self.frame_length = self.hop_length * 4
+
+    def preprocess(self, audio_data, sample_rate):
+        if sample_rate is None:
+            sample_rate = self.sample_rate
+        assert sample_rate == self.sample_rate
+
+        length = audio_data.shape[-1]
+        right_pad = math.ceil(length / self.hop_length) * self.hop_length - length
+        audio_data = nn.functional.pad(audio_data, (0, right_pad))
+
+        return audio_data
+
+    def encode(
+        self,
+        audio_data: torch.Tensor,
+        audio_lengths: torch.Tensor = None,
+        n_quantizers: int = None,
+        **kwargs,
+    ):
+        """Encode given audio data and return quantized latent codes
+
+        Parameters
+        ----------
+        audio_data : Tensor[B x T]
+            Audio data to encode
+        n_quantizers : int, optional
+            Number of quantizers to use, by default None
+            If None, all quantizers are used.
+
+        Returns
+        -------
+        dict
+            A dictionary with the following keys:
+            "z" : Tensor[B x D x T]
+                Quantized continuous representation of input
+            "codes" : Tensor[B x N x T]
+                Codebook indices for each codebook
+                (quantized discrete representation of input)
+            "latents" : Tensor[B x N*D x T]
+                Projected latents (continuous representation of input before quantization)
+            "vq/commitment_loss" : Tensor[1]
+                Commitment loss to train encoder to predict vectors closer to codebook
+                entries
+            "vq/codebook_loss" : Tensor[1]
+                Codebook loss to update the codebook
+            "length" : int
+                Number of samples in input audio
+        """
+        # pad to multiple of self.frame_length
+        if audio_data.ndim == 2:
+            audio_data = audio_data.unsqueeze(1)
+        # print(audio_data.shape)
+        length = audio_data.shape[-1]
+        right_pad = math.ceil(length / self.frame_length) * self.frame_length - length
+        audio_data = nn.functional.pad(audio_data, (0, right_pad))
+        if audio_lengths is None:
+            audio_lengths = torch.LongTensor([length + right_pad]).to(audio_data.device)
+
+        z = self.encoder(audio_data)
+        vq_results = self.quantizer(z, n_quantizers, **kwargs)
+        indices = vq_results.codes
+        indices_lens = torch.ceil(audio_lengths / self.frame_length).long()
+        return indices, indices_lens
+
+    def decode(self, indices: torch.Tensor, feature_lengths):
+        z = self.quantizer.decode(indices)
+        audio_lengths = feature_lengths * self.frame_length
+        return self.decoder(z), audio_lengths
+
+    def forward(
+        self,
+        audio_data: torch.Tensor,
+        template: torch.Tensor = None,
+        mask: torch.Tensor = None,
+        sample_rate: int = None,
+        n_quantizers: int = None,
+        **kwargs,
+    ):
+        """Model forward pass
+
+        Parameters
+        ----------
+        audio_data : Tensor[B x 1 x T]
+            Audio data to encode
+        sample_rate : int, optional
+            Sample rate of audio data in Hz, by default None
+            If None, defaults to `self.sample_rate`
+        n_quantizers : int, optional
+            Number of quantizers to use, by default None.
+            If None, all quantizers are used.
+
+        Returns
+        -------
+        dict
+            A dictionary with the following keys:
+            "z" : Tensor[B x D x T]
+                Quantized continuous representation of input
+            "codes" : Tensor[B x N x T]
+                Codebook indices for each codebook
+                (quantized discrete representation of input)
+            "latents" : Tensor[B x N*D x T]
+                Projected latents (continuous representation of input before quantization)
+            "vq/commitment_loss" : Tensor[1]
+                Commitment loss to train encoder to predict vectors closer to codebook
+                entries
+            "vq/codebook_loss" : Tensor[1]
+                Codebook loss to update the codebook
+            "length" : int
+                Number of samples in input audio
+            "audio" : Tensor[B x 1 x length]
+                Decoded audio data.
+        """
+        length = audio_data.shape[-1]
+        audio_data = self.preprocess(audio_data, sample_rate)
+        vq_results = self.encode(audio_data, n_quantizers, **kwargs)
+        z = vq_results[0] if isinstance(vq_results, tuple) else vq_results.z
+        x = self.decode(z)
+        return x[..., :length], vq_results
+
+
+if __name__ == "__main__":
+
+    def filter_state_dict_shapes(params, model):
+        model_state_dict = model.state_dict()
+        filtered_state_dict = {
+            k: v
+            for k, v in params.items()
+            if k in model_state_dict and v.shape == model_state_dict[k].shape
+        }
+        skipped_keys = set(params.keys()) - set(filtered_state_dict.keys())
+        if skipped_keys:
+            print(
+                f"Warning: Skipped loading some keys due to shape mismatch: {skipped_keys}"
+            )
+        return filtered_state_dict, skipped_keys
+
+    model = hydra.utils.instantiate(
+        OmegaConf.load("fish_speech/configs/modded_dac_vq.yaml")
+    )
+    sd = torch.load("checkpoints/openaudio-s1-mini/firefly-gan-large.pth")
+    filtered_sd, skipped_keys = filter_state_dict_shapes(sd, model)
+    print(f"Skipped keys: {skipped_keys}")
+    model.load_state_dict(filtered_sd, strict=False)
+    model.eval()
+
+    src_audio_path = "./test.wav"
+    wave_np, _ = librosa.load(src_audio_path, sr=44100, mono=False)
+    if len(wave_np.shape) == 1:
+        wave_np = wave_np[None, :]
+    wave_tensor = torch.from_numpy(wave_np).unsqueeze(1)
+
+    with torch.no_grad():
+        # encode 返回 (indices, indices_lens)
+        indices, indices_lens = model.encode(wave_tensor)
+        print(f"Indices shape: {indices.shape}")
+        print(f"Indices lengths: {indices_lens}")
+
+        # decode 需要 indices 和 feature_lengths 两个参数
+        fake_audio, audio_lengths = model.decode(indices, indices_lens)
+        print(f"Decoded audio shape: {fake_audio.shape}")
+        print(f"Audio lengths: {audio_lengths}")
+
+    # 保存重建的音频
+    sf.write("fake.wav", fake_audio.squeeze(1).cpu().numpy().T, 44100)

fish_speech/models/dac/rvq.py

@@ -0,0 +1,403 @@
+import math
+import typing as tp
+from dataclasses import dataclass
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from dac.nn.quantize import ResidualVectorQuantize
+from torch.nn.utils.parametrizations import weight_norm
+from torch.nn.utils.parametrize import remove_parametrizations
+
+
+def unpad1d(x: torch.Tensor, paddings: tp.Tuple[int, int]):
+    """Remove padding from x, handling properly zero padding. Only for 1d!"""
+    padding_left, padding_right = paddings
+    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
+    assert (padding_left + padding_right) <= x.shape[-1]
+    end = x.shape[-1] - padding_right
+    return x[..., padding_left:end]
+
+
+def get_extra_padding_for_conv1d(
+    x: torch.Tensor, kernel_size: int, stride: int, padding_total: int = 0
+) -> int:
+    """See `pad_for_conv1d`."""
+    length = x.shape[-1]
+    n_frames = (length - kernel_size + padding_total) / stride + 1
+    ideal_length = (math.ceil(n_frames) - 1) * stride + (kernel_size - padding_total)
+    return ideal_length - length
+
+
+def pad1d(
+    x: torch.Tensor,
+    paddings: tp.Tuple[int, int],
+    mode: str = "zeros",
+    value: float = 0.0,
+):
+    """Tiny wrapper around F.pad, just to allow for reflect padding on small input.
+    If this is the case, we insert extra 0 padding to the right
+    before the reflection happen.
+    """
+    length = x.shape[-1]
+    padding_left, padding_right = paddings
+    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
+    if mode == "reflect":
+        max_pad = max(padding_left, padding_right)
+        extra_pad = 0
+        if length <= max_pad:
+            extra_pad = max_pad - length + 1
+            x = F.pad(x, (0, extra_pad))
+        padded = F.pad(x, paddings, mode, value)
+        end = padded.shape[-1] - extra_pad
+        return padded[..., :end]
+    else:
+        return F.pad(x, paddings, mode, value)
+
+
+class CausalConvNet(nn.Module):
+    def __init__(
+        self,
+        in_channels,
+        out_channels,
+        kernel_size,
+        dilation=1,
+        stride=1,
+        groups=1,
+        padding=None,
+    ):
+        super(CausalConvNet, self).__init__()
+        self.conv = nn.Conv1d(
+            in_channels,
+            out_channels,
+            kernel_size,
+            stride=stride,
+            dilation=dilation,
+            groups=groups,
+        )
+        self.stride = stride
+        self.kernel_size = (kernel_size - 1) * dilation + 1
+        self.dilation = dilation
+        self.padding = self.kernel_size - self.stride
+
+    def forward(self, x):
+        pad = self.padding
+        extra_padding = get_extra_padding_for_conv1d(
+            x, self.kernel_size, self.stride, pad
+        )
+        x = pad1d(x, (pad, extra_padding), mode="constant", value=0)
+        return self.conv(x).contiguous()
+
+    def weight_norm(self, name="weight", dim=0):
+        self.conv = weight_norm(self.conv, name=name, dim=dim)
+        return self
+
+    def remove_weight_norm(self):
+        self.conv = remove_parametrizations(self.conv)
+        return self
+
+
+class CausalTransConvNet(nn.Module):
+    def __init__(
+        self, in_channels, out_channels, kernel_size, dilation=1, stride=1, padding=None
+    ):
+        super(CausalTransConvNet, self).__init__()
+        self.conv = nn.ConvTranspose1d(
+            in_channels, out_channels, kernel_size, stride=stride, dilation=dilation
+        )
+        self.stride = stride
+        self.kernel_size = kernel_size
+
+    def forward(self, x):
+        x = self.conv(x)
+        pad = self.kernel_size - self.stride
+        padding_right = math.ceil(pad)
+        padding_left = pad - padding_right
+        x = unpad1d(x, (padding_left, padding_right))
+        return x.contiguous()
+
+    def weight_norm(self, name="weight", dim=0):
+        self.conv = weight_norm(self.conv, name=name, dim=dim)
+        return self
+
+    def remove_weight_norm(self):
+        self.conv = remove_parametrizations(self.conv)
+        return self
+
+
+# ConvNeXt Block copied from https://github.com/fishaudio/fish-diffusion/blob/main/fish_diffusion/modules/convnext.py
+class ConvNeXtBlock(nn.Module):
+    r"""ConvNeXt Block. There are two equivalent implementations:
+    (1) DwConv -> LayerNorm (channels_first) -> 1x1 Conv -> GELU -> 1x1 Conv; all in (N, C, H, W)
+    (2) DwConv -> Permute to (N, H, W, C); LayerNorm (channels_last) -> Linear -> GELU -> Linear; Permute back
+    We use (2) as we find it slightly faster in PyTorch
+    Args:
+        dim (int): Number of input channels.
+        drop_path (float): Stochastic depth rate. Default: 0.0
+        layer_scale_init_value (float): Init value for Layer Scale. Default: 1e-6.
+        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0.
+        kernel_size (int): Kernel size for depthwise conv. Default: 7.
+        dilation (int): Dilation for depthwise conv. Default: 1.
+    """  # noqa: E501
+
+    def __init__(
+        self,
+        dim: int,
+        layer_scale_init_value: float = 1e-6,
+        mlp_ratio: float = 4.0,
+        kernel_size: int = 7,
+        dilation: int = 1,
+    ):
+        super().__init__()
+        convnet_type = CausalConvNet
+        self.dwconv = convnet_type(
+            dim,
+            dim,
+            kernel_size=kernel_size,
+            # padding=int(dilation * (kernel_size - 1) / 2),
+            groups=dim,
+            dilation=dilation,
+        )  # depthwise conv
+        self.norm = nn.LayerNorm(dim, eps=1e-6)
+        self.pwconv1 = nn.Linear(
+            dim, int(mlp_ratio * dim)
+        )  # pointwise/1x1 convs, implemented with linear layers
+        self.act = nn.GELU()
+        self.pwconv2 = nn.Linear(int(mlp_ratio * dim), dim)
+        self.gamma = (
+            nn.Parameter(layer_scale_init_value * torch.ones((dim)), requires_grad=True)
+            if layer_scale_init_value > 0
+            else None
+        )
+
+    def forward(self, x, apply_residual: bool = True):
+        input = x
+
+        x = self.dwconv(x)
+        x = x.permute(0, 2, 1)  # (N, C, L) -> (N, L, C)
+        x = self.norm(x)
+        x = self.pwconv1(x)
+        x = self.act(x)
+        x = self.pwconv2(x)
+
+        if self.gamma is not None:
+            x = self.gamma * x
+
+        x = x.permute(0, 2, 1)  # (N, L, C) -> (N, C, L)
+
+        if apply_residual:
+            x = input + x
+
+        return x
+
+
+@dataclass
+class VQResult:
+    z: torch.Tensor
+    codes: torch.Tensor
+    latents: torch.Tensor
+    codebook_loss: torch.Tensor
+    commitment_loss: torch.Tensor
+    semantic_distill_z: torch.Tensor | None = None
+
+
+class DownsampleResidualVectorQuantize(nn.Module):
+    def __init__(
+        self,
+        input_dim: int = 1024,
+        n_codebooks: int = 9,
+        codebook_dim: int = 8,
+        quantizer_dropout: float = 0.5,
+        codebook_size: int = 1024,
+        semantic_codebook_size: int = 4096,
+        downsample_factor: tuple[int] = (2, 2),
+        downsample_dims: tuple[int] | None = None,
+        pre_module: nn.Module | None = None,
+        post_module: nn.Module | None = None,
+        semantic_predictor_module: nn.Module | None = None,
+    ):
+        super().__init__()
+
+        if downsample_dims is None:
+            downsample_dims = [input_dim for _ in range(len(downsample_factor))]
+
+        all_dims = (input_dim,) + tuple(downsample_dims)
+
+        self.semantic_quantizer = ResidualVectorQuantize(
+            input_dim=input_dim,
+            n_codebooks=1,
+            codebook_size=semantic_codebook_size,
+            codebook_dim=codebook_dim,
+            quantizer_dropout=0.0,
+        )
+
+        self.quantizer = ResidualVectorQuantize(
+            input_dim=input_dim,
+            n_codebooks=n_codebooks,
+            codebook_size=codebook_size,
+            codebook_dim=codebook_dim,
+            quantizer_dropout=quantizer_dropout,
+        )
+
+        self.downsample_factor = downsample_factor
+        self.downsample_dims = downsample_dims
+
+        convnet_type = CausalConvNet
+        transconvnet_type = CausalTransConvNet
+
+        self.downsample = nn.Sequential(
+            *[
+                nn.Sequential(
+                    convnet_type(
+                        all_dims[idx],
+                        all_dims[idx + 1],
+                        kernel_size=factor,
+                        stride=factor,
+                    ),
+                    ConvNeXtBlock(dim=all_dims[idx + 1]),
+                )
+                for idx, factor in enumerate(downsample_factor)
+            ]
+        )
+
+        self.upsample = nn.Sequential(
+            *[
+                nn.Sequential(
+                    transconvnet_type(
+                        all_dims[idx + 1],
+                        all_dims[idx],
+                        kernel_size=factor,
+                        stride=factor,
+                    ),
+                    ConvNeXtBlock(dim=all_dims[idx]),
+                )
+                for idx, factor in reversed(list(enumerate(downsample_factor)))
+            ]
+        )
+        self.apply(self._init_weights)
+        self.pre_module = (
+            pre_module if pre_module is not None else nn.Identity()
+        )  # leave for transformer, LSTM or Mamba or something else
+        self.post_module = post_module if post_module is not None else nn.Identity()
+        self.semantic_predictor_module = (
+            semantic_predictor_module
+            if semantic_predictor_module is not None
+            else nn.Identity()
+        )
+
+    def _init_weights(self, m):
+        if isinstance(m, (nn.Conv1d, nn.Linear)):
+            nn.init.trunc_normal_(m.weight, std=0.02)
+            nn.init.constant_(m.bias, 0)
+
+    def forward(
+        self, z, n_quantizers: int = None, semantic_len: torch.Tensor = None, **kwargs
+    ):
+        # z: (B, D, T)
+        original_shape = z.shape
+        if semantic_len is None:
+            semantic_len = torch.LongTensor([z.shape[-1]])
+        z = self.downsample(z)
+        z = self.pre_module(z)  # B, T, D
+        (
+            semantic_z,
+            semantic_codes,
+            semantic_latents,
+            semantic_commitment_loss,
+            semantic_codebook_loss,
+        ) = self.semantic_quantizer(z)
+        residual_z = z - semantic_z
+        residual_z, codes, latents, commitment_loss, codebook_loss = self.quantizer(
+            residual_z, n_quantizers=n_quantizers
+        )
+        z = semantic_z + residual_z
+        commitment_loss = commitment_loss + semantic_commitment_loss
+        codebook_loss = codebook_loss + semantic_codebook_loss
+        codes = torch.cat([semantic_codes, codes], dim=1)
+        latents = torch.cat([semantic_latents, latents], dim=1)
+        z = self.post_module(z)
+        z = self.upsample(z)
+        # z: (B, D, T)
+
+        # semantic distillation (disabled here since only used in training)
+        # semantic_distill_z = self.semantic_predictor_module(semantic_z, semantic_len).mT  # wav2vec target is B, T, D
+
+        # Pad or crop z to match original shape
+        diff = original_shape[-1] - z.shape[-1]
+        right = 0
+        left = abs(diff) - right
+
+        if diff > 0:
+            z = F.pad(z, (left, right))
+        elif diff < 0:
+            z = z[..., left:]
+
+        results = VQResult(
+            z=z,
+            codes=codes,
+            latents=latents,
+            commitment_loss=commitment_loss,
+            codebook_loss=codebook_loss,
+        )
+
+        return results
+
+    # def encode(self, z):
+    #     z = self.downsample(z)
+    #     z = self.pre_module(z)
+    #     _, indices, _, _, _ = self.quantizer(z.mT)
+    #     indices = rearrange(indices, "g b l r -> b (g r) l")
+    #     return indices
+    #
+    def decode(self, indices: torch.Tensor):
+        # indices = rearrange(indices, "b (g r) l -> g b l r", g=self.residual_fsq.groups)
+
+        # print(f"indices: {indices.shape}, semantic_quantizer.codebook_size: {self.semantic_quantizer.codebook_size}, quantizer.codebook_size: {self.quantizer.codebook_size}, semantic min: {indices[:, 0].min()}, max: {indices[:, 0].max()}, quantizer min: {indices[:, 1:].min()}, max: {indices[:, 1:].max()}")
+
+        new_indices = torch.zeros_like(indices)
+        new_indices[:, 0] = torch.clamp(
+            indices[:, 0], max=self.semantic_quantizer.codebook_size - 1
+        )
+        new_indices[:, 1:] = torch.clamp(
+            indices[:, 1:], max=self.quantizer.codebook_size - 1
+        )
+
+        z_q_semantic = self.semantic_quantizer.from_codes(new_indices[:, :1])[0]
+        z_q_residual = self.quantizer.from_codes(new_indices[:, 1:])[0]
+        z_q = z_q_semantic + z_q_residual
+        z_q = self.post_module(z_q)
+        z_q = self.upsample(z_q)
+        return z_q
+
+    # def from_latents(self, latents: torch.Tensor):
+    #     z_q, z_p, codes = super().from_latents(latents)
+    #     z_q = self.upsample(z_q)
+    #     return z_q, z_p, codes
+
+
+if __name__ == "__main__":
+    rvq = DownsampleResidualVectorQuantize(
+        input_dim=512,
+        n_codebooks=8,
+        codebook_dim=8,
+        codebook_size=1024,
+        quantizer_dropout=0.5,
+        downsample_factor=[2, 2],
+    )
+    rvq.eval()
+    x = torch.randn(2, 512, 442)
+
+    result = rvq(x)
+    print(rvq)
+    print(result.latents.shape, result.codes.shape, result.z.shape)
+
+    # y = rvq.from_codes(result.codes)
+    # print(y[0].shape)
+
+    # y = rvq.from_latents(
+
+    result1 = rvq(x[:, :, :40])
+    print(result1.latents.shape, result1.codes.shape, result1.z.shape)
+
+    assert torch.allclose(result.z[:, :, :40], result1.z, atol=1e-8)
+    print("Success")

fish_speech/models/vqgan/modules/firefly.py

@@ -1,604 +0,0 @@
-import math
-from functools import partial
-from math import prod
-from typing import Callable
-
-import torch
-import torch.nn.functional as F
-from torch import nn
-from torch.nn.utils.parametrizations import weight_norm
-from torch.nn.utils.parametrize import remove_parametrizations
-from torch.utils.checkpoint import checkpoint
-
-
-def sequence_mask(length, max_length=None):
-    if max_length is None:
-        max_length = length.max()
-    x = torch.arange(max_length, dtype=length.dtype, device=length.device)
-    return x.unsqueeze(0) < length.unsqueeze(1)
-
-
-def init_weights(m, mean=0.0, std=0.01):
-    classname = m.__class__.__name__
-    if classname.find("Conv1D") != -1:
-        m.weight.data.normal_(mean, std)
-
-
-def get_padding(kernel_size, dilation=1):
-    return (kernel_size * dilation - dilation) // 2
-
-
-def unpad1d(x: torch.Tensor, paddings: tuple[int, int]):
-    """Remove padding from x, handling properly zero padding. Only for 1d!"""
-    padding_left, padding_right = paddings
-    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
-    assert (padding_left + padding_right) <= x.shape[-1]
-    end = x.shape[-1] - padding_right
-    return x[..., padding_left:end]
-
-
-def get_extra_padding_for_conv1d(
-    x: torch.Tensor, kernel_size: int, stride: int, padding_total: int = 0
-) -> int:
-    """See `pad_for_conv1d`."""
-    length = x.shape[-1]
-    n_frames = (length - kernel_size + padding_total) / stride + 1
-    # for tracer, math.ceil will make onnx graph become constant
-    if isinstance(n_frames, torch.Tensor):
-        ideal_length = (torch.ceil(n_frames).long() - 1) * stride + (
-            kernel_size - padding_total
-        )
-    else:
-        ideal_length = (math.ceil(n_frames) - 1) * stride + (
-            kernel_size - padding_total
-        )
-    return ideal_length - length
-
-
-def pad1d(
-    x: torch.Tensor,
-    paddings: tuple[int, int],
-    mode: str = "zeros",
-    value: float = 0.0,
-):
-    """Tiny wrapper around F.pad, just to allow for reflect padding on small input.
-    If this is the case, we insert extra 0 padding to the right
-    before the reflection happen.
-    """
-    length = x.shape[-1]
-    padding_left, padding_right = paddings
-    assert padding_left >= 0 and padding_right >= 0, (padding_left, padding_right)
-    if mode == "reflect":
-        max_pad = max(padding_left, padding_right)
-        extra_pad = 0
-        if length <= max_pad:
-            extra_pad = max_pad - length + 1
-            x = F.pad(x, (0, extra_pad))
-        padded = F.pad(x, paddings, mode, value)
-        end = padded.shape[-1] - extra_pad
-        return padded[..., :end]
-    else:
-        return F.pad(x, paddings, mode, value)
-
-
-class FishConvNet(nn.Module):
-    def __init__(
-        self, in_channels, out_channels, kernel_size, dilation=1, stride=1, groups=1
-    ):
-        super(FishConvNet, self).__init__()
-        self.conv = nn.Conv1d(
-            in_channels,
-            out_channels,
-            kernel_size,
-            stride=stride,
-            dilation=dilation,
-            groups=groups,
-        )
-        self.stride = stride
-        self.kernel_size = (kernel_size - 1) * dilation + 1
-        self.dilation = dilation
-
-    def forward(self, x):
-        pad = self.kernel_size - self.stride
-        extra_padding = get_extra_padding_for_conv1d(
-            x, self.kernel_size, self.stride, pad
-        )
-        x = pad1d(x, (pad, extra_padding), mode="constant", value=0)
-        return self.conv(x).contiguous()
-
-    def weight_norm(self, name="weight", dim=0):
-        self.conv = weight_norm(self.conv, name=name, dim=dim)
-        return self
-
-    def remove_parametrizations(self, name="weight"):
-        self.conv = remove_parametrizations(self.conv, name)
-        return self
-
-
-class FishTransConvNet(nn.Module):
-    def __init__(self, in_channels, out_channels, kernel_size, dilation=1, stride=1):
-        super(FishTransConvNet, self).__init__()
-        self.conv = nn.ConvTranspose1d(
-            in_channels, out_channels, kernel_size, stride=stride, dilation=dilation
-        )
-        self.stride = stride
-        self.kernel_size = kernel_size
-
-    def forward(self, x):
-        x = self.conv(x)
-        pad = self.kernel_size - self.stride
-        padding_right = math.ceil(pad)
-        padding_left = pad - padding_right
-        x = unpad1d(x, (padding_left, padding_right))
-        return x.contiguous()
-
-    def weight_norm(self, name="weight", dim=0):
-        self.conv = weight_norm(self.conv, name=name, dim=dim)
-        return self
-
-    def remove_parametrizations(self, name="weight"):
-        self.conv = remove_parametrizations(self.conv, name)
-        return self
-
-
-class ResBlock1(torch.nn.Module):
-    def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5)):
-        super().__init__()
-
-        self.convs1 = nn.ModuleList(
-            [
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[0]
-                ).weight_norm(),
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[1]
-                ).weight_norm(),
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[2]
-                ).weight_norm(),
-            ]
-        )
-        self.convs1.apply(init_weights)
-
-        self.convs2 = nn.ModuleList(
-            [
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[0]
-                ).weight_norm(),
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[1]
-                ).weight_norm(),
-                FishConvNet(
-                    channels, channels, kernel_size, stride=1, dilation=dilation[2]
-                ).weight_norm(),
-            ]
-        )
-        self.convs2.apply(init_weights)
-
-    def forward(self, x):
-        for c1, c2 in zip(self.convs1, self.convs2):
-            xt = F.silu(x)
-            xt = c1(xt)
-            xt = F.silu(xt)
-            xt = c2(xt)
-            x = xt + x
-        return x
-
-    def remove_parametrizations(self):
-        for conv in self.convs1:
-            conv.remove_parametrizations()
-        for conv in self.convs2:
-            conv.remove_parametrizations()
-
-
-class ParallelBlock(nn.Module):
-    def __init__(
-        self,
-        channels: int,
-        kernel_sizes: tuple[int] = (3, 7, 11),
-        dilation_sizes: tuple[tuple[int]] = ((1, 3, 5), (1, 3, 5), (1, 3, 5)),
-    ):
-        super().__init__()
-
-        assert len(kernel_sizes) == len(dilation_sizes)
-
-        self.blocks = nn.ModuleList()
-        for k, d in zip(kernel_sizes, dilation_sizes):
-            self.blocks.append(ResBlock1(channels, k, d))
-
-    def forward(self, x):
-        return torch.stack([block(x) for block in self.blocks], dim=0).mean(dim=0)
-
-    def remove_parametrizations(self):
-        for block in self.blocks:
-            block.remove_parametrizations()
-
-
-class HiFiGANGenerator(nn.Module):
-    def __init__(
-        self,
-        *,
-        hop_length: int = 512,
-        upsample_rates: tuple[int] = (8, 8, 2, 2, 2),
-        upsample_kernel_sizes: tuple[int] = (16, 16, 8, 2, 2),
-        resblock_kernel_sizes: tuple[int] = (3, 7, 11),
-        resblock_dilation_sizes: tuple[tuple[int]] = ((1, 3, 5), (1, 3, 5), (1, 3, 5)),
-        num_mels: int = 128,
-        upsample_initial_channel: int = 512,
-        pre_conv_kernel_size: int = 7,
-        post_conv_kernel_size: int = 7,
-        post_activation: Callable = partial(nn.SiLU, inplace=True),
-    ):
-        super().__init__()
-
-        assert (
-            prod(upsample_rates) == hop_length
-        ), f"hop_length must be {prod(upsample_rates)}"
-
-        self.conv_pre = FishConvNet(
-            num_mels,
-            upsample_initial_channel,
-            pre_conv_kernel_size,
-            stride=1,
-        ).weight_norm()
-
-        self.num_upsamples = len(upsample_rates)
-        self.num_kernels = len(resblock_kernel_sizes)
-
-        self.noise_convs = nn.ModuleList()
-        self.ups = nn.ModuleList()
-
-        for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
-            self.ups.append(
-                FishTransConvNet(
-                    upsample_initial_channel // (2**i),
-                    upsample_initial_channel // (2 ** (i + 1)),
-                    k,
-                    stride=u,
-                ).weight_norm()
-            )
-
-        self.resblocks = nn.ModuleList()
-        for i in range(len(self.ups)):
-            ch = upsample_initial_channel // (2 ** (i + 1))
-            self.resblocks.append(
-                ParallelBlock(ch, resblock_kernel_sizes, resblock_dilation_sizes)
-            )
-
-        self.activation_post = post_activation()
-        self.conv_post = FishConvNet(
-            ch, 1, post_conv_kernel_size, stride=1
-        ).weight_norm()
-        self.ups.apply(init_weights)
-        self.conv_post.apply(init_weights)
-
-    def forward(self, x):
-        x = self.conv_pre(x)
-
-        for i in range(self.num_upsamples):
-            x = F.silu(x, inplace=True)
-            x = self.ups[i](x)
-
-            if self.training and self.checkpointing:
-                x = checkpoint(
-                    self.resblocks[i],
-                    x,
-                    use_reentrant=False,
-                )
-            else:
-                x = self.resblocks[i](x)
-
-        x = self.activation_post(x)
-        x = self.conv_post(x)
-        x = torch.tanh(x)
-
-        return x
-
-    def remove_parametrizations(self):
-        for up in self.ups:
-            up.remove_parametrizations()
-        for block in self.resblocks:
-            block.remove_parametrizations()
-        self.conv_pre.remove_parametrizations()
-        self.conv_post.remove_parametrizations()
-
-
-# DropPath copied from timm library
-def drop_path(
-    x, drop_prob: float = 0.0, training: bool = False, scale_by_keep: bool = True
-):
-    """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
-
-    This is the same as the DropConnect impl I created for EfficientNet, etc networks, however,
-    the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
-    See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for
-    changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use
-    'survival rate' as the argument.
-
-    """  # noqa: E501
-
-    if drop_prob == 0.0 or not training:
-        return x
-    keep_prob = 1 - drop_prob
-    shape = (x.shape[0],) + (1,) * (
-        x.ndim - 1
-    )  # work with diff dim tensors, not just 2D ConvNets
-    random_tensor = x.new_empty(shape).bernoulli_(keep_prob)
-    if keep_prob > 0.0 and scale_by_keep:
-        random_tensor.div_(keep_prob)
-    return x * random_tensor
-
-
-class DropPath(nn.Module):
-    """Drop paths (Stochastic Depth) per sample  (when applied in main path of residual blocks)."""  # noqa: E501
-
-    def __init__(self, drop_prob: float = 0.0, scale_by_keep: bool = True):
-        super(DropPath, self).__init__()
-        self.drop_prob = drop_prob
-        self.scale_by_keep = scale_by_keep
-
-    def forward(self, x):
-        return drop_path(x, self.drop_prob, self.training, self.scale_by_keep)
-
-    def extra_repr(self):
-        return f"drop_prob={round(self.drop_prob,3):0.3f}"
-
-
-class LayerNorm(nn.Module):
-    r"""LayerNorm that supports two data formats: channels_last (default) or channels_first.
-    The ordering of the dimensions in the inputs. channels_last corresponds to inputs with
-    shape (batch_size, height, width, channels) while channels_first corresponds to inputs
-    with shape (batch_size, channels, height, width).
-    """  # noqa: E501
-
-    def __init__(self, normalized_shape, eps=1e-6, data_format="channels_last"):
-        super().__init__()
-        self.weight = nn.Parameter(torch.ones(normalized_shape))
-        self.bias = nn.Parameter(torch.zeros(normalized_shape))
-        self.eps = eps
-        self.data_format = data_format
-        if self.data_format not in ["channels_last", "channels_first"]:
-            raise NotImplementedError
-        self.normalized_shape = (normalized_shape,)
-
-    def forward(self, x):
-        if self.data_format == "channels_last":
-            return F.layer_norm(
-                x, self.normalized_shape, self.weight, self.bias, self.eps
-            )
-        elif self.data_format == "channels_first":
-            u = x.mean(1, keepdim=True)
-            s = (x - u).pow(2).mean(1, keepdim=True)
-            x = (x - u) / torch.sqrt(s + self.eps)
-            x = self.weight[:, None] * x + self.bias[:, None]
-            return x
-
-
-# ConvNeXt Block copied from https://github.com/fishaudio/fish-diffusion/blob/main/fish_diffusion/modules/convnext.py
-class ConvNeXtBlock(nn.Module):
-    r"""ConvNeXt Block. There are two equivalent implementations:
-    (1) DwConv -> LayerNorm (channels_first) -> 1x1 Conv -> GELU -> 1x1 Conv; all in (N, C, H, W)
-    (2) DwConv -> Permute to (N, H, W, C); LayerNorm (channels_last) -> Linear -> GELU -> Linear; Permute back
-    We use (2) as we find it slightly faster in PyTorch
-
-    Args:
-        dim (int): Number of input channels.
-        drop_path (float): Stochastic depth rate. Default: 0.0
-        layer_scale_init_value (float): Init value for Layer Scale. Default: 1e-6.
-        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0.
-        kernel_size (int): Kernel size for depthwise conv. Default: 7.
-        dilation (int): Dilation for depthwise conv. Default: 1.
-    """  # noqa: E501
-
-    def __init__(
-        self,
-        dim: int,
-        drop_path: float = 0.0,
-        layer_scale_init_value: float = 1e-6,
-        mlp_ratio: float = 4.0,
-        kernel_size: int = 7,
-        dilation: int = 1,
-    ):
-        super().__init__()
-
-        self.dwconv = FishConvNet(
-            dim,
-            dim,
-            kernel_size=kernel_size,
-            # padding=int(dilation * (kernel_size - 1) / 2),
-            groups=dim,
-        )  # depthwise conv
-        self.norm = LayerNorm(dim, eps=1e-6)
-        self.pwconv1 = nn.Linear(
-            dim, int(mlp_ratio * dim)
-        )  # pointwise/1x1 convs, implemented with linear layers
-        self.act = nn.GELU()
-        self.pwconv2 = nn.Linear(int(mlp_ratio * dim), dim)
-        self.gamma = (
-            nn.Parameter(layer_scale_init_value * torch.ones((dim)), requires_grad=True)
-            if layer_scale_init_value > 0
-            else None
-        )
-        self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()
-
-    def forward(self, x, apply_residual: bool = True):
-        input = x
-
-        x = self.dwconv(x)
-        x = x.permute(0, 2, 1)  # (N, C, L) -> (N, L, C)
-        x = self.norm(x)
-        x = self.pwconv1(x)
-        x = self.act(x)
-        x = self.pwconv2(x)
-
-        if self.gamma is not None:
-            x = self.gamma * x
-
-        x = x.permute(0, 2, 1)  # (N, L, C) -> (N, C, L)
-        x = self.drop_path(x)
-
-        if apply_residual:
-            x = input + x
-
-        return x
-
-
-class ConvNeXtEncoder(nn.Module):
-    def __init__(
-        self,
-        input_channels: int = 3,
-        depths: list[int] = [3, 3, 9, 3],
-        dims: list[int] = [96, 192, 384, 768],
-        drop_path_rate: float = 0.0,
-        layer_scale_init_value: float = 1e-6,
-        kernel_size: int = 7,
-    ):
-        super().__init__()
-        assert len(depths) == len(dims)
-
-        self.downsample_layers = nn.ModuleList()
-        stem = nn.Sequential(
-            FishConvNet(
-                input_channels,
-                dims[0],
-                kernel_size=7,
-                # padding=3,
-                # padding_mode="replicate",
-                # padding_mode="zeros",
-            ),
-            LayerNorm(dims[0], eps=1e-6, data_format="channels_first"),
-        )
-        self.downsample_layers.append(stem)
-
-        for i in range(len(depths) - 1):
-            mid_layer = nn.Sequential(
-                LayerNorm(dims[i], eps=1e-6, data_format="channels_first"),
-                nn.Conv1d(dims[i], dims[i + 1], kernel_size=1),
-            )
-            self.downsample_layers.append(mid_layer)
-
-        self.stages = nn.ModuleList()
-        dp_rates = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))]
-
-        cur = 0
-        for i in range(len(depths)):
-            stage = nn.Sequential(
-                *[
-                    ConvNeXtBlock(
-                        dim=dims[i],
-                        drop_path=dp_rates[cur + j],
-                        layer_scale_init_value=layer_scale_init_value,
-                        kernel_size=kernel_size,
-                    )
-                    for j in range(depths[i])
-                ]
-            )
-            self.stages.append(stage)
-            cur += depths[i]
-
-        self.norm = LayerNorm(dims[-1], eps=1e-6, data_format="channels_first")
-        self.apply(self._init_weights)
-
-    def _init_weights(self, m):
-        if isinstance(m, (nn.Conv1d, nn.Linear)):
-            nn.init.trunc_normal_(m.weight, std=0.02)
-            nn.init.constant_(m.bias, 0)
-
-    def forward(
-        self,
-        x: torch.Tensor,
-    ) -> torch.Tensor:
-        for i in range(len(self.downsample_layers)):
-            x = self.downsample_layers[i](x)
-            x = self.stages[i](x)
-
-        return self.norm(x)
-
-
-class FireflyArchitecture(nn.Module):
-    def __init__(
-        self,
-        backbone: nn.Module,
-        head: nn.Module,
-        quantizer: nn.Module,
-        spec_transform: nn.Module,
-    ):
-        super().__init__()
-
-        self.backbone = backbone
-        self.head = head
-        self.quantizer = quantizer
-        self.spec_transform = spec_transform
-        self.downsample_factor = math.prod(self.quantizer.downsample_factor)
-
-    def forward(self, x: torch.Tensor, template=None, mask=None) -> torch.Tensor:
-        if self.spec_transform is not None:
-            x = self.spec_transform(x)
-
-        x = self.backbone(x)
-        if mask is not None:
-            x = x * mask
-
-        if self.quantizer is not None:
-            vq_result = self.quantizer(x)
-            x = vq_result.z
-
-            if mask is not None:
-                x = x * mask
-
-        x = self.head(x, template=template)
-
-        if x.ndim == 2:
-            x = x[:, None, :]
-
-        if self.vq is not None:
-            return x, vq_result
-
-        return x
-
-    def encode(self, audios, audio_lengths):
-        audios = audios.float()
-
-        mels = self.spec_transform(audios)
-        mel_lengths = audio_lengths // self.spec_transform.hop_length
-        mel_masks = sequence_mask(mel_lengths, mels.shape[2])
-        mel_masks_float_conv = mel_masks[:, None, :].float()
-        mels = mels * mel_masks_float_conv
-
-        # Encode
-        encoded_features = self.backbone(mels) * mel_masks_float_conv
-        feature_lengths = mel_lengths // self.downsample_factor
-
-        return self.quantizer.encode(encoded_features), feature_lengths
-
-    def decode(self, indices, feature_lengths) -> torch.Tensor:
-        mel_masks = sequence_mask(
-            feature_lengths * self.downsample_factor,
-            indices.shape[2] * self.downsample_factor,
-        )
-        mel_masks_float_conv = mel_masks[:, None, :].float()
-        audio_lengths = (
-            feature_lengths * self.downsample_factor * self.spec_transform.hop_length
-        )
-
-        audio_masks = sequence_mask(
-            audio_lengths,
-            indices.shape[2] * self.downsample_factor * self.spec_transform.hop_length,
-        )
-        audio_masks_float_conv = audio_masks[:, None, :].float()
-
-        z = self.quantizer.decode(indices) * mel_masks_float_conv
-        x = self.head(z) * audio_masks_float_conv
-
-        return x, audio_lengths
-
-    def remove_parametrizations(self):
-        if hasattr(self.backbone, "remove_parametrizations"):
-            self.backbone.remove_parametrizations()
-
-        if hasattr(self.head, "remove_parametrizations"):
-            self.head.remove_parametrizations()
-
-    @property
-    def device(self):
-        return next(self.parameters()).device

fish_speech/models/vqgan/modules/fsq.py

@@ -1,116 +0,0 @@
-from dataclasses import dataclass
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from einops import rearrange
-from vector_quantize_pytorch import GroupedResidualFSQ
-
-from .firefly import ConvNeXtBlock, FishConvNet, FishTransConvNet
-
-
-@dataclass
-class FSQResult:
-    z: torch.Tensor
-    codes: torch.Tensor
-    latents: torch.Tensor
-
-
-class DownsampleFiniteScalarQuantize(nn.Module):
-    def __init__(
-        self,
-        input_dim: int = 512,
-        n_codebooks: int = 9,
-        n_groups: int = 1,
-        levels: tuple[int] = (8, 5, 5, 5),  # Approximate 2**10
-        downsample_factor: tuple[int] = (2, 2),
-        downsample_dims: tuple[int] | None = None,
-    ):
-        super().__init__()
-
-        if downsample_dims is None:
-            downsample_dims = [input_dim for _ in range(len(downsample_factor))]
-
-        all_dims = (input_dim,) + tuple(downsample_dims)
-
-        self.residual_fsq = GroupedResidualFSQ(
-            dim=all_dims[-1],
-            levels=levels,
-            num_quantizers=n_codebooks,
-            groups=n_groups,
-        )
-
-        self.downsample_factor = downsample_factor
-        self.downsample_dims = downsample_dims
-
-        self.downsample = nn.Sequential(
-            *[
-                nn.Sequential(
-                    FishConvNet(
-                        all_dims[idx],
-                        all_dims[idx + 1],
-                        kernel_size=factor,
-                        stride=factor,
-                    ),
-                    ConvNeXtBlock(dim=all_dims[idx + 1]),
-                )
-                for idx, factor in enumerate(downsample_factor)
-            ]
-        )
-
-        self.upsample = nn.Sequential(
-            *[
-                nn.Sequential(
-                    FishTransConvNet(
-                        all_dims[idx + 1],
-                        all_dims[idx],
-                        kernel_size=factor,
-                        stride=factor,
-                    ),
-                    ConvNeXtBlock(dim=all_dims[idx]),
-                )
-                for idx, factor in reversed(list(enumerate(downsample_factor)))
-            ]
-        )
-
-        self.apply(self._init_weights)
-
-    def _init_weights(self, m):
-        if isinstance(m, (nn.Conv1d, nn.Linear)):
-            nn.init.trunc_normal_(m.weight, std=0.02)
-            nn.init.constant_(m.bias, 0)
-
-    def forward(self, z) -> FSQResult:
-        original_shape = z.shape
-        z = self.downsample(z)
-        quantized, indices = self.residual_fsq(z.mT)
-        result = FSQResult(
-            z=quantized.mT,
-            codes=indices.mT,
-            latents=z,
-        )
-        result.z = self.upsample(result.z)
-
-        # Pad or crop z to match original shape
-        diff = original_shape[-1] - result.z.shape[-1]
-        left = diff // 2
-        right = diff - left
-
-        if diff > 0:
-            result.z = F.pad(result.z, (left, right))
-        elif diff < 0:
-            result.z = result.z[..., -left:right]
-
-        return result
-
-    def encode(self, z):
-        z = self.downsample(z)
-        _, indices = self.residual_fsq(z.mT)
-        indices = rearrange(indices, "g b l r -> b (g r) l")
-        return indices
-
-    def decode(self, indices: torch.Tensor):
-        indices = rearrange(indices, "b (g r) l -> g b l r", g=self.residual_fsq.groups)
-        z_q = self.residual_fsq.get_output_from_indices(indices)
-        z_q = self.upsample(z_q.mT)
-        return z_q

fish_speech/models/vqgan/utils.py

@@ -1,94 +0,0 @@
-import matplotlib
-import torch
-from matplotlib import pyplot as plt
-
-matplotlib.use("Agg")
-
-
-def convert_pad_shape(pad_shape):
-    l = pad_shape[::-1]
-    pad_shape = [item for sublist in l for item in sublist]
-    return pad_shape
-
-
-def sequence_mask(length, max_length=None):
-    if max_length is None:
-        max_length = length.max()
-    x = torch.arange(max_length, dtype=length.dtype, device=length.device)
-    return x.unsqueeze(0) < length.unsqueeze(1)
-
-
-def init_weights(m, mean=0.0, std=0.01):
-    classname = m.__class__.__name__
-    if classname.find("Conv") != -1:
-        m.weight.data.normal_(mean, std)
-
-
-def get_padding(kernel_size, dilation=1):
-    return int((kernel_size * dilation - dilation) / 2)
-
-
-def plot_mel(data, titles=None):
-    fig, axes = plt.subplots(len(data), 1, squeeze=False)
-
-    if titles is None:
-        titles = [None for i in range(len(data))]
-
-    plt.tight_layout()
-
-    for i in range(len(data)):
-        mel = data[i]
-
-        if isinstance(mel, torch.Tensor):
-            mel = mel.float().detach().cpu().numpy()
-
-        axes[i][0].imshow(mel, origin="lower")
-        axes[i][0].set_aspect(2.5, adjustable="box")
-        axes[i][0].set_ylim(0, mel.shape[0])
-        axes[i][0].set_title(titles[i], fontsize="medium")
-        axes[i][0].tick_params(labelsize="x-small", left=False, labelleft=False)
-        axes[i][0].set_anchor("W")
-
-    return fig
-
-
-def slice_segments(x, ids_str, segment_size=4):
-    ret = torch.zeros_like(x[:, :, :segment_size])
-    for i in range(x.size(0)):
-        idx_str = ids_str[i]
-        idx_end = idx_str + segment_size
-        ret[i] = x[i, :, idx_str:idx_end]
-
-    return ret
-
-
-def rand_slice_segments(x, x_lengths=None, segment_size=4):
-    b, d, t = x.size()
-    if x_lengths is None:
-        x_lengths = t
-    ids_str_max = torch.clamp(x_lengths - segment_size + 1, min=0)
-    ids_str = (torch.rand([b], device=x.device) * ids_str_max).to(dtype=torch.long)
-    ret = slice_segments(x, ids_str, segment_size)
-    return ret, ids_str
-
-
-@torch.jit.script
-def fused_add_tanh_sigmoid_multiply(in_act, n_channels):
-    n_channels_int = n_channels[0]
-    t_act = torch.tanh(in_act[:, :n_channels_int, :])
-    s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
-    acts = t_act * s_act
-
-    return acts
-
-
-def avg_with_mask(x, mask):
-    assert mask.dtype == torch.float, "Mask should be float"
-
-    if mask.ndim == 2:
-        mask = mask.unsqueeze(1)
-
-    if mask.shape[1] == 1:
-        mask = mask.expand_as(x)
-
-    return (x * mask).sum() / mask.sum()

fish_speech/scheduler.py

@@ -1,40 +0,0 @@
-import math
-
-
-def get_cosine_schedule_with_warmup_lr_lambda(
-    current_step: int,
-    *,
-    num_warmup_steps: int | float,
-    num_training_steps: int,
-    num_cycles: float = 0.5,
-    final_lr_ratio: float = 0.0,
-):
-    if 0 < num_warmup_steps < 1:  # float mode
-        num_warmup_steps = int(num_warmup_steps * num_training_steps)
-
-    if current_step < num_warmup_steps:
-        return float(current_step) / float(max(1, num_warmup_steps))
-
-    progress = float(current_step - num_warmup_steps) / float(
-        max(1, num_training_steps - num_warmup_steps)
-    )
-
-    return max(
-        final_lr_ratio,
-        0.5 * (1.0 + math.cos(math.pi * float(num_cycles) * 2.0 * progress)),
-    )
-
-
-def get_constant_schedule_with_warmup_lr_lambda(
-    current_step: int,
-    *,
-    num_warmup_steps: int | float,
-    num_training_steps: int | None = None,
-):
-    if 0 < num_warmup_steps < 1:  # float mode
-        num_warmup_steps = int(num_warmup_steps * num_training_steps)
-
-    if current_step < num_warmup_steps:
-        return float(current_step) / float(max(1, num_warmup_steps))
-
-    return 1.0

fish_speech/train.py

@@ -1,141 +0,0 @@
-import os
-
-os.environ["USE_LIBUV"] = "0"
-import sys
-from typing import Optional
-
-import hydra
-import lightning as L
-import pyrootutils
-import torch
-from lightning import Callback, LightningDataModule, LightningModule, Trainer
-from lightning.pytorch.loggers import Logger
-from lightning.pytorch.strategies import DDPStrategy
-from omegaconf import DictConfig, OmegaConf
-
-os.environ.pop("SLURM_NTASKS", None)
-os.environ.pop("SLURM_JOB_NAME", None)
-os.environ.pop("SLURM_NTASKS_PER_NODE", None)
-
-# register eval resolver and root
-pyrootutils.setup_root(__file__, indicator=".project-root", pythonpath=True)
-
-# Allow TF32 on Ampere GPUs
-torch.set_float32_matmul_precision("high")
-torch.backends.cudnn.allow_tf32 = True
-
-# register eval resolver
-OmegaConf.register_new_resolver("eval", eval)
-
-import fish_speech.utils as utils
-
-log = utils.RankedLogger(__name__, rank_zero_only=True)
-
-
-@utils.task_wrapper
-def train(cfg: DictConfig) -> tuple[dict, dict]:
-    """Trains the model. Can additionally evaluate on a testset, using best weights obtained during
-    training.
-    This method is wrapped in optional @task_wrapper decorator, that controls the behavior during
-    failure. Useful for multiruns, saving info about the crash, etc.
-    Args:
-        cfg (DictConfig): Configuration composed by Hydra.
-    Returns:
-        Tuple[dict, dict]: Dict with metrics and dict with all instantiated objects.
-    """  # noqa: E501
-
-    # set seed for random number generators in pytorch, numpy and python.random
-    if cfg.get("seed"):
-        L.seed_everything(cfg.seed, workers=False)
-
-    if cfg.get("deterministic"):
-        torch.use_deterministic_algorithms(True)
-
-    log.info(f"Instantiating datamodule <{cfg.data._target_}>")
-    datamodule: LightningDataModule = hydra.utils.instantiate(cfg.data)
-
-    log.info(f"Instantiating model <{cfg.model._target_}>")
-    model: LightningModule = hydra.utils.instantiate(cfg.model)
-
-    log.info("Instantiating callbacks...")
-    callbacks: list[Callback] = utils.instantiate_callbacks(cfg.get("callbacks"))
-
-    log.info("Instantiating loggers...")
-    logger: list[Logger] = utils.instantiate_loggers(cfg.get("logger"))
-
-    log.info(f"Instantiating trainer <{cfg.trainer._target_}>")
-    trainer: Trainer = hydra.utils.instantiate(
-        cfg.trainer,
-        callbacks=callbacks,
-        logger=logger,
-    )
-
-    object_dict = {
-        "cfg": cfg,
-        "datamodule": datamodule,
-        "model": model,
-        "callbacks": callbacks,
-        "logger": logger,
-        "trainer": trainer,
-    }
-
-    if logger:
-        log.info("Logging hyperparameters!")
-        utils.log_hyperparameters(object_dict)
-
-    if cfg.get("train"):
-        log.info("Starting training!")
-
-        ckpt_path = cfg.get("ckpt_path")
-        auto_resume = False
-
-        resume_ckpt_path = utils.get_latest_checkpoint(cfg.paths.ckpt_dir)
-        if resume_ckpt_path is not None:
-            ckpt_path = resume_ckpt_path
-            auto_resume = True
-
-        if ckpt_path is not None:
-            log.info(f"Resuming from checkpoint: {ckpt_path}")
-
-        # resume weights only is disabled for auto-resume
-        if cfg.get("resume_weights_only") and auto_resume is False:
-            log.info("Resuming weights only!")
-            ckpt = torch.load(ckpt_path, map_location=model.device)
-            if "state_dict" in ckpt:
-                ckpt = ckpt["state_dict"]
-            err = model.load_state_dict(ckpt, strict=False)
-            log.info(f"Error loading state dict: {err}")
-            ckpt_path = None
-
-        trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
-
-    train_metrics = trainer.callback_metrics
-
-    if cfg.get("test"):
-        log.info("Starting testing!")
-        ckpt_path = trainer.checkpoint_callback.best_model_path
-        if ckpt_path == "":
-            log.warning("Best ckpt not found! Using current weights for testing...")
-            ckpt_path = cfg.get("ckpt_path")
-
-        trainer.test(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
-        log.info(f"Best ckpt path: {ckpt_path}")
-
-    test_metrics = trainer.callback_metrics
-
-    # merge train and test metrics
-    metric_dict = {**train_metrics, **test_metrics}
-
-    return metric_dict, object_dict
-
-
-@hydra.main(
-    version_base="1.3", config_path="./configs", config_name="llama_pretrain.yaml"
-)
-def main(cfg: DictConfig) -> Optional[float]:
-    # train the model
-    train(cfg)
-
-
-if __name__ == "__main__":
-    main()

tools/e2e_webui.py

@@ -1,232 +0,0 @@
-import io
-import re
-import wave
-
-import gradio as gr
-
-from fish_speech.utils.schema import ServeMessage, ServeTextPart, ServeVQPart
-
-from .fish_e2e import FishE2EAgent, FishE2EEventType
-
-
-def wav_chunk_header(sample_rate=44100, bit_depth=16, channels=1):
-    buffer = io.BytesIO()
-
-    with wave.open(buffer, "wb") as wav_file:
-        wav_file.setnchannels(channels)
-        wav_file.setsampwidth(bit_depth // 8)
-        wav_file.setframerate(sample_rate)
-
-    wav_header_bytes = buffer.getvalue()
-    buffer.close()
-    return wav_header_bytes
-
-
-class ChatState:
-    def __init__(self):
-        self.conversation = []
-        self.added_systext = False
-        self.added_sysaudio = False
-
-    def get_history(self):
-        results = []
-        for msg in self.conversation:
-            results.append({"role": msg.role, "content": self.repr_message(msg)})
-
-        # Process assistant messages to extract questions and update user messages
-        for i, msg in enumerate(results):
-            if msg["role"] == "assistant":
-                match = re.search(r"Question: (.*?)\n\nResponse:", msg["content"])
-                if match and i > 0 and results[i - 1]["role"] == "user":
-                    # Update previous user message with extracted question
-                    results[i - 1]["content"] += "\n" + match.group(1)
-                    # Remove the Question/Answer format from assistant message
-                    msg["content"] = msg["content"].split("\n\nResponse: ", 1)[1]
-        return results
-
-    def repr_message(self, msg: ServeMessage):
-        response = ""
-        for part in msg.parts:
-            if isinstance(part, ServeTextPart):
-                response += part.text
-            elif isinstance(part, ServeVQPart):
-                response += f"<audio {len(part.codes[0]) / 21:.2f}s>"
-        return response
-
-
-def clear_fn():
-    return [], ChatState(), None, None, None
-
-
-async def process_audio_input(
-    sys_audio_input, sys_text_input, audio_input, state: ChatState, text_input: str
-):
-    if audio_input is None and not text_input:
-        raise gr.Error("No input provided")
-
-    agent = FishE2EAgent()  # Create new agent instance for each request
-
-    # Convert audio input to numpy array
-    if isinstance(audio_input, tuple):
-        sr, audio_data = audio_input
-    elif text_input:
-        sr = 44100
-        audio_data = None
-    else:
-        raise gr.Error("Invalid audio format")
-
-    if isinstance(sys_audio_input, tuple):
-        sr, sys_audio_data = sys_audio_input
-    else:
-        sr = 44100
-        sys_audio_data = None
-
-    def append_to_chat_ctx(
-        part: ServeTextPart | ServeVQPart, role: str = "assistant"
-    ) -> None:
-        if not state.conversation or state.conversation[-1].role != role:
-            state.conversation.append(ServeMessage(role=role, parts=[part]))
-        else:
-            state.conversation[-1].parts.append(part)
-
-    if state.added_systext is False and sys_text_input:
-        state.added_systext = True
-        append_to_chat_ctx(ServeTextPart(text=sys_text_input), role="system")
-    if text_input:
-        append_to_chat_ctx(ServeTextPart(text=text_input), role="user")
-        audio_data = None
-
-    result_audio = b""
-    async for event in agent.stream(
-        sys_audio_data,
-        audio_data,
-        sr,
-        1,
-        chat_ctx={
-            "messages": state.conversation,
-            "added_sysaudio": state.added_sysaudio,
-        },
-    ):
-        if event.type == FishE2EEventType.USER_CODES:
-            append_to_chat_ctx(ServeVQPart(codes=event.vq_codes), role="user")
-        elif event.type == FishE2EEventType.SPEECH_SEGMENT:
-            append_to_chat_ctx(ServeVQPart(codes=event.vq_codes))
-            yield state.get_history(), wav_chunk_header() + event.frame.data, None, None
-        elif event.type == FishE2EEventType.TEXT_SEGMENT:
-            append_to_chat_ctx(ServeTextPart(text=event.text))
-            yield state.get_history(), None, None, None
-
-    yield state.get_history(), None, None, None
-
-
-async def process_text_input(
-    sys_audio_input, sys_text_input, state: ChatState, text_input: str
-):
-    async for event in process_audio_input(
-        sys_audio_input, sys_text_input, None, state, text_input
-    ):
-        yield event
-
-
-def create_demo():
-    with gr.Blocks() as demo:
-        state = gr.State(ChatState())
-
-        with gr.Row():
-            # Left column (70%) for chatbot and notes
-            with gr.Column(scale=7):
-                chatbot = gr.Chatbot(
-                    [],
-                    elem_id="chatbot",
-                    bubble_full_width=False,
-                    height=600,
-                    type="messages",
-                )
-
-                # notes = gr.Markdown(
-                #     """
-                # # Fish Agent
-                # 1. 此Demo为Fish Audio自研端到端语言模型Fish Agent 3B版本.
-                # 2. 你可以在我们的官方仓库找到代码以及权重，但是相关内容全部基于 CC BY-NC-SA 4.0 许可证发布.
-                # 3. Demo为早期灰度测试版本，推理速度尚待优化.
-                # # 特色
-                # 1. 该模型自动集成ASR与TTS部分，不需要外挂其它模型，即真正的端到端，而非三段式(ASR+LLM+TTS).
-                # 2. 模型可以使用reference audio控制说话音色.
-                # 3. 可以生成具有较强情感与韵律的音频.
-                # """
-                # )
-                notes = gr.Markdown(
-                    """
-                    # Fish Agent
-                    1. This demo is Fish Audio's self-researh end-to-end language model, Fish Agent version 3B.
-                    2. You can find the code and weights in our official repo in [gitub](https://github.com/fishaudio/fish-speech) and [hugging face](https://huggingface.co/fishaudio/fish-agent-v0.1-3b), but the content is released under a CC BY-NC-SA 4.0 licence.
-                    3. The demo is an early alpha test version, the inference speed needs to be optimised.
-                    # Features
-                    1. The model automatically integrates ASR and TTS parts, no need to plug-in other models, i.e., true end-to-end, not three-stage (ASR+LLM+TTS).
-                    2. The model can use reference audio to control the speech timbre. 
-                    3. The model can generate speech with strong emotion.
-                """
-                )
-
-            # Right column (30%) for controls
-            with gr.Column(scale=3):
-                sys_audio_input = gr.Audio(
-                    sources=["upload"],
-                    type="numpy",
-                    label="Give a timbre for your assistant",
-                )
-                sys_text_input = gr.Textbox(
-                    label="What is your assistant's role?",
-                    value="You are a voice assistant created by Fish Audio, offering end-to-end voice interaction for a seamless user experience. You are required to first transcribe the user's speech, then answer it in the following format: 'Question: [USER_SPEECH]\n\nAnswer: [YOUR_RESPONSE]\n'. You are required to use the following voice in this conversation.",
-                    type="text",
-                )
-                audio_input = gr.Audio(
-                    sources=["microphone"], type="numpy", label="Speak your message"
-                )
-
-                text_input = gr.Textbox(label="Or type your message", type="text")
-
-                output_audio = gr.Audio(
-                    label="Assistant's Voice",
-                    streaming=True,
-                    autoplay=True,
-                    interactive=False,
-                )
-
-                send_button = gr.Button("Send", variant="primary")
-                clear_button = gr.Button("Clear")
-
-        # Event handlers
-        audio_input.stop_recording(
-            process_audio_input,
-            inputs=[sys_audio_input, sys_text_input, audio_input, state, text_input],
-            outputs=[chatbot, output_audio, audio_input, text_input],
-            show_progress=True,
-        )
-
-        send_button.click(
-            process_text_input,
-            inputs=[sys_audio_input, sys_text_input, state, text_input],
-            outputs=[chatbot, output_audio, audio_input, text_input],
-            show_progress=True,
-        )
-
-        text_input.submit(
-            process_text_input,
-            inputs=[sys_audio_input, sys_text_input, state, text_input],
-            outputs=[chatbot, output_audio, audio_input, text_input],
-            show_progress=True,
-        )
-
-        clear_button.click(
-            clear_fn,
-            inputs=[],
-            outputs=[chatbot, state, audio_input, output_audio, text_input],
-        )
-
-    return demo
-
-
-if __name__ == "__main__":
-    demo = create_demo()
-    demo.launch(server_name="127.0.0.1", server_port=7860, share=True)

tools/export_onnx.py

@@ -1,150 +0,0 @@
-import onnxruntime
-import torch
-import torch.nn.functional as F
-
-from fish_speech.conversation import CODEBOOK_PAD_TOKEN_ID
-from tools.vqgan.extract_vq import get_model
-
-PAD_TOKEN_ID = torch.LongTensor([CODEBOOK_PAD_TOKEN_ID])
-
-
-class Encoder(torch.nn.Module):
-    def __init__(self, model):
-        super().__init__()
-        self.model = model
-        self.model.spec_transform.spectrogram.return_complex = False
-
-    def forward(self, audios):
-        mels = self.model.spec_transform(audios)
-        encoded_features = self.model.backbone(mels)
-
-        z = self.model.quantizer.downsample(encoded_features)
-        _, indices = self.model.quantizer.residual_fsq(z.transpose(-2, -1))
-        _, b, l, _ = indices.shape
-        return indices.permute(1, 0, 3, 2).long().view(b, -1, l)
-
-
-class Decoder(torch.nn.Module):
-    def __init__(self, model):
-        super().__init__()
-        self.model = model
-        self.model.head.training = False
-        self.model.head.checkpointing = False
-
-    def get_codes_from_indices(self, cur_index, indices):
-
-        _, quantize_dim, _ = indices.shape
-        d_dim = self.model.quantizer.residual_fsq.rvqs[cur_index].codebooks.shape[2]
-
-        if (
-            quantize_dim
-            < self.model.quantizer.residual_fsq.rvqs[cur_index].num_quantizers
-        ):
-            assert (
-                self.model.quantizer.residual_fsq.rvqs[cur_index].quantize_dropout > 0.0
-            ), "quantize dropout must be greater than 0 if you wish to reconstruct from a signal with less fine quantizations"
-            indices = F.pad(
-                indices,
-                (
-                    0,
-                    self.model.quantizer.residual_fsq.rvqs[cur_index].num_quantizers
-                    - quantize_dim,
-                ),
-                value=-1,
-            )
-
-        mask = indices == -1
-        indices = indices.masked_fill(mask, 0)
-
-        all_codes = torch.gather(
-            self.model.quantizer.residual_fsq.rvqs[cur_index].codebooks.unsqueeze(1),
-            dim=2,
-            index=indices.permute(2, 0, 1).unsqueeze(-1).repeat(1, 1, 1, d_dim),
-        )
-
-        all_codes = all_codes.masked_fill(mask.permute(2, 0, 1).unsqueeze(-1), 0.0)
-
-        scales = (
-            self.model.quantizer.residual_fsq.rvqs[cur_index]
-            .scales.unsqueeze(1)
-            .unsqueeze(1)
-        )
-        all_codes = all_codes * scales
-
-        return all_codes
-
-    def get_output_from_indices(self, cur_index, indices):
-        codes = self.get_codes_from_indices(cur_index, indices)
-        codes_summed = codes.sum(dim=0)
-        return self.model.quantizer.residual_fsq.rvqs[cur_index].project_out(
-            codes_summed
-        )
-
-    def forward(self, indices) -> torch.Tensor:
-        batch_size, _, length = indices.shape
-        dims = self.model.quantizer.residual_fsq.dim
-        groups = self.model.quantizer.residual_fsq.groups
-        dim_per_group = dims // groups
-
-        # indices = rearrange(indices, "b (g r) l -> g b l r", g=groups)
-        indices = indices.view(batch_size, groups, -1, length).permute(1, 0, 3, 2)
-
-        # z_q = self.model.quantizer.residual_fsq.get_output_from_indices(indices)
-        z_q = torch.empty((batch_size, length, dims))
-        for i in range(groups):
-            z_q[:, :, i * dim_per_group : (i + 1) * dim_per_group] = (
-                self.get_output_from_indices(i, indices[i])
-            )
-
-        z = self.model.quantizer.upsample(z_q.transpose(1, 2))
-        x = self.model.head(z)
-        return x
-
-
-def main(firefly_gan_vq_path, llama_path, export_prefix):
-    GanModel = get_model("firefly_gan_vq", firefly_gan_vq_path, device="cpu")
-    enc = Encoder(GanModel)
-    dec = Decoder(GanModel)
-    audio_example = torch.randn(1, 1, 96000)
-    indices = enc(audio_example)
-    torch.onnx.export(
-        enc,
-        audio_example,
-        f"{export_prefix}encoder.onnx",
-        dynamic_axes={
-            "audio": {0: "batch_size", 2: "audio_length"},
-        },
-        do_constant_folding=False,
-        opset_version=18,
-        verbose=False,
-        input_names=["audio"],
-        output_names=["prompt"],
-    )
-
-    torch.onnx.export(
-        dec,
-        indices,
-        f"{export_prefix}decoder.onnx",
-        dynamic_axes={
-            "prompt": {0: "batch_size", 2: "frame_count"},
-        },
-        do_constant_folding=False,
-        opset_version=18,
-        verbose=False,
-        input_names=["prompt"],
-        output_names=["audio"],
-    )
-
-    test_example = torch.randn(1, 1, 96000 * 5)
-    encoder_session = onnxruntime.InferenceSession(f"{export_prefix}encoder.onnx")
-    decoder_session = onnxruntime.InferenceSession(f"{export_prefix}decoder.onnx")
-
-    # check graph has no error
-    onnx_enc_out = encoder_session.run(["prompt"], {"audio": test_example.numpy()})[0]
-    torch_enc_out = enc(test_example)
-    onnx_dec_out = decoder_session.run(["audio"], {"prompt": onnx_enc_out})[0]
-    torch_dec_out = dec(torch_enc_out)
-
-
-if __name__ == "__main__":
-    main("checkpoints/pre/firefly-gan-vq-fsq-8x1024-21hz-generator.pth", None, "test_")

tools/extract_model.py

@@ -1,21 +0,0 @@
-import click
-import torch
-from loguru import logger
-
-
-@click.command()
-@click.argument("model_path")
-@click.argument("output_path")
-def main(model_path, output_path):
-    if model_path == output_path:
-        logger.error("Model path and output path are the same")
-        return
-
-    logger.info(f"Loading model from {model_path}")
-    state_dict = torch.load(model_path, map_location="cpu")["state_dict"]
-    torch.save(state_dict, output_path)
-    logger.info(f"Model saved to {output_path}")
-
-
-if __name__ == "__main__":
-    main()

tools/fish_e2e.py

@@ -1,298 +0,0 @@
-import base64
-import ctypes
-import io
-import json
-import os
-import struct
-from dataclasses import dataclass
-from enum import Enum
-from typing import AsyncGenerator, Union
-
-import httpx
-import numpy as np
-import ormsgpack
-import soundfile as sf
-
-from fish_speech.utils.schema import (
-    ServeChatRequest,
-    ServeMessage,
-    ServeTextPart,
-    ServeVQGANDecodeRequest,
-    ServeVQGANEncodeRequest,
-    ServeVQPart,
-)
-
-
-class CustomAudioFrame:
-    def __init__(self, data, sample_rate, num_channels, samples_per_channel):
-        if len(data) < num_channels * samples_per_channel * ctypes.sizeof(
-            ctypes.c_int16
-        ):
-            raise ValueError(
-                "data length must be >= num_channels * samples_per_channel * sizeof(int16)"
-            )
-
-        self._data = bytearray(data)
-        self._sample_rate = sample_rate
-        self._num_channels = num_channels
-        self._samples_per_channel = samples_per_channel
-
-    @property
-    def data(self):
-        return memoryview(self._data).cast("h")
-
-    @property
-    def sample_rate(self):
-        return self._sample_rate
-
-    @property
-    def num_channels(self):
-        return self._num_channels
-
-    @property
-    def samples_per_channel(self):
-        return self._samples_per_channel
-
-    @property
-    def duration(self):
-        return self.samples_per_channel / self.sample_rate
-
-    def __repr__(self):
-        return (
-            f"CustomAudioFrame(sample_rate={self.sample_rate}, "
-            f"num_channels={self.num_channels}, "
-            f"samples_per_channel={self.samples_per_channel}, "
-            f"duration={self.duration:.3f})"
-        )
-
-
-class FishE2EEventType(Enum):
-    SPEECH_SEGMENT = 1
-    TEXT_SEGMENT = 2
-    END_OF_TEXT = 3
-    END_OF_SPEECH = 4
-    ASR_RESULT = 5
-    USER_CODES = 6
-
-
-@dataclass
-class FishE2EEvent:
-    type: FishE2EEventType
-    frame: np.ndarray = None
-    text: str = None
-    vq_codes: list[list[int]] = None
-
-
-client = httpx.AsyncClient(
-    timeout=None,
-    limits=httpx.Limits(
-        max_connections=None,
-        max_keepalive_connections=None,
-        keepalive_expiry=None,
-    ),
-)
-
-
-class FishE2EAgent:
-    def __init__(self):
-        self.llm_url = "http://localhost:8080/v1/chat"
-        self.vqgan_url = "http://localhost:8080"
-        self.client = httpx.AsyncClient(timeout=None)
-
-    async def get_codes(self, audio_data, sample_rate):
-        audio_buffer = io.BytesIO()
-        sf.write(audio_buffer, audio_data, sample_rate, format="WAV")
-        audio_buffer.seek(0)
-        # Step 1: Encode audio using VQGAN
-        encode_request = ServeVQGANEncodeRequest(audios=[audio_buffer.read()])
-        encode_request_bytes = ormsgpack.packb(
-            encode_request, option=ormsgpack.OPT_SERIALIZE_PYDANTIC
-        )
-        encode_response = await self.client.post(
-            f"{self.vqgan_url}/v1/vqgan/encode",
-            data=encode_request_bytes,
-            headers={"Content-Type": "application/msgpack"},
-        )
-        encode_response_data = ormsgpack.unpackb(encode_response.content)
-        codes = encode_response_data["tokens"][0]
-        return codes
-
-    async def stream(
-        self,
-        system_audio_data: np.ndarray | None,
-        user_audio_data: np.ndarray | None,
-        sample_rate: int,
-        num_channels: int,
-        chat_ctx: dict | None = None,
-    ) -> AsyncGenerator[bytes, None]:
-
-        if system_audio_data is not None:
-            sys_codes = await self.get_codes(system_audio_data, sample_rate)
-        else:
-            sys_codes = None
-        if user_audio_data is not None:
-            user_codes = await self.get_codes(user_audio_data, sample_rate)
-        # Step 2: Prepare LLM request
-        if chat_ctx is None:
-            sys_parts = [
-                ServeTextPart(
-                    text='您是由 Fish Audio 设计的语音助手，提供端到端的语音交互，实现无缝用户体验。首先转录用户的语音，然后使用以下格式回答："Question: [用户语音]\n\nAnswer: [你的回答]\n"。'
-                ),
-            ]
-            if system_audio_data is not None:
-                sys_parts.append(ServeVQPart(codes=sys_codes))
-            chat_ctx = {
-                "messages": [
-                    ServeMessage(
-                        role="system",
-                        parts=sys_parts,
-                    ),
-                ],
-            }
-        else:
-            if chat_ctx["added_sysaudio"] is False and sys_codes:
-                chat_ctx["added_sysaudio"] = True
-                chat_ctx["messages"][0].parts.append(ServeVQPart(codes=sys_codes))
-
-        prev_messages = chat_ctx["messages"].copy()
-        if user_audio_data is not None:
-            yield FishE2EEvent(
-                type=FishE2EEventType.USER_CODES,
-                vq_codes=user_codes,
-            )
-        else:
-            user_codes = None
-
-        request = ServeChatRequest(
-            messages=prev_messages
-            + (
-                [
-                    ServeMessage(
-                        role="user",
-                        parts=[ServeVQPart(codes=user_codes)],
-                    )
-                ]
-                if user_codes
-                else []
-            ),
-            streaming=True,
-            num_samples=1,
-        )
-
-        # Step 3: Stream LLM response and decode audio
-        buffer = b""
-        vq_codes = []
-        current_vq = False
-
-        async def decode_send():
-            nonlocal current_vq
-            nonlocal vq_codes
-
-            data = np.concatenate(vq_codes, axis=1).tolist()
-            # Decode VQ codes to audio
-            decode_request = ServeVQGANDecodeRequest(tokens=[data])
-            decode_response = await self.client.post(
-                f"{self.vqgan_url}/v1/vqgan/decode",
-                data=ormsgpack.packb(
-                    decode_request,
-                    option=ormsgpack.OPT_SERIALIZE_PYDANTIC,
-                ),
-                headers={"Content-Type": "application/msgpack"},
-            )
-            decode_data = ormsgpack.unpackb(decode_response.content)
-
-            # Convert float16 audio data to int16
-            audio_data = np.frombuffer(decode_data["audios"][0], dtype=np.float16)
-            audio_data = (audio_data * 32768).astype(np.int16).tobytes()
-
-            audio_frame = CustomAudioFrame(
-                data=audio_data,
-                samples_per_channel=len(audio_data) // 2,
-                sample_rate=44100,
-                num_channels=1,
-            )
-            yield FishE2EEvent(
-                type=FishE2EEventType.SPEECH_SEGMENT,
-                frame=audio_frame,
-                vq_codes=data,
-            )
-
-            current_vq = False
-            vq_codes = []
-
-        async with self.client.stream(
-            "POST",
-            self.llm_url,
-            data=ormsgpack.packb(request, option=ormsgpack.OPT_SERIALIZE_PYDANTIC),
-            headers={"Content-Type": "application/msgpack"},
-        ) as response:
-
-            async for chunk in response.aiter_bytes():
-                buffer += chunk
-
-                while len(buffer) >= 4:
-                    read_length = struct.unpack("I", buffer[:4])[0]
-                    if len(buffer) < 4 + read_length:
-                        break
-
-                    body = buffer[4 : 4 + read_length]
-                    buffer = buffer[4 + read_length :]
-                    data = ormsgpack.unpackb(body)
-
-                    if data["delta"] and data["delta"]["part"]:
-                        if current_vq and data["delta"]["part"]["type"] == "text":
-                            async for event in decode_send():
-                                yield event
-                        if data["delta"]["part"]["type"] == "text":
-                            yield FishE2EEvent(
-                                type=FishE2EEventType.TEXT_SEGMENT,
-                                text=data["delta"]["part"]["text"],
-                            )
-                        elif data["delta"]["part"]["type"] == "vq":
-                            vq_codes.append(np.array(data["delta"]["part"]["codes"]))
-                            current_vq = True
-
-        if current_vq and vq_codes:
-            async for event in decode_send():
-                yield event
-
-        yield FishE2EEvent(type=FishE2EEventType.END_OF_TEXT)
-        yield FishE2EEvent(type=FishE2EEventType.END_OF_SPEECH)
-
-
-# Example usage:
-async def main():
-    import torchaudio
-
-    agent = FishE2EAgent()
-
-    # Replace this with actual audio data loading
-    with open("uz_story_en.m4a", "rb") as f:
-        audio_data = f.read()
-
-    audio_data, sample_rate = torchaudio.load("uz_story_en.m4a")
-    audio_data = (audio_data.numpy() * 32768).astype(np.int16)
-
-    stream = agent.stream(audio_data, sample_rate, 1)
-    if os.path.exists("audio_segment.wav"):
-        os.remove("audio_segment.wav")
-
-    async for event in stream:
-        if event.type == FishE2EEventType.SPEECH_SEGMENT:
-            # Handle speech segment (e.g., play audio or save to file)
-            with open("audio_segment.wav", "ab+") as f:
-                f.write(event.frame.data)
-        elif event.type == FishE2EEventType.ASR_RESULT:
-            print(event.text, flush=True)
-        elif event.type == FishE2EEventType.TEXT_SEGMENT:
-            print(event.text, flush=True, end="")
-        elif event.type == FishE2EEventType.END_OF_TEXT:
-            print("\nEnd of text reached.")
-        elif event.type == FishE2EEventType.END_OF_SPEECH:
-            print("End of speech reached.")
-
-
-if __name__ == "__main__":
-    import asyncio
-
-    asyncio.run(main())

tools/run_webui.py

@@ -9,8 +9,8 @@ from loguru import logger
 pyrootutils.setup_root(__file__, indicator=".project-root", pythonpath=True)
 
 from fish_speech.inference_engine import TTSInferenceEngine
+from fish_speech.models.dac.inference import load_model as load_decoder_model
 from fish_speech.models.text2semantic.inference import launch_thread_safe_queue
-from fish_speech.models.vqgan.inference import load_model as load_decoder_model
 from fish_speech.utils.schema import ServeTTSRequest
 from tools.webui import build_app
 from tools.webui.inference import get_inference_wrapper

tools/server/agent/__init__.py

@@ -1,57 +0,0 @@
-import struct
-from functools import partial
-
-import ormsgpack
-
-from tools.server.agent.generate import generate_responses
-from tools.server.agent.pre_generation_utils import prepare_messages
-
-
-def execute_request(input_queue, tokenizer, config, request, device):
-    """
-    This function prepares the conversation, encodes the request,
-    sends the generation request, and handles decoding/streaming.
-    It returns a response generator (ServeResponse or ServeStreamResponse).
-    """
-    prompt, im_end_id = prepare_messages(request, tokenizer, config)
-    yield from generate_responses(
-        input_queue, tokenizer, config, request, prompt, im_end_id, device
-    )
-
-
-def response_generator(req, llama_queue, tokenizer, config, device):
-    """
-    Non-streaming response wrapper for the chat endpoint.
-    Only returns the final result.
-    """
-    generator = execute_request(llama_queue, tokenizer, config, req, device)
-    return next(generator)
-
-
-async def streaming_generator(req, llama_queue, tokenizer, config, device, json_mode):
-    """
-    Streaming response wrapper for the chat endpoint.
-    Returns the response in chunks.
-    """
-    generator = execute_request(llama_queue, tokenizer, config, req, device)
-    for i in generator:
-        if json_mode:
-            body = i.model_dump_json().encode("utf-8")
-            yield b"data: " + body + b"\n\n"
-        else:
-            body = ormsgpack.packb(i, option=ormsgpack.OPT_SERIALIZE_PYDANTIC)
-            yield struct.pack("I", len(body)) + body
-
-
-def get_response_generator(
-    llama_queue, tokenizer, config, req, device, json_mode
-) -> partial:
-    """
-    Get the correct response generator based on the request.
-    """
-    if not req.streaming:
-        return partial(response_generator, req, llama_queue, tokenizer, config, device)
-    else:
-        return partial(
-            streaming_generator, req, llama_queue, tokenizer, config, device, json_mode
-        )

tools/server/agent/generate.py

@@ -1,119 +0,0 @@
-import time
-
-from fish_speech.utils.schema import ServeMessage, ServeResponse, ServeStreamResponse
-from tools.server.agent.generation_utils import (
-    initialize_decode_buffers,
-    process_response_tokens,
-    send_reset_buffer,
-)
-from tools.server.agent.pre_generation_utils import (
-    create_generation_request,
-    send_generation_request,
-)
-
-
-def generate_responses(
-    input_queue, tokenizer, config, request, prompt, im_end_id, device
-):
-    """
-    Main generation function that handles the conversation, encodes the request,
-    sends the generation request, and handles decoding/streaming.
-    It returns a response generator (ServeResponse or ServeStreamResponse).
-    """
-    stats = {}
-    start = time.time()
-    stats["start_time"] = start
-    stats["tokens_count"] = 0
-
-    # Prepare and send the generation request
-    req = create_generation_request(prompt, request, im_end_id, device)
-    response_queue = send_generation_request(input_queue, req)
-    decode_buffer, parts, finished = initialize_decode_buffers(request.num_samples)
-
-    while True:
-        response = response_queue.get()
-
-        # Handle abnormal finish or error
-        if response in ["stop", "error"]:
-            finish_reason = response
-            break
-
-        # Process the response tokens
-        is_first_token = stats["tokens_count"] == 0
-        responses = process_response_tokens(
-            response,
-            tokenizer,
-            config,
-            request,
-            decode_buffer,
-            parts,
-            finished,
-            im_end_id,
-            stats,
-            start,
-            is_first_token,
-        )
-
-        # Yield the responses if streaming
-        if request.streaming and responses:
-            for r in responses:
-                yield r
-
-        stats["tokens_count"] += 1
-
-        # Check if all samples are finished
-        if all(finished):
-            finish_reason = "stop"
-            break
-
-    # Finalize the response
-    final_responses = finalize_response(
-        request, finished, decode_buffer, tokenizer, parts, stats, finish_reason
-    )
-    for fr in final_responses:
-        yield fr
-
-
-def finalize_response(
-    request, finished, decode_buffer, tokenizer, parts, stats, finish_reason
-):
-    """
-    Finalize the response by sending the remaining text buffers.
-    """
-    responses = []
-
-    # Send the remaining text buffers
-    for sample_id in range(request.num_samples):
-        responses.extend(
-            send_reset_buffer(sample_id, decode_buffer, tokenizer, parts, request)
-        )
-
-    # Calculate the final stats
-    stats["total_time"] = (time.time() - stats["start_time"]) * 1000
-    stats["total_tokens"] = stats["tokens_count"]
-
-    # If streaming, send the final chunks for each sample
-    if request.streaming:
-        for sample_id in range(request.num_samples):
-            if finished[sample_id]:
-                continue
-            responses.append(
-                ServeStreamResponse(
-                    finish_reason=finish_reason, stats=stats, sample_id=sample_id
-                )
-            )
-    else:
-        # If not streaming, send the full messages for each sample
-        full_messages = [
-            ServeMessage(role="assistant", parts=parts[i])
-            for i in range(request.num_samples)
-        ]
-        responses.append(
-            ServeResponse(
-                messages=full_messages,
-                finish_reason=finish_reason,
-                stats=stats,
-            )
-        )
-
-    return responses

tools/server/agent/generation_utils.py

@@ -1,122 +0,0 @@
-import time
-
-from fish_speech.utils.schema import (
-    ServeStreamDelta,
-    ServeStreamResponse,
-    ServeTextPart,
-    ServeVQPart,
-)
-
-
-def initialize_decode_buffers(num_samples):
-    """Initialise the decode buffers for each sample."""
-    decode_buffer = [[] for _ in range(num_samples)]
-    parts = [[] for _ in range(num_samples)]
-    finished = [False for _ in range(num_samples)]
-    return decode_buffer, parts, finished
-
-
-def send_reset_buffer(sample_id, decode_buffer, tokenizer, parts, request):
-    """Send the remaining text buffer for a sample."""
-    if len(decode_buffer[sample_id]) == 0:
-        return []
-
-    decoded = tokenizer.decode(decode_buffer[sample_id])
-    part = ServeTextPart(text=decoded)
-
-    responses = []
-    if request.streaming:
-        responses.append(ServeStreamResponse(delta=ServeStreamDelta(part=part)))
-    else:
-        parts[sample_id].append(part)
-
-    decode_buffer[sample_id] = []
-    return responses
-
-
-def handle_semantic_tokens(tokens, config, sample_id, parts, request):
-    """Handle the semantic tokens returned by the model."""
-    responses = []
-    _tokens = tokens[1:].clone()
-
-    if not config.share_codebook_embeddings:
-        for i in range(len(_tokens)):
-            _tokens[i] -= config.codebook_size * i
-
-    # If streaming, send the VQ parts directly
-    if request.streaming:
-        responses.append(
-            ServeStreamResponse(
-                sample_id=sample_id,
-                delta=ServeStreamDelta(part=ServeVQPart(codes=_tokens.tolist())),
-            )
-        )
-    else:
-        # If not streaming, accumulate the VQ parts
-        if not parts[sample_id] or not isinstance(parts[sample_id][-1], ServeVQPart):
-            parts[sample_id].append(ServeVQPart(codes=_tokens.tolist()))
-        else:
-            # Accumulate the codes
-            for codebook_id, value in enumerate(_tokens):
-                parts[sample_id][-1].codes[codebook_id].append(value.item())
-
-    return responses
-
-
-def process_response_tokens(
-    response,
-    tokenizer,
-    config,
-    request,
-    decode_buffer,
-    parts,
-    finished,
-    im_end_id,
-    stats,
-    start,
-    is_first_token,
-):
-    """Process the response tokens returned by the model."""
-    responses = []
-    for sample_id, tokens in enumerate(response):
-        if finished[sample_id]:
-            continue
-
-        # End of the conversation
-        if tokens[0] == im_end_id:
-            finished[sample_id] = True
-            # Send the remaining text buffer
-            responses.extend(
-                send_reset_buffer(sample_id, decode_buffer, tokenizer, parts, request)
-            )
-            if request.streaming:
-                responses.append(
-                    ServeStreamResponse(
-                        sample_id=sample_id,
-                        finish_reason="stop",
-                        stats=stats,
-                    )
-                )
-            continue
-
-        # Check if the token is semantic
-        is_semantic = (
-            tokenizer.semantic_begin_id <= tokens[0] <= tokenizer.semantic_end_id
-        )
-
-        if is_semantic:
-            # Before the semantic tokens, send the remaining text buffer
-            responses.extend(
-                send_reset_buffer(sample_id, decode_buffer, tokenizer, parts, request)
-            )
-            responses.extend(
-                handle_semantic_tokens(tokens, config, sample_id, parts, request)
-            )
-        else:
-            # Accumulate the text tokens (not implemented?)
-            decode_buffer[sample_id].append(tokens[0, 0])
-
-    if is_first_token:
-        stats["time_to_first_token"] = (time.time() - start) * 1000
-
-    return responses

tools/server/agent/pre_generation_utils.py

@@ -1,72 +0,0 @@
-import queue
-
-from fish_speech.conversation import Conversation, Message
-from fish_speech.models.text2semantic.inference import GenerateRequest
-from fish_speech.tokenizer import IM_END_TOKEN
-
-
-def prepare_messages(request, tokenizer, config):
-    """
-    Reorganise the provided list of messages into a conversation.
-    Encode the conversation for inference.
-    """
-    # Convert the messages to ConversationMessage objects
-    messages = [msg.to_conversation_message() for msg in request.messages]
-
-    if len(messages) < 1:
-        raise ValueError("At least one message is required")
-
-    # Check the last message to determine the next step
-    last_role = messages[-1].role
-    match last_role:
-        case "user":
-            # The last message is from the user, ask the assistant to respond with a new message
-            messages.append(
-                Message(role="assistant", parts=[], add_im_end=False, modality="voice")
-            )
-        case "raw":
-            # The last message is raw text, ask the assistant to complete it
-            messages[-1].add_im_start = False
-            messages[-1].add_im_end = False
-            messages[-1].modality = "voice"
-        case "assistant":
-            # The last message is from the assistant, ask the assistant to continue
-            messages[-1].add_im_end = False
-        case _:
-            # We expect it to be assistant if not user or raw
-            raise ValueError("The last message must be from the assistant, user or raw")
-
-    # Create a conversation object and encode it for inference
-    conv = Conversation(messages=messages)
-    prompt = conv.encode_for_inference(
-        tokenizer=tokenizer, num_codebooks=config.num_codebooks
-    )
-    im_end_id = tokenizer.get_token_id(IM_END_TOKEN)
-
-    return prompt, im_end_id
-
-
-def create_generation_request(prompt, request, im_end_id, device):
-    """
-    Convert the request into a dictionary that can be sent to the model for generation.
-    """
-    req = {
-        "prompt": prompt.to(device),
-        "max_new_tokens": request.max_new_tokens,
-        "im_end_id": im_end_id,
-        "temperature": request.temperature,
-        "top_p": request.top_p,
-        "repetition_penalty": request.repetition_penalty,
-        "num_samples": request.num_samples,
-        "early_stop_threshold": request.early_stop_threshold,
-    }
-    return req
-
-
-def send_generation_request(input_queue, req):
-    """
-    Send the generation request to the model and return a queue to get the response.
-    """
-    response_queue = queue.Queue()
-    input_queue.put(GenerateRequest(req, response_queue))
-    return response_queue

tools/server/model_manager.py

@@ -3,11 +3,11 @@ from funasr import AutoModel
 from loguru import logger
 
 from fish_speech.inference_engine import TTSInferenceEngine
+from fish_speech.models.dac.inference import load_model as load_decoder_model
 from fish_speech.models.text2semantic.inference import (
     launch_thread_safe_queue,
     launch_thread_safe_queue_agent,
 )
-from fish_speech.models.vqgan.inference import load_model as load_decoder_model
 from fish_speech.utils.schema import ServeTTSRequest
 from tools.server.inference import inference_wrapper as inference