唱出来 mellotron 开源项目 demo

TZLD ab54463a9a master 4 anos atrás
__pycache__ ab54463a9a master 4 anos atrás
data ab54463a9a master 4 anos atrás
filelists ab54463a9a master 4 anos atrás
text ab54463a9a master 4 anos atrás
venv ab54463a9a master 4 anos atrás
waveglow ab54463a9a master 4 anos atrás
LICENSE ab54463a9a master 4 anos atrás
README.md ab54463a9a master 4 anos atrás
audio_processing.py ab54463a9a master 4 anos atrás
audio_stereo.wav ab54463a9a master 4 anos atrás
audio_stereo_1.wav ab54463a9a master 4 anos atrás
data_utils.py ab54463a9a master 4 anos atrás
distributed.py ab54463a9a master 4 anos atrás
fp16_optimizer.py ab54463a9a master 4 anos atrás
hparams.py ab54463a9a master 4 anos atrás
inference.ipynb ab54463a9a master 4 anos atrás
inference.py ab54463a9a master 4 anos atrás
layers.py ab54463a9a master 4 anos atrás
logger.py ab54463a9a master 4 anos atrás
loss_function.py ab54463a9a master 4 anos atrás
loss_scaler.py ab54463a9a master 4 anos atrás
mellotron_logo.png ab54463a9a master 4 anos atrás
mellotron_utils.py ab54463a9a master 4 anos atrás
model.py ab54463a9a master 4 anos atrás
modules.py ab54463a9a master 4 anos atrás
multiproc.py ab54463a9a master 4 anos atrás
plotting_utils.py ab54463a9a master 4 anos atrás
requirements.txt ab54463a9a master 4 anos atrás
stft.py ab54463a9a master 4 anos atrás
train.py ab54463a9a master 4 anos atrás
utils.py ab54463a9a master 4 anos atrás
yin.py ab54463a9a master 4 anos atrás

README.md

Mellotron

Rafael Valle*, Jason Li*, Ryan Prenger and Bryan Catanzaro

In our recent paper we propose Mellotron: a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data.

By explicitly conditioning on rhythm and continuous pitch contours from an audio signal or music score, Mellotron is able to generate speech in a variety of styles ranging from read speech to expressive speech, from slow drawls to rap and from monotonous voice to singing voice.

Visit our website for audio samples.

Pre-requisites

  1. NVIDIA GPU + CUDA cuDNN

Setup

  1. Clone this repo: git clone https://github.com/NVIDIA/mellotron.git
  2. CD into this repo: cd mellotron
  3. Initialize submodule: git submodule init; git submodule update
  4. Install PyTorch
  5. Install Apex
  6. Install python requirements or build docker image
    • Install python requirements: pip install -r requirements.txt

Training

  1. Update the filelists inside the filelists folder to point to your data
  2. python train.py --output_directory=outdir --log_directory=logdir
  3. (OPTIONAL) tensorboard --logdir=outdir/logdir

Training using a pre-trained model

Training using a pre-trained model can lead to faster convergence
By default, the speaker embedding layer is ignored

  1. Download our published Mellotron model trained on LibriTTS or LJS
  2. python train.py --output_directory=outdir --log_directory=logdir -c models/mellotron_libritts.pt --warm_start

Multi-GPU (distributed) and Automatic Mixed Precision Training

  1. python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

Inference demo

  1. jupyter notebook --ip=127.0.0.1 --port=31337
  2. Load inference.ipynb
  3. (optional) Download our published WaveGlow model

Related repos

WaveGlow Faster than real time Flow-based Generative Network for Speech Synthesis.

Acknowledgements

This implementation uses code from the following repos: Keith Ito, Prem Seetharaman, Chengqi Deng, Patrice Guyot, as described in our code.