F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (Oct 2024)

Title: F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
Link: https://arxiv.org/abs/2410.06885
Date: 15 Oct 2024
Authors: Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu and Xie Chen

Summary:

This research paper introduces F5-TTS, a novel text-to-speech system that generates highly natural and expressive speech without the need for complex components like duration models or phoneme alignment. The authors achieve this by leveraging the "flow matching" technique with Diffusion Transformer (DiT), a powerful neural architecture. F5-TTS tackles the limitations of previous methods, such as slow convergence and low robustness, by refining the text representation with ConvNeXt blocks and introducing a novel "Sway Sampling" strategy for flow steps during inference. This strategy significantly improves performance and efficiency, allowing for faster training and a real-time factor of 0.15. The paper provides a comprehensive overview of the F5-TTS system, including its architecture, training process, and evaluation on various datasets. It also presents detailed ablation studies comparing F5-TTS to other state-of-the-art models, demonstrating its superior performance and robustness in zero-shot speech generation.

Key Topics:

Flow Matching, Text-to-Speech, Diffusion Model, Zero-Shot TTS, Sway Sampling