Introduction to ConvTasNet

►Preface

In recent years, the "time-domain speech separation model" has become an important direction in breaking through traditional frequency-domain methods in the field of speech processing. Conv-TasNet (Convolutional Time-domain Audio Separation Network) is a representative framework in this area. Unlike previous approaches that relied on STFT (Short-Time Fourier Transform) to convert speech into the frequency domain, Conv-TasNet directly performs end-to-end learning in the "time domain." It uses a learnable encoder and decoder paired with a deep convolutional network as the separation module, achieving speech source separation and noise suppression with low latency and high accuracy.

►Introduction to ConvTasNet

The core advantage of Conv-TasNet lies in its lightweight structure and real-time processing capability, making it particularly suitable for embedded devices, mobile devices, and voice application scenarios requiring high responsiveness, such as conference systems, smart assistants, communication preprocessing, and speech recognition frontends. Since its introduction in 2018, Conv-TasNet has significantly surpassed many traditional baselines (such as TasNet and uPIT) in terms of separation quality and has become the foundational design for various subsequent speech separation models (such as DPRNN, DPTNet, SepFormer, etc.).

Conv-TasNet model architecture:

Conv-TasNet mainly consists of three parts:

1. Encoder (learnable encoder)
   • Project the original waveform (e.g., 8kHz or 16kHz) into high-dimensional feature representations through 1D convolution.
   • There is no need to perform STFT; the model completely determines the most suitable features.
   • Encoder output: similar to 'speech representation vector'
2. Separator (Speech Separator): TCN Temporal Convolutional Network
   • Dilated Convolution (dilated conv)
   • Residual structure (residual + skip)
   • Hierarchical receptive field expansion
3. Decoder (learnable decoder)
   • Use 1D transposed convolution (Transposed Conv)
   • Reconstruct the separated features back into a waveform
   • Each mask corresponds to a separated audio stream.

►Advantages of ConvTasNet

1. Low Latency

Because it does not require STFT/iSTFT, it is particularly suitable for:

• Real-time call noise reduction
• Meeting system

• Mobile devices (phones, AI earbuds, edge devices)

   

2. High-quality separation (starting point of SOTA time-domain model)

Conv-TasNet, when proposed, outperformed:

• uPIT + BLSTM
• TasNet

• Various STFT-based baseline SDR performances have improved significantly.

   

3. Lightweight and easy to deploy

Compared to the later DPRNN and SepFormer,
The model size of Conv-TasNet is usually smaller, with lower latency.

Therefore, it is well-suited to be deployed in:

• Embedded/IoT devices
• ONNX / TFLite Inference

►Summary

Through the explanation above, I believe everyone has a better understanding ofUse Conv-TasNetLooking forward to gaining a deeper understanding, stay tuned for the next blog post!

►Q&A

Question 1: CWhat is the biggest difference between onv-TasNet and traditional speech separation methods that use STFT?

Conv-TasNet processes waveforms directly in the 'time domain' without requiring STFT and phase estimation, thus avoiding the resolution limitations and phase errors of frequency domain methods. This enables it toLow latency, high-quality separationHas significant advantages, especially suitable for real-time applications.

Question Two:What types of speech tasks can Conv-TasNet accomplish?

It can be used for various speech processing needs:

• Multi-speaker voice separation (speech separation)

• Noise reduction (denoising/noise suppression)

• Echo suppression

Question 3: Will subsequent models be stronger than it?

In terms of 'accuracy,' the subsequent DPRNN, DPTNet, and SepFormer can surpass Conv-TasNet on large datasets.

Question 4: Is Conv-TasNet still mainstream?

In terms of 'deployment difficulty,' 'latency,' and 'real-time performance,' Conv-TasNet remains one of the mainstream models most suitable for deployment on edge devices.

Question 5: What are the bottlenecks of deploying Conv-TasNet on ONNX / DSP / NPU? Which operations are the most computationally intensive?

Deployment bottlenecks are often due to the heavy computation of TCN's extensive 1D convolutions and deep residual stacking, the lack of efficient hardware kernel support for dilated convolutions, and the slower performance of ConvTranspose (decoder) on certain hardware.

►References

https://github.com/JusperLee/Conv-TasNet