Skip to content

Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Aug 2026 · arXiv.org · Vol abs/2608.28472 · 0 citations · 38 references
Computer Science Engineering

TL;DR

This paper proposes a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation, and achieves robust speech classification and enhancement gains at low-power sampling-rates.

Abstract

Deep neural networks (DNNs) based on state-space models (SSMs) are increasingly applied to speech processing, but typically operate on pulse-code-modulated (PCM) audio. This constrains deployment on low-power, always-on edge devices, which commonly use single-bit pulse-density-modulated (PDM) micro-electromechanical (MEMS) microphones for their noise robustness, low cost, and variable sampling rates that enable low-power operation. In fact, converting PDM to PCM requires low-pass filtering and decimation, imposing costly overhead on resource-constrained hardware. While prior works have attempted to process PDM signals directly, they require long training times and generalize poorly across sampling rates. In this paper, we show that the SSM has two key properties that remediate these issues: its continuous-time parametrization allows it to produce a consistent representation of the input audio signal, regardless of the modulation strategy and sampling rate, and its long-term memory enables this representation to be aggressively downsampled without needing any anti-aliasing operations. We then propose a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation. We show that our proposed architecture achieves robust speech classification and enhancement gains at low-power sampling-rates (512 kHz) and similar performance to state-of-the-art algorithms operating on PCM data when tested on standard PDM sampling-rates of 2 MHz. Moreover, we show that the SSM's output can be downsampled by more than 65,000 times, thus significantly reducing the number of processing timesteps in downstream layers.

View source

Similar papers

Preprint Sep 2026

Configurable-Bandwidth Time-Frequency Modeling for Efficient Full-Band Speech Enhancement Across Sampling Rates

Speech enhancement systems are often developed for a fixed sampling rate, while time-frequency models become more expensive as the number of frequency bins increases. We propose TF-Refiner, a sampling-frequency-independent model that decouples the deep analysis bandwidth from the full-band input and output. A deep enco...

U.H Shin, Wooseok Kim, Hyung-Min Park · 0 citations
Sep 2026

Narrowband Voice Communication Using Streaming Neural Compression

Low-bitrate speech communication on resource-constrained edge devices remains challenging due to stringent computational, memory, and bandwidth constraints. We present TinyCall, a lightweight neural audio codec designed for real-time speech communication on low-power platforms such as the ESP32 microcontroller and Rasp...

Da-Hong Luo, Anannya Trehan, Aritrik Ghosh et al. · 0 citations
Preprint Oct 2026

StreamSE: Neural Audio Codec-based Streaming Speech Enhancement for On-device Inference

Recent generative speech enhancement (SE) models achieve high perceptual quality, but their non-causal processing and expensive computation make them difficult to use for streaming, on-device inference. We propose StreamSE, a 20M-parameter streaming SE model that operates in the latent space of a frozen pretrained neur...

Hyunjin Choi, Zinu Kim, Junwon Moon et al. · 0 citations
Preprint Oct 2026

Lyric: Wave-Domain Computing for Efficient Spoken-Digit Recognition

Low-power speech recognition requires reducing audio acquisition and feature-processing costs as well as neural inference. We present Lyric, a speech-recognition front end that uses wave-domain computing to extract features before digitization. Our proposed system uses passive acoustic resonators to separate speech by...

Jeeven Balasubramaniam, Nakul Garg · 0 citations
Preprint Sep 2026

Real-Time Music Source Separation on a Low-Power Audio DSP

Real-time music source separation is validated on desktop CPUs and GPUs. Does any published system fit the embedded audio hardware it targets? On a commercial audio DSP (2 MB SRAM, 2.07 GMAC/s measured), none does, and the constraints eliminate different models: memory rules out the 16-51 M parameter TasNet/X-UMX famil...

Jia-Nan Li, Li Liu, K. Malsky et al. · 0 citations
Open access 2026

Performance Evaluation of DenseNet-121 and Bi-LSTM with Attention Mechanism based Vocal Signal Separation

This paper presents a supervised attention-based deep learning approach for separating single-channel audio signals that establishes a low complexity, scalable, and interpretable solution to realistic problems in source separation.

Arfana Chowdhary, Parveen Kumar Lehana · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.