Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech
Investigates training-side benefits of mask-and-replace diffusion for zero-shot text-to-speech beyond inference-time token revision.
SPEECH · AUDIO · MUSIC
Hounsu Kim.
Ph.D. student at KAIST
Music and Audio Computing Lab · Advisor: Juhan Nam
Speech representations &
generative models.
I’m a Ph.D. student in the Music and Audio Computing Lab at KAIST, advised by Prof. Juhan Nam. My current research focuses on speech representation learning and generative modeling, particularly neural speech codecs, audio tokenization, and zero-shot speech generation. I study how disentangled representations and discrete diffusion can support efficient and controllable speech synthesis. This work builds on my research in music generation, piano transcription, and expressive instrument synthesis. I also work on evaluating audio-language models.
Latest
Beyond Token Revision is available as a preprint.
SDP-Codec at Interspeech · VocalCoachBench at NeurIPS E&D.
RESEARCH IN DETAIL
Publications & preprints
Speech, audio, and music.
Click a figure to take a closer look.
VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing
A benchmark for evaluating audio-language models on expert vocal coaching feedback for singing.
SDP-Codec: A Speaker-Decoupled Speech Codec with Pitch Injection for Low-Bitrate Coding and Zero-Shot Voice Conversion
Neural speech codec for disentangled speech representation learning, low-bitrate coding, and zero-shot voice conversion.
D3PIA: A Discrete Denoising Diffusion Model for Piano Accompaniment Generation From Lead Sheet
A lightweight discrete diffusion model for lead-sheet-conditioned symbolic music generation.
D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription
Discrete diffusion-based refinement model for structured piano transcription.
CoNMod: Controllable Neural Frame-based Modulation Effects
Best Student Paper Award
Controllable neural audio effect modeling.
Expressive Acoustic Guitar Sound Synthesis with an Instrument-Specific Input Representation and Diffusion Outpainting
Controllable acoustic instrument synthesis using instrument-specific representations and diffusion outpainting.
BACKGROUND
Research & education
Research experience
Recognition
20261st place · VoiceMOS Challenge
Track 2
2024Best Student Paper · DAFx
CoNMod, co-author
Education
2023 - present
Ph.D Student in Graduate School of Culture Technology
KAIST
Music and Audio Computing Lab (Advisor - Juhan Nam)
2021 - 2023
MSc in Graduate School of Culture Technology
KAIST
Music and Audio Computing Lab (Advisor - Juhan Nam)
2015 - 2021
BSc in Mechanical Engineering
KAIST
2012 - 2015
Diploma, High School
Gyeonggi Science High School for the Gifted
WORKING WITH OTHERS
Advising & teaching
Academic advising
Mar 2026 - present
Tristan George George
Department of Computer Science, KAIST
Dec 2024 - Apr 2025
Donghoon Kang
Department of Computer Science, KAIST
Sep 2023 - Mar 2024
Gyubin Lee
Department of Electronic Engineering, KAIST
Teaching assistant · KAIST
Musical Applications of Machine Learning
Spring 2025 · Spring 2024 · Fall 2022
Fundamentals of Computer Music
Fall 2024 · Fall 2023
Introduction to Culture Technology
Spring 2022
Projects & technical skills
Projects
ToplAIner - Acoustic Guitar Conditioned Singing Voice Generation [Poster] ↗
2023 Fall KAIST GCT731 Topics in Music Technology <Generative AI for Music> Final Project
Technical skills
- Python & PyTorch
- Deep Learning & Generative Models
- Audio, Speech & Music Signal Processing
- Representation Learning & Audio Tokenization
- Diffusion Models, Flow-based Models & Neural Codecs
- Experiment Management, Model Training & Evaluation
- Linux Server Administration, Docker, SLURM & Git