All projects全部项目
Voice Privacy · Audio ML

SceneGuard

Training-Time Voice Protection with Scene-Consistent Audible Background Noise

用场景一致的可听背景噪声进行训练时语音保护

Rui Sang, Yuxuan Liu

AAAI-26 Workshop on Artificial Intelligence for Cyber Security (AICS) · Accepted已接收 · First author· 第一作者

SceneGuard protects published speech by adding scene-matched audible noise whose timing and strength are optimized to reduce speaker-identity similarity while preserving intelligibility.

SceneGuard 通过添加与录音场景匹配的可听噪声来保护已发布语音；噪声的时序与强度经过优化，以降低说话人身份相似度，同时尽量保持可懂度。

Paper Code Audio Demo音频演示
Three-step SceneGuard concept showing original speech, scene-matched protection, and useful speech with reduced voice-cloning risk
30-second overview
30 秒概览

What SceneGuard does

SceneGuard 做什么

Input
输入

Speech + scene context

语音 + 场景上下文

Speech \(x(t)\), an acoustic scene \(s\), and a scene-specific noise library \(\mathcal{N}_s\).

语音 \(x(t)\)、声学场景 \(s\)，以及该场景对应的噪声库 \(\mathcal{N}_s\)。

Output
输出

Protected speech

受保护语音

An audio signal \(x'(t)\) containing audible noise selected to match the recording context.

带有可听噪声的音频信号 \(x'(t)\)，噪声按录音场景选取。

Goal
目标

Reduce speaker identity

降低说话人身份信息

Lower ECAPA speaker similarity while retaining speech intelligibility and practical usability.

降低 ECAPA 说话人相似度，同时尽量保持可懂度与可用性。

Core idea
核心思路

Optimize placement + strength

优化投放位置与强度

Learn a temporal mask \(m(t)\) and global gain \(\gamma\) for a scene-matched noise sample.

为场景匹配噪声学习时间掩码 \(m(t)\) 与全局增益 \(\gamma\)。

Method in one line:一句话方法： SceneGuard is a scene-conditioned audio protection pipeline with per-sample gradient-based mask and gain optimization—not a newly trained neural-network architecture.SceneGuard 是按样本优化掩码与增益的场景条件音频保护流程，而不是新训练的神经网络架构。
Interactive evidence
交互证据

Audio demo

音频演示

The comparison interface is ready for curated Clean / Baseline / SceneGuard samples, with matching waveforms and mel spectrograms.

对比界面已预留 Clean / Baseline / SceneGuard 样本位置，并配套波形图与 Mel 频谱。

Clean

Original speech before protection.保护前的原始语音。

Random / Gaussian baseline

A matched-SNR additive-noise baseline.匹配 SNR 的加性噪声基线。

SceneGuard

Scene-conditioned noise with optimized mask and gain.场景条件噪声，掩码与增益经过优化。

Demo assets pending.演示音频待补充。 No redistributable SceneGuard demo audio is currently available in the public repository.公开仓库目前没有可再分发的 SceneGuard 演示音频。 This structure will activate after sample provenance and redistribution rights are documented.在样本来源与再分发权限确认后再启用实际音频。

No audio is embedded yet. The public repository contains no WAV examples, and the TAU noise dataset has non-commercial redistribution restrictions. No synthetic or untraceable sample has been substituted.目前尚未嵌入音频。公开仓库没有 WAV 样例，且 TAU 噪声数据有非商业再分发限制。这里没有用无法追溯来源的合成样本代替。
Problem & threat model
问题与威胁模型

Protecting speech before an attacker collects it

在攻击者收集之前保护语音

Conventional paradigm
常见范式

Imperceptible perturbation

不可感知扰动

Prior proactive defenses often constrain perturbations to be difficult to hear, but such low-energy signals may be vulnerable to compression, filtering, or purification.

已有主动防御常把扰动限制为难以听见，但这类低能量信号可能容易被压缩、滤波或净化去掉。

SceneGuard design choice
SceneGuard 的设计选择

Scene-matched audible protection

场景匹配的可听保护

SceneGuard trades strict imperceptibility for context-aware noise selection and tests whether protection persists under common preprocessing operations.

SceneGuard 用场景感知的噪声选择换取不再追求严格不可听，并检验保护是否能在常见预处理后仍然有效。

Paper-described threat model Full end-to-end BERT-VITS2 fine-tuning was not performed because of computational constraints; speaker-embedding degradation is used as a proxy, complemented by a zero-shot evaluation.因计算资源限制，未做完整端到端 BERT-VITS2 微调；说话人嵌入退化作为代理指标，并辅以 zero-shot 评估。

Signal transformation
信号变换

Input → output

输入 → 输出

$$x'(t) = x(t) + \gamma\,m(t)\odot n_k(t), \qquad n_k(t)\sim\mathcal{N}_s$$
x(t) clean input speech干净输入语音
s predicted or user-provided scene预测或用户指定的场景
n_k(t) noise sampled from the scene library从场景噪声库采样的噪声
m(t) temporal mask in \([0,1]^T\)时间掩码，取值 \([0,1]^T\)
γ global noise strength全局噪声强度
x'(t) protected speech output受保护语音输出
System architecture
系统架构

Pretrained perception, per-sample optimization

预训练感知模块，按样本优化

PANNs, ECAPA-TDNN, and Whisper are pretrained components. SceneGuard does not train them; gradients update only the temporal mask and global gain for each sample.

PANNs、ECAPA-TDNN 和 Whisper 都是预训练组件。SceneGuard 不训练它们；梯度只更新每个样本的时间掩码与全局增益。

SceneGuard architecture showing scene and noise selection, mask and gain optimization, protected speech generation, usability and robustness evaluation, and training-time and zero-shot attacks
Ablation
消融

Why optimize the mixture?

为什么要优化混合过程？

Direct mixing
直接混合

Fixed / unoptimized placement

固定、未优化的投放

2.8%
paper-reported protection
论文报告的保护强度

SIM 0.972 · STOI 0.989 · WER 3.2%

SceneGuard

Learned mask + constrained gain

学习掩码 + 受约束增益

5.5%
paper-reported protection
论文报告的保护强度

SIM 0.945 · STOI 0.986 · WER 3.6%

Paper Table 6 Optimization improves reported similarity degradation by 2.7 percentage points under the same SNR constraint, with a 0.003 STOI decrease and 0.4-point WER increase.在相同 SNR 约束下，优化使报告的相似度下降提高 2.7 个百分点；STOI 仅下降 0.003，WER 上升 0.4 个百分点。

Optimization objective
优化目标

What is actually optimized?

实际优化了什么？

$$\mathcal{L}_{\mathrm{default}} = \lambda_{\mathrm{SIM}}\,\mathrm{cos}\!\left(e(x'),e(x)\right) + \lambda_{\mathrm{REG}}\left(\|\nabla m\|_2^2+\gamma^2\right)$$ $$\text{subject to}\quad \mathrm{SNR}\!\left(x,\gamma m\odot n_k\right)\in[10,20]\ \mathrm{dB}$$
Speaker similarity说话人相似度 Minimize cosine similarity between clean and protected ECAPA embeddings.最小化干净语音与受保护语音 ECAPA 嵌入的余弦相似度。
Mask smoothness掩码平滑 Penalize abrupt temporal changes that may create unstable or spiky masks.惩罚可能导致掩码尖峰或不稳定的剧烈时间变化。
Energy penalty能量惩罚 Regularize the global noise strength \(\gamma\).对全局噪声强度 \(\gamma\) 做正则。
SNR constraintSNR 约束 Bound the paper’s default operating range to 10–20 dB.将论文默认工作区间限制在 10–20 dB。
General paper formulation and disabled terms论文一般形式与默认关闭项
$$\mathcal{L}=\lambda_{\mathrm{SIM}}\mathcal{L}_{\mathrm{SIM}} +\lambda_{\mathrm{REG}}\mathcal{L}_{\mathrm{REG}} +\lambda_{\mathrm{ASR}}\mathcal{L}_{\mathrm{ASR}} +\lambda_{\mathrm{SCN}}\mathcal{L}_{\mathrm{SCN}}$$

In the default experiments, \(\lambda_{\mathrm{ASR}}=0\) and \(\lambda_{\mathrm{SCN}}=0\). Scene consistency is introduced mainly by selecting noise from the scene-specific library; usability is enforced primarily through SNR and evaluated after optimization.

默认实验中 \(\lambda_{\mathrm{ASR}}=0\)、\(\lambda_{\mathrm{SCN}}=0\)。场景一致性主要靠从对应噪声库取样来引入；可用性主要靠 SNR 约束保证，并在优化后评估。

Evaluation framework
评估框架

Four views of the protection–quality trade-off

保护与质量权衡的四个视角

Protection

保护效果

Speaker Similarity (SIM) ↓
Cosine similarity between speaker embeddings.说话人嵌入之间的余弦相似度。

Usability

可用性

WER ↓ · STOI ↑ · PESQ ↑
Transcription, intelligibility, and objective quality.转写、可懂度与客观音质。

Robustness

鲁棒性

SIM after preprocessing
MP3, spectral subtraction, low-pass, and downsampling.MP3、谱减、低通与降采样。

Zero-shot

SIM + success rate ↓
Cloning with clean versus protected reference audio.分别用干净参考与受保护参考做克隆。

Paper-reported results
论文报告结果

Main comparison

主要对比

Comparison of speaker similarity and word error rate for clean data, random noise, Gaussian noise, and SceneGuard
SIM1.000 → 0.945 training-attack proxy comparison训练攻击代理对比
STOI0.98695% CI [0.980, 0.992]
WER3.60% usability evaluation可用性评估
Effect size2.18 Cohen’s d; paper reports p < 10−15Cohen’s d；论文报告 p < 10−15
View exact values from Paper Table 1查看 Paper Table 1 精确数值
Training data训练数据 SIM ↓WER (%) ↓PESQ ↑STOI ↑
Clean1.0000.004.641.00
Random noise0.9655.821.850.97
Gaussian noise0.9685.281.920.98
SceneGuard0.9452.772.220.99
Visible trade-off:可见权衡： the separate usability evaluation reports PESQ 2.034 (95% CI [1.840, 2.233]), below the paper’s stated ideal threshold of 3.0. SceneGuard preserves high measured intelligibility, but not pristine perceptual quality.单独的可用性评估报告 PESQ 为 2.034（95% CI [1.840, 2.233]），低于论文给出的理想阈值 3.0。SceneGuard 保持了较高的可懂度，但不是接近无损的感知音质。

Paper Tables 1–2 Table 1 reports WER 2.77% and PESQ 2.22; the separate usability summary reports WER 3.60% and PESQ 2.034. The paper does not document why the summaries differ, so they are shown separately.Table 1 报告 WER 2.77%、PESQ 2.22；单独的可用性摘要报告 WER 3.60%、PESQ 2.034。论文未说明两组摘要为何不同，因此分开展示。

Interactive ablation
交互消融

SNR trade-off explorer

SNR 权衡查看器

Move across the four precomputed settings to inspect the reported protection–usability balance.

在四个预计算结果之间切换，查看论文报告的保护与可用性权衡。

SNR ablation showing that lower SNR increases protection but reduces STOI and increases word error rate
Protection保护强度 5.5%
SIM ↓0.945
STOI ↑0.986
WER ↓3.6%

Visualization of paper-reported, precomputed ablation results—no model inference runs in the browser.这是论文报告的预计算消融可视化，浏览器中不运行模型推理。

Preprocessing robustness
预处理鲁棒性

Protection after common preprocessing

常见预处理后的保护效果

Horizontal bar chart of speaker similarity after MP3 compression, spectral subtraction, low-pass filtering, and downsampling

Paper Table 3 Lower SIM indicates greater speaker-identity degradation. The public repository does not contain the per-sample files needed for independent recomputation.更低的 SIM 表示说话人身份退化更强。公开仓库目前没有可供独立复算的逐样本文件。

Zero-shot evaluation
Zero-shot 评估

Protected reference audio lowers reported cloning similarity

受保护参考音频会降低报告的克隆相似度

Clean reference干净参考 SIM 0.618 Attack success rate 20.0%攻击成功率 20.0%
SceneGuard referenceSceneGuard 参考 SIM 0.588 Attack success rate 13.3%攻击成功率 13.3%

Paper Table 4 No publishable zero-shot synthesized audio or raw result file is present in the public repository, so this section reports metrics only.公开仓库没有可发布的 zero-shot 合成音频或原始结果文件，因此本节只报告指标。

Engineering view
工程细节

Implementation details

实现细节

Scene classifier场景分类器 PANNs CNN14, pretrained on AudioSet
Speaker encoder说话人编码器 ECAPA-TDNN, 192-dimensional embeddings
ASR evaluatorASR 评估器 Whisper Base
Referenced TTS architecture参考 TTS 架构 BERT-VITS2; no full fine-tuning in the workshop experimentBERT-VITS2；workshop 实验中未做完整微调
Optimizer优化器 Adam, learning rate 0.01
Default optimization默认优化设置 50 epochs, gradient clipping max norm 1.0
Default SNR默认 SNR 10–20 dB
Reported runtime报告运行时间 Approximately 10–15 s/sample on one RTX A6000单张 RTX A6000 上约 10–15 秒/样本
Speech / noise data语音 / 噪声数据 LibriTTS; TAU Urban Acoustic Scenes 2022
Noise library噪声库 Approximately 50,000 three-second clips across 10 scene categories约 50,000 条 3 秒片段，覆盖 10 类场景
Training-attack split训练攻击划分 100 training samples and 40 test samples100 条训练样本，40 条测试样本
Repository status:仓库现状： the public code contains the mixer, optimizer, metrics, and helper scripts, but does not currently include result CSVs, audio samples, checkpoints, or a complete PANNs / robustness / zero-shot reproduction pipeline.公开代码包含 mixer、优化器、指标与辅助脚本，但目前没有结果 CSV、音频样本、checkpoint，也没有完整的 PANNs / 鲁棒性 / zero-shot 复现流程。
Limitations
局限

Evidence boundary

证据边界

SceneGuard trades pristine audio quality for robust, audible protection (reported PESQ ≈ 2.03). The workshop evaluation uses speaker-embedding degradation as a proxy rather than full end-to-end TTS fine-tuning, and does not yet cover human listening studies or adaptive scene-aware attacks.SceneGuard 用可听、更鲁棒的保护换取不再追求接近无损的音质（报告 PESQ ≈ 2.03）。workshop 评估用说话人嵌入退化作为代理，而不是完整端到端 TTS 微调，也尚未覆盖听感实验或自适应场景感知攻击。
Reproducibility
可复现性

Public pipeline

公开流程

Build noise library构建噪声库 Assign scene labels分配场景标签 Generate protection生成保护 Evaluate评估
Source code源代码 Public implementation and scripts公开实现与脚本 Environment环境 Conda dependency specificationConda 依赖说明 Experiment scripts实验脚本 Preparation, defense, evaluation, and figures数据准备、防御生成、评估与作图

Evidence boundary Paper tables are the source for the numerical results on this page. Values such as negative final speaker similarity and very low SNR variance from the README are excluded because their dataset/configuration cannot be traced to public artifacts.本页数值以论文表格为准。README 中无法追溯到公开实验设置的结果（例如负的最终说话人相似度、极低 SNR 方差）未采用。