We recently worked on extending VoiceWukong, a benchmark for deepfake voice detection, to Hindi.

VoiceWukong evaluates deepfake voice detectors mainly on English and Chinese. One of the gaps we noticed was cross-language evaluation, especially for languages like Hindi.

So we decided to build a small Hindi dataset and run our own deepfake voice detection experiment.

The approach

The overall pipeline was:

flowchart TB
    A[Real Hindi Speech] --> B[Generate Synthetic Speech]
    B --> C[Real + Fake Dataset]
    C --> D[Train WavLM Detector]
    D --> E[Evaluate]

    A ~~~ D
    B ~~~ E

We started with real Hindi speech, generated synthetic versions using XTTS-v2, combined the two and trained a detector to distinguish between real and generated speech.

Building the dataset

We used the SPRINGLab/IndicTTS-Hindi dataset as the source of real Hindi speech.

We collected 500 real Hindi voice samples. We also kept 8 separate real voice clips as reference samples for XTTS-v2.

The reference clips were kept separate from the samples used for detector evaluation.

stream = load_dataset(
    "SPRINGLab/IndicTTS-Hindi",
    split="train",
    streaming=True
)

stream = stream.shuffle(
    seed=SEED,
    buffer_size=4000
)

We converted the audio to mono and resampled it to 16 kHz for the detector.

def to_mono_16k(wav_array, sr_in):
    x = np.asarray(
        wav_array,
        dtype=np.float32
    )

    if x.ndim > 1:
        x = x.mean(axis=0)

    if sr_in != 16000:
        x = librosa.resample(
            x,
            orig_sr=sr_in,
            target_sr=16000
        )

    return x

This gave us:

  • 500 real Hindi clips
  • 8 separate reference clips

Generating the deepfake voices

Next, we used XTTS-v2 to generate synthetic Hindi speech.

The reference clips were used for voice cloning, while the text came from the real Hindi samples.

flowchart LR
    A[Hindi Transcript] --> C[XTTS-v2]
    B[Reference Voice] --> C
    C --> D[Synthetic Hindi Speech]

The generation step was:

def xtts_generate(text, speaker_wav):
    wav = tts.tts(
        text=text,
        speaker_wav=speaker_wav,
        language="hi"
    )

    wav = np.asarray(
        wav,
        dtype=np.float32
    ).squeeze()

    return wav

We then resampled the generated audio from XTTS-v2's 24 kHz output to 16 kHz.

wav24 = xtts_generate(
    text,
    reference_voice
)

wav16 = librosa.resample(
    wav24,
    orig_sr=24000,
    target_sr=16000
)

sf.write(
    output_path,
    wav16,
    16000
)

We generated 500 fake Hindi clips, giving us 1,000 clips in total.

500 real
500 fake
---------
1000 total

The same transcripts were used for the real and generated samples, giving us real and fake samples based on the same speech content.

Preparing the dataset

We combined the real and generated samples and created a stratified train, validation and test split.

flowchart LR
    A[1000 Clips] --> B[700 Train]
    A --> C[100 Validation]
    A --> D[200 Test]

The final split was:

  • 700 training clips
  • 100 validation clips
  • 200 test clips

The dataset remained balanced between real and fake samples.

train_df, test_df = train_test_split(
    data,
    test_size=0.20,
    stratify=data["label"],
    random_state=SEED
)

train_df, val_df = train_test_split(
    train_df,
    test_size=0.10 / 0.80,
    stratify=train_df["label"],
    random_state=SEED
)

We also standardized the audio length to 6 seconds, padding shorter clips and cropping longer clips.

Building the detector

For detection, we used microsoft/wavlm-base-plus as the speech representation model.

We froze the WavLM backbone and added an attentive statistics pooling layer followed by a small classification head.

flowchart TD
    A[Audio] --> B[WavLM Base Plus]
    B --> C[Attentive Statistics Pooling]
    C --> D[MLP Classifier]
    D --> E[Real / Fake]

The attentive pooling layer learns which parts of the audio representation are more useful for classification.

class AttentiveStatsPool(nn.Module):
    def __init__(self, dim):
        super().__init__()

        self.attn = nn.Sequential(
            nn.Linear(dim, 128),
            nn.Tanh(),
            nn.Linear(128, 1)
        )

    def forward(self, x):
        w = torch.softmax(
            self.attn(x),
            dim=1
        )

        mean = torch.sum(
            w * x,
            dim=1
        )

        var = torch.sum(
            w * (x - mean.unsqueeze(1)) ** 2,
            dim=1
        )

        std = torch.sqrt(
            var.clamp(min=1e-8)
        )

        return torch.cat(
            [mean, std],
            dim=-1
        )

The classification head was a small MLP:

self.head = nn.Sequential(
    nn.Linear(2 * D, 256),
    nn.ReLU(),
    nn.Dropout(0.3),

    nn.Linear(256, 64),
    nn.ReLU(),
    nn.Dropout(0.3),

    nn.Linear(64, 1)
)

With the WavLM backbone frozen, only around 0.51M parameters were trainable.

Detector on cuda
Trainable parameters: 0.51M
Backbone frozen: True

Training

We trained the classification head using binary cross-entropy with logits.

criterion = nn.BCEWithLogitsLoss(
    pos_weight=torch.tensor(
        [neg / pos],
        dtype=torch.float32,
        device=DEVICE
    )
)

loss = criterion(
    model(wav),
    y
)

loss.backward()

torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    5.0
)

opt.step()

We used validation EER to select the best checkpoint.

The best validation checkpoint reached 0% EER during training.

We then restored that checkpoint and evaluated it only on the held-out test set.

Results

The final evaluation was done on 200 previously unseen Hindi clips.

Metric Result
Equal Error Rate (EER) 1.00%
Accuracy 98.50%
F1 Score 0.9851
ROC-AUC 0.9997

The confusion matrix was:

Predicted Real Predicted Fake
Actual Real 98 2
Actual Fake 1 99

That means 197 out of 200 test samples were classified correctly.

The evaluation code was:

vy, vp = collect_scores(val_dl)

val_eer, threshold = compute_eer(
    vy,
    vp
)

ty, tp = collect_scores(test_dl)

test_eer, _ = compute_eer(
    ty,
    tp
)

pred = (
    tp >= threshold
).astype(int)

accuracy = accuracy_score(
    ty,
    pred
)

f1 = f1_score(
    ty,
    pred
)

auc = roc_auc_score(
    ty,
    tp
)

The final output was:

EER      : 1.00%
Accuracy : 98.50%
F1       : 0.9851
ROC-AUC  : 0.9997

How does this compare with VoiceWukong?

The original VoiceWukong benchmark evaluated 12 deepfake voice detectors on large English and Chinese datasets.

AASIST2 was the strongest detector in their benchmark, with an EER of 13.50% on English and 13.54% on Chinese.

VoiceWukong also tested several types of audio manipulation, including:

  • Noise
  • Replay
  • Resampling
  • Time stretching
  • Volume changes
  • Fade in/out

These manipulations could significantly reduce detector performance.

This puts our result into perspective.

Our 1.00% EER was measured on a relatively small Hindi dataset where the synthetic speech was generated using a single model, XTTS-v2.

It is therefore not a direct comparison with the much larger VoiceWukong benchmark.

The two experiments are measuring different things.

What we learned

Our Hindi experiment was small and focused on one voice generation model, so there are clear limitations.

Still, it gave us a starting point for studying deepfake voice detection in Hindi and exploring how these systems behave outside the languages commonly used in existing benchmarks.

There are several things we would want to test next:

  • Add more Hindi speakers and samples
  • Generate voices using multiple TTS and voice conversion models
  • Test cross-generator performance
  • Add noise, replay, resampling and other audio manipulations
  • Compare Hindi performance with English and Chinese
  • Test shorter audio clips
  • Evaluate whether the detector generalizes across speakers and accents

The next stage would look something like this:

flowchart TB
    A[Current Hindi Experiment] --> B[More Diverse Data]
    B --> C[Multiple Voice Generators]
    C --> D[Real-world Audio Conditions]

    A ~~~ C
    B ~~~ D

Then, from there:

flowchart TD
    A[Real-world Audio Conditions] --> B[Cross-language Evaluation]
    B --> C[Robust Deepfake Detection]

Our experiment was a small step toward studying these questions for Hindi.

Voice generation is getting better quickly.

The next challenge is making sure that detection systems can keep up, not just for English and Chinese, but across languages.

Contributors: Abhippsa Bhanja Deo, Anshuman Yadav, Antaryami Sahu, Pratik Tiwari, Rakesh Kumar Panda

Base paper: VoiceWukong: Benchmarking Deepfake Voice Detection