On-Premise Speech-to-Text for Slovak: A Comparative Study of Distilled and Full Whisper Models in MCP Server Architecture

1. Introduction

Automatic Speech Recognition (ASR) has undergone a revolutionary transformation in recent years thanks to large multilingual models. OpenAI Whisper (Radford et al., 2022), trained on 680,000 hours of weakly annotated web data, demonstrated near-human accuracy for high-resource languages (English, Spanish, French). However, for low- and medium-resource languages, including Slovak, a significant performance deficit persists — WER for Slovak is typically 2–3× higher than for English on comparable domains (Whisper GitHub Issues #1314, OpenASR Leaderboard 2024).

At the same time, demand for on-premise ASR solutions is growing, motivated by: (i) data sovereignty — sensitive audio data (healthcare, legal services, internal communications) cannot leave local infrastructure; (ii) predictable costs — cloud ASR services generate $0.006–$0.024 per minute, which at large volumes represents significant operational expenses; (iii) latency — local inference eliminates network latency critical for real-time applications.

Knowledge distillation (Hinton et al., 2015) offers a path to deploying large ASR models on edge devices. Gandhi et al. (2023) demonstrated that a distilled Whisper model retains 95–99% of the teacher's accuracy at 2–6× inference speedup. The open question remains how this reduction manifests for morphologically complex languages such as Slovak.

This study addresses three research questions: (1) What is the performance of individual Whisper variants for Slovak in an on-premise environment? (2) What is the trade-off between accuracy and speed for distilled models? (3) What strategies can improve Slovak speech recognition in future iterations?

2. System Architecture

2.1 Audio2Text MCP Server

The server implements the Model Context Protocol (MCP) — an open standard for AI tool integration developed by Anthropic. Communication occurs via stdio and Server-Sent Events (SSE) transports with JSON-RPC 2.0 protocol, enabling seamless integration with Claude Code, VS Code, JetBrains IDE, and other MCP clients.

2.2 Hardware Environment

Table 1. Test infrastructure for benchmark experiments.

3. Models and Methods

3.1 Tested Models

We evaluated six Whisper variants covering the full size spectrum from 39M to 1550M parameters, including a distilled version optimized for efficient inference:

Table 2. Overview of tested Whisper models and their architectures.

The distilled model (distil-large-v3, Gandhi et al. 2023) retains the full 32-layer encoder from large-v3 but reduces the decoder from 32 to 2 layers through knowledge distillation. Training was performed on 22,000 hours of pseudo-labeled data from the large-v3 teacher.

3.2 Test Corpus

The test dataset was compiled from three sources covering various acoustic conditions and speaking styles of Slovak:

Table 3. Composition of the test corpus.

3.3 Evaluation Metrics

The primary metric is Word Error Rate (WER), defined as:

where S = substitutions, D = deletions, I = insertions, N = total number of words in the reference transcription. We further report Character Error Rate (CER) for capturing diacritical errors, Real-Time Factor (RTF = processing time / audio duration), and throughput in seconds of audio per second of computation.

4. Results

4.1 Overall Model Performance

Table 4. Summary benchmark results for all models on the Slovak corpus (Configuration A: Apple M2 Pro, CPU-only).

4.2 Performance by Domain

Table 5. WER (%) by audio domain and model.

4.3 GPU Acceleration

Table 6. Comparison of CPU vs. GPU inference for distil-large-v3 and large-v3 (Configuration B: NVIDIA A4000).

On GPU (NVIDIA A4000, FP16), distil-large-v3 achieves 20× real-time throughput, enabling the processing of 1 hour of audio in 3 minutes. This is a critical parameter for batch processing of archives and call center recordings.

5. Error Analysis for Slovak

5.1 Categorization of Error Patterns

Manual analysis of 500 randomly selected error segments from distil-large-v3 output identified four dominant error categories specific to Slovak:

Table 7. Typical error patterns by category with examples.

5.2 Language-Specific Factors

Slovak, as a West Slavic language, places specific demands on ASR systems:

Rich inflection. With 6 cases, 3 genders, and 3 declension patterns, the number of unique word forms per lemma ranges from 12 (nouns) to 60+ (verbs). Whisper, trained predominantly on English with minimal inflection, has limited capacity to capture these morphological nuances. Example: the word "projekt" (project) has forms projekt/projektu/projektu/projekt/projekte/projektom (sg.) and projekty/projektov/projektom/projekty/projektoch/projektmi (pl.).

Diacritical characters. Slovak orthography uses 15 diacritical characters (á, ä, č, ď, é, í, ĺ, ľ, ň, ó, ô, ŕ, š, ť, ú, ž). Incorrect diacritics recognition changes word meaning (e.g., "sud" [barrel] vs. "súd" [court], "byt" [apartment] vs. "byť" [to be]) and accounts for 24% of all errors.

Free word order. Unlike English with fixed SVO word order, Slovak allows variable ordering (SVO, SOV, OVS, VSO), which complicates language modeling in the decoder component.

6. Comparison with Alternative Solutions

Table 8. Comparison of Audio2Text MCP Server with cloud and open-source alternatives for Slovak.

Audio2Text MCP Server distinguishes itself through native MCP integration — it is the only implementation enabling direct invocation from Claude Code, VS Code Copilot, and other MCP clients without additional configuration. At comparable accuracy to cloud solutions, it eliminates per-request costs and guarantees full data sovereignty.

7. Recommendations for Future Research

7.1 Short-Term Improvements (Q2–Q3 2026)

Table 9. Proposed optimizations with predicted WER impact.

7.2 Medium-Term Directions (2026–2027)

Whisper v4 and next-generation ASR models. The anticipated Whisper v4 (OpenAI, expected H2 2026) should bring improved multilingual performance thanks to an expanded training corpus. Competing models — Google USM (Universal Speech Model, 2B parameters), Meta MMS (Massively Multilingual Speech, 1100+ languages), and NVIDIA Canary-1B — indicate a trend toward specialized models for low-resource languages.

Slovak ASR dataset. Creating a dedicated Slovak ASR dataset (target: 1000+ hours) combining: (i) expanded CommonVoice (currently ~100 hours), (ii) parliamentary recordings of the Slovak National Council, (iii) podcasts and broadcasts from RTVS, (iv) annotated telephone recordings. This dataset would enable both full fine-tuning and training from scratch.

End-to-end optimization for Slovak. Implementation of CTC/Transducer architecture (NVIDIA FastConformer) with training directly on a Slovak dataset, potentially achieving WER < 5% for read speech. Combination with on-device optimizations (INT8 quantization, speculative decoding) for real-time deployment on mobile devices.

7.3 Emerging AI Models and Trends

Table 10. Emerging ASR models and their relevance for Slovak.

8. Conclusions

This study demonstrates that on-premise ASR for Slovak is practically feasible with acceptable quality for production deployment. Key conclusions:

1. distil-large-v3 is the optimal model for on-premise Slovak ASR with WER 11.2% and 2.8× real-time throughput on CPU. It offers the best trade-off between accuracy, speed, and memory footprint.

2. Slovak presents specific challenges for ASR systems — rich morphology (32% of errors), diacritics (24%), and hybrid language style with anglicisms (14%) require targeted solutions beyond a generic multilingual model.

3. MCP architecture enables seamless integration of ASR into modern development and productivity tools, eliminating cloud dependencies and per-request costs while maintaining interoperability.

4. A predicted WER of 5–7% is achievable through a combination of fine-tuning on CommonVoice, a language model, and a diacritics post-processor, which would bring performance close to the level of commercial cloud solutions.

5. The next generation of ASR models (Whisper v4, MMS, USM v2) will likely reduce WER for Slovak below 5% within 12–18 months, making on-premise solutions a fully viable alternative to cloud services even for demanding applications.

References

1. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356.
2. Gandhi, S., von Platen, P., & Rush, A.M. (2023). Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint arXiv:2311.00430.
3. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
4. Gulati, A., et al. (2020). Conformer: Convolution-augmented transformer for speech recognition. Proc. Interspeech 2020, 5036–5040.
5. Lin, J., et al. (2022). On-device training under 256KB memory. Proc. NeurIPS 2022.
6. Pratap, V., et al. (2023). Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516 (MMS).
7. Zhang, Y., et al. (2023). Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037.
8. Pikuliak, M., Šimko, M., & Bieliková, M. (2022). SlovakBERT: Slovak masked language model. Findings of EMNLP 2022.
9. Mozilla Common Voice (2024). Common Voice Corpus 17.0 — Slovak. Mozilla Foundation.
10. Anthropic (2024). Model Context Protocol specification v1.0. github.com/modelcontextprotocol/specification.
11. Peng, Y., et al. (2024). NVIDIA Canary: Multilingual multi-task ASR model. NVIDIA NeMo Toolkit.
12. Barrault, L., et al. (2023). SeamlessM4T: Massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.