MathReader: Text-to-Speech for Mathematical Documents

ICASSP 2025 (Oral)

Sieun Hyeon1, Kyudan Jung2, Nam-Joon Kim1, Hyun Gon Ryu3, Jaeyoung Do1,4
1Department of Electrical and Computer Engineering, Seoul National University,
2Department of Mathematics, Chung-Ang University,
3NVIDIA,
4Interdisciplinary Program in Artificial Intelligence, Seoul National University
Paper arXiv Code Demo
subtitle of the main image

This figure shows an example of a TTS document reader skipping a formula from a document. (1) is a part of the document we used for testing.Because the document is old and of low quality, Microsoft Edge skips reading it aloud (See (2)). However, MathReader reads this correctly (See (3)).

Abstract

TTS (Text-to-Speech) document reader from Microsoft, Adobe, Apple, and OpenAI have been serviced worldwide. They provide relatively good TTS results for general plain text, but sometimes skip contents or provide unsatisfactory results for mathematical expressions. This is because most modern academic papers are written in LaTeX, and when LaTeX formulas are compiled, they are rendered as distinctive text forms within the document. However, traditional TTS document readers output only the text as it is recognized, without considering the mathematical meaning of the formulas. To address this issue, we propose MathReader, which effectively integrates OCR, a fine-tuned T5 model, and TTS. MathReader demonstrated a lower Word Error Rate (WER) than existing TTS document readers, such as Microsoft Edge and Adobe Acrobat, when processing documents containing mathematical formulas. MathReader reduced the WER from 0.510 to 0.281 compared to Microsoft Edge, and from 0.617 to 0.281 compared to Adobe Acrobat. This will significantly contribute to alleviating the inconvenience faced by users who want to listen to documents, especially those who are visually impaired.

Method

MathReader is designed as a five-stage pipeline to accurately convert PDF documents containing mathematical expressions into speech:

1. OCR (Optical Character Recognition)
- Used Nougat-small, based on a hierarchical vision transformer, to convert PDF files into mmd (markup) files.
- These files contain both regular text and LaTeX formulas.
- Since TTS models cannot directly read LaTeX (due to special characters), further processing is required.

2. Extract Formulas
- Nougat marks LaTeX formulas with special symbols (e.g., `\[ ... \]` or `\( ... \)`), making it possible to identify and separate all formulas from the text.

3. Fine-tuned T5 (LaTeX → Spoken English Translation)
- T5-small was fine-tuned on existing (LaTeX, spoken English) datasets.
- Treats formula reading as a translation task, converting mathematical LaTeX into natural spoken English.
- T5-small was chosen for speed and efficiency, while still producing good quality results.

4. Replace LaTeX with Spoken English
- The translated formulas are inserted back into the mmd file, replacing the LaTeX code with readable English text.

5. TTS (Text-to-Speech)
- The final processed text, now free of LaTeX, is passed to VITS, a TTS model.
- This generates speech without errors or skipped formulas.

Diagram illustrating the method used in the research

Our pipeline for reading the document correctly. The method integrates OCR (Nougat-small) + Formula Extraction + LaTeX Translation (T5-small) + Replacement + TTS (VITS) into a seamless system that reads both text and mathematical expressions aloud accurately.

Diagram illustrating the method used in the research

The pipeline can be summarized as : Input PDF → OCR → Extract LaTeX → Translate via T5 → Replace with spoken English → TTS output.

Experiment Results

Diagram illustrating the method used in the research

BibTeX


@INPROCEEDINGS{HyeonICASSP25,
    author={Hyeon, Sieun and Jung, Kyudan and Kim, Nam-Joon and Ryu, Hyun Gon and Do, Jaeyoung},
    booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
    title={MathReader : Text-to-Speech for Mathematical Documents}, 
    year={2025},
    pages={1-5},
    keywords={Text recognition;Error analysis;Pipelines;Optical character recognition;Graphics processing units;Signal processing;Real-time systems;Mathematical models;Text to speech;Speech processing;OCR;T5;TTS;document reader;LaTeX},
    doi={10.1109/ICASSP49660.2025.10890531}
  }