Reading Lisan al-Dawat with 4.87% character error

We got the best first reading of Lisan al-Dawat down to 4.87% character error on our three-page benchmark. GPT-6 Astra, with its thinking time set to high, came first. Claude Fable 5.1 was close at 4.93% and cost less than half as much per page.

For Alishraaq, that gives us a useful starting point: a draft that a reader can check against the book, with tools to catch recurring mistakes along the way.

Why we’re doing this

I’m building Alishraaq to make our community’s books and writing easier to read, search and study. Much of that material lives in scanned pages. You can look at a page, but searching for a phrase across a collection requires text the computer can actually read.

Lisan al-Dawat makes this work unusually demanding. It uses an Arabic-derived script for a language with Gujarati roots, and a page may move between Lisan al-Dawat and Arabic, with an English phrase in the middle. Small vowel marks and honorifics sit close to the letters. A model can produce a fluent sentence while changing what the page says.

We needed to know how much of the transcription a model could handle, what it would get wrong, and how much work would remain for someone who knows the language.

The full page was too much at once

Our early full-page attempts left out text and struggled with tiny marks. On one Sahifa test, GPT-5.5 returned just 432 characters against a reviewed transcription of 1,170. We were also running into the output limit. Sending the whole page in one go wasn’t giving us a usable reading.

Image size was part of the problem. DPI means dots per inch: it describes how densely a page is rendered. Pixels are the actual little squares that make up the image. A higher DPI setting can preserve more detail from a good source, but it cannot recover detail already lost in a blurry scan. One of our source scans was around 150 pixels per inch. Enlarging it made the blur bigger.

There was another place to lose detail. A model may shrink a tall page to fit its image limits. A tiny mark that looks clear when we zoom in on a monitor can become only a few pixels by the time the model reads it.

We split the page into three overlapping horizontal bands and sent them in reading order. Each band gave the text more room in the image. The overlap kept words near a cut visible in the next band, and we instructed the model to transcribe repeated lines only once. This helped with the small marks that full-page views were losing. An already blurred word still needed a better source or a human decision.

How we set up the test

We chose one page from each of three kinds of community writing: a sermon booklet, a historical reference book and a guide to worship. Here is exactly what we used:

• Al-Iqtibasaat al-Nooraniyah, Majlis 6 — Karachi, 1446H (2024). Iqtibasaat are printed selections from religious discourses. This booklet belongs to Ashara Mubaraka, the annual Muharram gatherings in which the community remembers Imam Husain and Karbala. We used printed page 9 of the sixth booklet. Its flowing Lisan al-Dawat prose moves into Arabic quotations and English phrases such as “peace of mind,” giving us a test of reading several languages on the same line.
• Tohfat Laali Akhbar al-Hudat. This is a historical and biographical guide to the Dawoodi Bohra community’s religious leaders and learned figures in India, organised by town and city. It records names, family relationships, important dates and burial places. The work dates to 1414H; our scanned copy is the second edition, dated 1433H. We used printed page 79, in the Burhanpur section. Its names, honorifics and dates make small transcription errors particularly consequential.
• Nayyir Sahifat al-Salaat wal-Ibaadaat, Part 1 — the Busaheba Sahifa. Often shortened to Sahifa al-Salaat wal-Ibaadaat, this is a practical guide to prayer, worship and religious observances, with Arabic texts and explanations in Lisan al-Dawat. Our copy is dated 1420H. We used printed page 170, which discusses fasting during travel in Ramadan. Its red headings, bullet points and footnote test whether the model can preserve the structure of detailed instructions.

Together, these pages gave us different print styles and reading problems, including mixed languages, dates and small marks. Every model got the same page images and instructions explaining Lisan al-Dawat and its common reading traps. We asked for a faithful transcription, with uncertain readings marked for review.

A small program sent the pages to the models, saved their responses, and recorded the time and model-call cost. We compared the text with our reviewed reference. We also repeated Tohfat Lali three times to see how much a model’s answer changed, and tested a second reading in which the model checked its draft against the images.

The main score is character error rate: how many additions, deletions or replacements are needed to make the transcription match the reference. At 4.87%, that is roughly five corrections for every hundred reference characters. Spacing and small marks count too. We also measured errors at the word level, because a page can look close while containing many words that still need attention.

Three pages give us a way to choose the next model to try on our books. They are a small sample, and the ranking may change on other pages.

Checking that the models were doing the reading

We also went back through saved responses, available agent traces and run records to look for shortcuts: a model referring to an answer it had already seen, claiming to look something up, or asking another model to do the work. The test runner provided images and instructions without giving the models browsing tools or access to the reference transcription.

The traces helped us understand a subtler problem too. A model sometimes used its knowledge of a name or historical event to settle an ambiguous reading. That can produce plausible text while moving away from the printed letters. It is one reason the scan stays beside the transcription during review.

This check has limits. Some providers expose only summaries or hide the reasoning entirely, and our reference text evolved during development. We cannot certify that every run was free of contamination. The results below describe the retained comparison and guide our next production trial.

What we learned

The second reading was useful often enough to keep experimenting with it. It also made some answers worse. Gemini Flash improved from 6.33% to 5.65% on the matched three-page comparison. Opus went from 5.54% to 6.52%. We need to look at what changed before calling a page finished.

Try a page yourself

Choose two models, zoom into the source, or switch between the raw reading, automatic cleanup and the model’s recheck. “Automatic cleanup” opens a real before-and-after example. You can also explore how scores and costs changed when we gave the models more time to think.

Which model would I use?

For our next batch, my practical order is:

1. Claude Fable 5.1, high, as the default. Its 4.93% first-reading error was close to Astra’s result, at about $0.23 per page. It offers the balance I would try first.
2. Gemini 3.7 Flash for a large, cost-sensitive batch. It reached 6.33% at about 1.4 cents per page. That leaves more correction work, but makes it inexpensive to generate drafts. Its first reading plus verification cost about 2.7 cents per page in this test.
3. GPT-6 Astra, high, when the lowest measured error matters most. It won at 4.87%, at about $0.58 per page. I would use it on difficult material or when reducing the initial error burden is worth the extra spend.

That order is my judgment about usefulness for Alishraaq. The table below ranks the models strictly by first-reading error, using the best complete setting we tested for each model.

Getting from 4.87% error to production-quality text

A first reading still needs someone who can read the book. We built checks around the mistakes that kept coming back, so that person has a useful list of places to inspect.

The ye, te, ne catcher

Small words such as ye, te and ne turn up throughout Lisan al-Dawat. In the print we were working from, the final ه could be read as ر. That gave us ير where we expected يه, and the same mistake in other particles. Every character was valid, so a check for unexpected characters let these readings through.

The catcher looks for those spellings and checks the words around them. For example, تر بعد gets a suggestion to read ته بعد. A نر before a verb can be a misreading of the negative particle نه. These are examples from our Sahifa checking rules:

A blanket replacement would damage valid words. نر can mean “male,” and تر has legitimate comparative uses. We keep explicit exceptions for those contexts. The checker records the page and line, along with its reason and suggested reading, and leaves the transcription untouched for review.

Do-chashmi he

Do-chashmi he, ھ, needs its own check. It belongs in words such as چھے and پڑھي. Our early OCR traps included چه where the source required چھے. Turning every heh-shaped letter into the ordinary ه would erase that distinction.

The text checker flags do-chashmi in an unlikely position, using the preceding letter and exceptions such as a Hijri year marker. Finding a missing do-chashmi needs another look at the print: چه contains perfectly valid letters. We made checking چھے an explicit instruction in the verification pass because a model could otherwise skim past it again. The spelling is word-specific; ڈهور, for example, uses ordinary heh in our transcription rules.

The small marks and word endings

Dagger alif, or khara zabar, also caused trouble. In اٰپ, the small vertical mark was sometimes read as a pesh, giving اُپ. Our fixer has a rule for that specific word. It also handles the known اُنے reading of انے. Those rules came from earlier OCR work; neither fired in this three-page benchmark’s saved first readings.

We also check for a lost final ں and confusions such as ني becoming في when we re-read the images. An unusual word can be compared with our existing Lisan al-Dawat collection to help find likely alternatives. Frequency gives the reviewer another clue, and the scan still decides the reading.

Taking a page through review

Before these checks, a mechanical cleanup pass makes the character encoding consistent. Some of those substitutions look almost identical on screen. In this benchmark, they left all five of Astra high’s first readings unchanged, so its 4.87% character error stayed at 4.87% after automatic cleanup.

The wider checks give us places to spend our attention. On an earlier 662-page Sahifa packet, the language checker produced 1,618 findings, including 108 ye flags and 522 contextual te flags. It applied zero automatic edits. We also check for stray page numbers and missing spaces, and test whether the font can display the text.

The model gets the scan and draft together for another reading. A human reviewer can then open the page beside the transcription, resolve the flagged words and check names and small honorific marks. Approved text is assembled into paragraphs that flow on a phone, while keeping its source-page reference.

This benchmark measures first readings and model verification. We have not measured a final error rate after human review.

The next step for Alishraaq is to put the strongest candidates through a larger batch and measure the correction time. That is the work that will tell us how quickly we can make the next book searchable.