Tag Archives: HTR

Automated Text Recognition with ChatGPT 4

During a presentation for the Digital Meetup at the Department of Archaeology, Conservation and History at the University of Oslo yesterday, the historian Ian Milligan (University of Waterloo) mentioned briefly the stunning results he got when using ChatGPT for automated text recognition of historical documents. I don’t remember which materials he had used, but I was keen to try it out and see how the recognition performs compared to Transkribus public models‘ since this is my text recognition platform of choice.

There were a couple of obstacles, though. Firstly, I tried the University of Oslo’s ChatGPT, which is currently the GPT-3.5 Turbo and can process data that I, as a researcher employed by UiO, cannot share with third parties unless they are vetted and approved by the IT Department. This version, though, does not allow for the upload of images or accessing image sources via a URL. I tried my private, free subscription to Open AI’s ChatGPT and GPT-3.5 model and was equally unsuccessful. What was I doing wrong?

In my confusion, I turned to my social network on Mastodon to ask for advice: My colleague Christof Schöch from the University of Trier responded and pointed me towards a test he had performed with a historical image with some text on where ChatGPT was able to recognize text and transcribe it correctly. However, there was no info on which version was used or how this was done. I began doubting my ability to see an upload button or any information on how I would get an image into ChatGPT. A quick Google Search pointed me mainly toward business applications which used the Open AI API, not something I felt I wanted to dive into right now. Then, I guess it was on Reddit that someone pointed out that there’s an image upload function in ChatGPT-4 with the Plus subscription. Okay, so I first checked the UiO version of ChatGPT-4, where this feature was unavailable. The Open AI platform led me to a subscription, and I decided to try out Plus since it has access to Beta features and other specifications. 25$ later, finally, I had access. After some tweaking of settings, I managed to get the little image upload button to show up and voilà!

First test: Letter from Christopher Hansteen, 1828

I decided to go all in and test the automated text recognition abilities of ChatGPT-4 with a letter by Norwegian astronomer Christopher Hansteen to his wife.1 The letter is part of the collection of the private correspondence held by the National Library of Norway and can be seen and downloaded from their website. Following Christopher’s example, I only prompted ChatGPT with the following image and instructions:

Digital image of a letter written by Christopher Hansteen to his wife Johanne Christine Andrea Hansteen on October 18th, 1828.

Source: National Library of Norway, Brevs. 291 Brev fra Hansteen, Christopher (1784-1873) til Hansteen, Johanne Cathrine Andrea (1787-1840) | url = http://urn.nb.no/URN:NBN:no-nb_digimanus_175851

What text do you recognize in the image?

ChatGPT answered:

The image you’ve provided appears to be a handwritten manuscript or letter, dated “Tübingen 15 October 1818.” The writing is in cursive and may be in German, given the date and location mentioned. However, due to the complexity of the handwriting and the possibility of old Germanic script, I cannot provide a full and accurate transcription of the content.

If you need information or interpretation regarding specific parts of the document, please point them out, and I will do my best to assist you.

Hmm, so that was anticlimactic. I tried with a slightly better prompt:

The image that I provided is a letter from Christopher Hansteen, astronomer, to his wife. Please transcribe the first to lines of text, below the date.

This time, the response was more promising:

I apologize for the oversight earlier. Let me take another look.

From the image you provided and the context you’ve given, the first two lines below the date appear to read:

“Geliebte Christe! Ich schreibe Dir in aller Eile einen …”

However, it’s worth noting that handwritten manuscripts, especially in cursive and from historical periods, can be challenging to transcribe accurately without expertise in the specific handwriting style and language. If this transcription is crucial, I recommend consulting with a historian or archivist who specializes in 19th-century German manuscripts.

Still, I couldn’t get more out of it without investing in significantly better prompt engineering and better knowledge of the text at hand!

Second test: Ethica Complementoria, 1674

So, instead, I turned to more familiar materials: the version of the Ethica Complementoria print I currently work on, the German print from 1674, printed in Copenhagen. I have already transcribed and proofread the text several times and trained and improved an HTR model for it in Transkribus. Here, I could use the results from the latest version of my model and the validation set to see whether ChatGPT-4 performs worse/equally well/better than my model, ethica1674_v2.

The State and University Library Hamburg, Germany, provides the image. It is page 46 of the print and shows a combination of German (printed in Fraktur) and Latin (printed in Roman cursive type).

I uploaded the image and gave ChatGPT the following context and instructions:

Ok, let’s try another image. The image I provide is of a 1674 print of the Ethica Complementoria. It’s in early new high German. The font is a Fraktur. Please take your time and transcribe the first 10 lines of text. Here is the image:

ChatGPT provided the output – funnily enough – as code (i.e. in a code box, in plain text):

Certainly. Given the characteristics you mentioned (early new high German, Fraktur font), I’ll do my best to transcribe the first 10 lines from the image you provided:


und seyn will, hierinn nicht bewandert
wird er nicht unbillig veringert
wie jener Professor, welcher von einem
hochgelahrten Fürsten unterthänig
bat, die panaceam contra Peste ihm
gnädigst zu offenbahren; denn der selbige
Prinz ließ andeuten, daß hiero-
glyphicum ex Metamorphosi von Æsculapio
in Draconem converso, wie derselbe habe
die Pest curiret; also sollte es auch an-

Please note that while I’ve done my best to provide an accurate transcription, the nuances of early modern German and the Fraktur font can be tricky. It’s always a good idea to cross-check with an expert in historical German texts for a definitive transcription.

Okay, so that was impressive! There are a couple of mistakes, mainly small things like wrongly recognized characters and a couple of entirely made-up words, like “hochgelahrten” instead of “hocherleuchten” and the apparent normalization of historical typography to fit into a modern script, but impressive nonetheless.

I wondered whether I could improve its text recognition abilities by providing a good example of a properly-recognized text based on the image. I copied the recognized text from the Transkribus validation set (so it’s NOT 100% accurate, but almost) and gave it this prompt:

Look at your transcription and compare it to the transcription done with an expert tool, Transkribus. I have provided it below. Can you improve your transcription by learning from its transcription?

und ſeyn will / hierinn nicht bewandert
iſt / wird er nicht unbillich verringert /
wie jener Profeſſor, welcher von einem
hocherleuchten Fuͤrſten unterthånig
bate / die Panaceam contra Peſtem ihm
gnådigſt zu offenbahren / dem der loͤb-
liche Printz ließ andeuten / das hiero-
glyphicum ex Metamorphoſi von Æſculapio
in Draconem converſo, wie derſelbe habe
die Peſt curiret / alſo ſolte er es auch an-

ChatGPT delivered and gave the following output:

Given the expert transcription from Transkribus, here’s an improved version of the first 10 lines:

und ſeyn will / hierinn nicht bewandert
iſt / wird er nicht unbillich verringert /
wie jener Profeſſor, welcher von einem
hocherleuchten Fuͤrſten unterthånig
bate / die Panaceam contra Peſtem ihm
gnådigſt zu offenbahren / dem der loͤb-
liche Printz ließ andeuten / das hiero-
glyphicum ex Metamorphoſi von Æſculapio
in Draconem converſo, wie derſelbe habe
die Peſt curiret / alſo ſolte er es auch an-

Thank you for sharing the transcription from Transkribus. It’s always beneficial to compare and refine transcriptions for historical documents.

I realized my prompt was a bit misleading, so I reformulated what I actually wanted it to do:

Now that you know how an expert transcription looks like, can you transcribe lines 10 to 20 from the previous image in the same way as the expert?

Obediently, ChatGPT delivered:

Certainly. Using the provided expert transcription as a guide, I’ll transcribe lines 10 to 20 from the image:

ſchlagen / das iſt : Er ſolte in ſeiner
Kunſt wohl geuͤbt und erfahren ſeyn/
ſeilig ſtudiren / dabei ſtilſchweigen und
wohl nachdencken / ſapienti ſat dicium. Sie
lehren / iſt gut predigen. Es giebet
viel / mancher fuͤr einen groſſen Mann
auſſ / mit prächtigen Kleidern/ und hoch
trabenden Worten herprangen/ und
iſt doch ein muͤßiggänger Ignorant: phi-
loſophus nihil debet eſſe, qua eſſe ipſum

This transcription takes into account the nuances of the Fraktur font and the early new high German language.

Now I was really impressed! This is a stunningly accurate transcription of the page, especially given the change of font and language in the short passage. It has neatly applied the special characters and punctuation, too.

In my excitement, I posted a thread of my experiment to Mastodon, which, I guess, is the closest I can get to “going viral” (at the point of writing, I have 27 boosts/replies) given my network and reach. Christopher Pollin suggested chaining the text recognition with a transformation into XML/TEI like he has done using proper prompt engineering and thus creating a workflow from image to marked-up text using ChatGPT-4. I haven’t tried this part yet, but I will investigate it.

In Conclusion

Unexpectedly, ChatGPT-4 performed on a task I did not anticipate being able to do even, quite outstandingly. I guess that with sound prompt engineering, we can achieve a high enough performance for most use cases: searches in historical documents. It will, additionally, make the work of the textual scholar easier by taking away time-consuming and rather dull tasks like manual transcription and encoding. It is, however, questionable how it scales up. I used one page of text and let it transcribe ten lines each time.

I will contact the University of Oslo ChatGPT team and ask if and when they plan on integrating the image upload function into their service. This could be an option for researchers who need historical documents transcribed and post-processed (like: translated) quickly, without needing to set up specialized software or training HTR models.

1 Note: ChatGPTs transcription and subsequent attempt to read the letter are completely bollocks. The first line reads “Tobolsk 18. Oktober 1828”. The text is not written in German, but in Dano-Norwegian and says: “Tak mangfoldige Gange for Dine 2 trøstende kjerlige Breve. Ja vist har jeg været mere uroelig, end jeg har ladet mig mærke med;” Cited after the transcription by Kari Høgvold, accessible on the University of Oslo Library’s website as a PDF – What ChatGPT probably does here is that it assumes a letter written by a husband to his wife in that period of time will have a beginning like “Geliebte Christe! Ich schreibe Dir in aller Eile einen […]”. By the way: Hansteen’s wife’s name is Johanne Cathrine Andrea.

Detail of a screenshot of the Transkribus Web App with upper part of the title page of the 1678 Danish Ethica Complementoria print on the left-hand side and the transcription and annotation on the right-hand side.

Transcribing the 1st Danish Translation of the Ethica Complementoria and the Tranchierbuch from 1678

Guest article by Håvard Loeng, research assistant in the project “Training HTR-Models for a Bilingual Digital Edition of the Ethica Complementoria,” University of Oslo, 2023.

As a research assistant, my main task has been to provide a complete transcription of the Danish translation of the Ethica Complementoria and the appended Trencheerboc, published in 1678 in Copenhagen. Based on the copy held at the Gunnerus Library at the Norwegian University of Science and Technology, I have worked with a digitized version of the book in the automatic text recognition and transcription tool Transkribus, which made the process much easier. Transkribus enables automatic transcriptions of digitized material based on public AI models and lets you create models on your own. For the Ethica, we trained a model based on approximately 30 manually transcribed pages of the book and used the public AI model NorFraktur as a base model. The NorFraktur model was created by the National Library of Norway and is trained on 16th and 17th-century Danish and German prints, which fit well with the Ethica print. Our model achieved an estimated error rate of 2.5%, a satisfactory result. I ran our model on the entire book and began proofreading the transcription provided by the model. 

Having Norwegian as my first language was an advantage in the proofreading process. As the main written form of Norwegian (bokmål) is basically reformed Danish, modern written Danish poses few difficulties to Norwegian readers. 17th-century Danish, however, is more challenging. The spelling is somewhat recognizable, but it is not always consistent and may take some time to get used to. For example, the Norwegian word hode (English head) is in modern Danish spelled hoved, but in the Ethica, it is spelled hoffued, hofved, or hoffved. The vocabulary and phrasing ring archaic to modern ears, and the sentence structure is sometimes hard to follow. Both the historical Danish dictionary ODS (https://ordnet.dk/ods) and the online library of the National Library of Norway (https://www.nb.no/search) have been beneficial in cases of uncertainty, like finding occurrences of outdated words and phrases. 

The digitized version I worked with was of a copy of the Ethica located at the Gunnerus library at the Norwegian University of Science and Technology in Trondheim (https://ntnu.tind.io/record/149690). The print in this copy and the digitization are of good quality, making the work process reasonably smooth. Some instances of weak impressions and damage to the book were the main issues, as this reduced the legibility. Also, there are a few pages where some words and letters have disappeared in «the fold» when scanned. I could assume what was missing in most cases, but these pages must be checked with the physical book. 

In addition to the transcription, I have equipped the text with tags denoting structural elements, such as headings and page numbers, changes in fonts, etc. This is to prepare the text for the planned digital edition. 

Working with Ethica has been both challenging and fun, and I thank Annika for all the good support and advice along the way! 

Ethica Complementoria Digital Scholarly Edition – Revisit Pt. 1

Establishing Ground Truth

Yesterday and today, I spent some hours manually transcribing pages from the 1674 print of the Ethica Complementoria. The edition, printed in Copenhagen by Christian Wering and published by Wolf Lambrecht, survived in only one copy, held by the State and University Library Hamburg. The copy has been digitised a while ago and can be accessed and downloaded freely from their website. I have argued in my book about the print and transmission history of the Ethica Complementoria that this edition is the source for the Danish translation published in 1678 in Copenhagen. I’m preparing a bilingual digital scholarly edition of the German and the Danish Ethica.

However, with the print being digitised and available, you might wonder why I would manually transcribe it! Actually, I don’t plan to transcribe the entire print manually. I am using Transkribus to do the work for me. I have already established a transcription for the Danish print using the Transkribus app and the NorFraktur HTR model provided by the National Library of Norway. The performance was OK; however, there’s no way around a manual quality check when preparing a transcription for a scholarly edition. The NorFraktur model was trained on multilingual materials: 77 Danish-Norwegian and 23 German small blackletter prints from the mid-16th to the mid-17th century. With a CER of 2.0%, it’s quite good. However, with our print from the late 17th century in Danish, we found it underperformed a bit. We decided to improve the model by training it on manually transcribed pages from the digitised copy held by the Norwegian University of Science and Technology library. This improved the quality slightly, but a few issues remained. Since we were going for a quality check of the entire transcription anyway, we corrected all ‘misreadings’. A couple of pages are left; the first round of manual quality checks will be done by the end of September!

For the German print, we initially thought of using NorFraktur, too. However, there are now models for German print which perform significantly better than NorFraktur. Results will be significantly better. For what it’s worth, I will compare two approaches: Firstly, testing the quality of a German model, like the Transkribus Print Multi-Language model, with a CER of 1.6%. Secondly, I will train my model with manually transcribed pages to see if it can outperform the Transkribus Print model: for only this particular print.

Once I have the results, I will report back!

Training HTR-Models for a Bilingual Digital Edition of the Ethica Complementoria

In mid-June, I will start a small sub-project and finally get to work with the Ethica Complementoria again! I won a small grant from the University of Oslo’s “Teksthub+DH“, a network to support digital humanities focusing on language and textual sources. The network itself is part of a university-wide endeavour to strengthen IT in research. With the grant money, I was able to hire a research assistant. This is what we are going to do:

We aim to create a dataset to be used as the basis for a bilingual (Danish/German) digital scholarly edition of one of the most popular books on ‘etiquette’ in early modern Germany and Northern Europe: the Ethica Complementoria.

Originally written in German, the book made its way to the Nordic region through the Danish translation from 1678. This first Danish print will be published in parallel with the German version used for the translation.

The transcription project is part of a larger project on the book and revision history of the Ethica Complementoria, led by Annika Rockenberger and will be conducted by Håvard Loeng. An overview of all editions has been published digitally at the Herzog August library: http://diglib.hab.de/ebooks/ed000738/start.htm.

Manual transcription of two 300+ page texts is not feasible anymore. However, traditional Optical Character Recognition (OCR) yields inferior results for older printed books. Therefore, we aim to test, evaluate, improve, and build upon the NorFraktur model from the National Library of Norway. NorFraktur is a public Handwritten Text Recognition (HTR) model in Transkribus. It was trained on the HTR algorithm developed by READ Coop to recognise manuscripts and older prints automatically.

The development project contributes to both a digital scholarly edition with open access (planned as part of the publications by the Norwegian Language and Literature Society at bokselskap.no) and to the improvement and expansion of an open HTR model that the scholarly community can reuse for early modern prints in Norwegian (including Danish and German).