Tag Archives: Transkribus

Detail of a screenshot of the Transkribus Web App with upper part of the title page of the 1678 Danish Ethica Complementoria print on the left-hand side and the transcription and annotation on the right-hand side.

Transcribing the 1st Danish Translation of the Ethica Complementoria and the Tranchierbuch from 1678

Guest article by Håvard Loeng, research assistant in the project “Training HTR-Models for a Bilingual Digital Edition of the Ethica Complementoria,” University of Oslo, 2023.

As a research assistant, my main task has been to provide a complete transcription of the Danish translation of the Ethica Complementoria and the appended Trencheerboc, published in 1678 in Copenhagen. Based on the copy held at the Gunnerus Library at the Norwegian University of Science and Technology, I have worked with a digitized version of the book in the automatic text recognition and transcription tool Transkribus, which made the process much easier. Transkribus enables automatic transcriptions of digitized material based on public AI models and lets you create models on your own. For the Ethica, we trained a model based on approximately 30 manually transcribed pages of the book and used the public AI model NorFraktur as a base model. The NorFraktur model was created by the National Library of Norway and is trained on 16th and 17th-century Danish and German prints, which fit well with the Ethica print. Our model achieved an estimated error rate of 2.5%, a satisfactory result. I ran our model on the entire book and began proofreading the transcription provided by the model. 

Having Norwegian as my first language was an advantage in the proofreading process. As the main written form of Norwegian (bokmål) is basically reformed Danish, modern written Danish poses few difficulties to Norwegian readers. 17th-century Danish, however, is more challenging. The spelling is somewhat recognizable, but it is not always consistent and may take some time to get used to. For example, the Norwegian word hode (English head) is in modern Danish spelled hoved, but in the Ethica, it is spelled hoffued, hofved, or hoffved. The vocabulary and phrasing ring archaic to modern ears, and the sentence structure is sometimes hard to follow. Both the historical Danish dictionary ODS (https://ordnet.dk/ods) and the online library of the National Library of Norway (https://www.nb.no/search) have been beneficial in cases of uncertainty, like finding occurrences of outdated words and phrases. 

The digitized version I worked with was of a copy of the Ethica located at the Gunnerus library at the Norwegian University of Science and Technology in Trondheim (https://ntnu.tind.io/record/149690). The print in this copy and the digitization are of good quality, making the work process reasonably smooth. Some instances of weak impressions and damage to the book were the main issues, as this reduced the legibility. Also, there are a few pages where some words and letters have disappeared in «the fold» when scanned. I could assume what was missing in most cases, but these pages must be checked with the physical book. 

In addition to the transcription, I have equipped the text with tags denoting structural elements, such as headings and page numbers, changes in fonts, etc. This is to prepare the text for the planned digital edition. 

Working with Ethica has been both challenging and fun, and I thank Annika for all the good support and advice along the way! 

Ethica Complementoria Digital Scholarly Edition – Revisit Pt. 1

Establishing Ground Truth

Yesterday and today, I spent some hours manually transcribing pages from the 1674 print of the Ethica Complementoria. The edition, printed in Copenhagen by Christian Wering and published by Wolf Lambrecht, survived in only one copy, held by the State and University Library Hamburg. The copy has been digitised a while ago and can be accessed and downloaded freely from their website. I have argued in my book about the print and transmission history of the Ethica Complementoria that this edition is the source for the Danish translation published in 1678 in Copenhagen. I’m preparing a bilingual digital scholarly edition of the German and the Danish Ethica.

However, with the print being digitised and available, you might wonder why I would manually transcribe it! Actually, I don’t plan to transcribe the entire print manually. I am using Transkribus to do the work for me. I have already established a transcription for the Danish print using the Transkribus app and the NorFraktur HTR model provided by the National Library of Norway. The performance was OK; however, there’s no way around a manual quality check when preparing a transcription for a scholarly edition. The NorFraktur model was trained on multilingual materials: 77 Danish-Norwegian and 23 German small blackletter prints from the mid-16th to the mid-17th century. With a CER of 2.0%, it’s quite good. However, with our print from the late 17th century in Danish, we found it underperformed a bit. We decided to improve the model by training it on manually transcribed pages from the digitised copy held by the Norwegian University of Science and Technology library. This improved the quality slightly, but a few issues remained. Since we were going for a quality check of the entire transcription anyway, we corrected all ‘misreadings’. A couple of pages are left; the first round of manual quality checks will be done by the end of September!

For the German print, we initially thought of using NorFraktur, too. However, there are now models for German print which perform significantly better than NorFraktur. Results will be significantly better. For what it’s worth, I will compare two approaches: Firstly, testing the quality of a German model, like the Transkribus Print Multi-Language model, with a CER of 1.6%. Secondly, I will train my model with manually transcribed pages to see if it can outperform the Transkribus Print model: for only this particular print.

Once I have the results, I will report back!

The image shows a visualisation of a useful filename structure for a scan of the title page of a rare book. It has four parts, divided by underscores. The first part is the library information using its standard abbreviation, the second part is the shelfmark, the third page indicates the page number or part of the document, and the fourth part contains additional information in a short form.

Ethica Complementoria Digital Scholarly Edition – Redux. Pt. 3: Filenames and Co-workers

Day 3 of my summer research leave

Filenames

After a bit of a hiccup yesterday, I’m back on track. It’s day 3 of my research leave, and I still haven’t gotten any of the “big tasks” started. If there is anything to be learned from previous iterations of this and other projects, it’s that without proper data management, the big tasks will never be finished. All of the available time in a side project like this will eventually be eaten by trying to understand where I left off last time, what needs to be done and where the files are. So the first step – in a way, the first “big task” – is to get an overview of what I have, where I left off, and what needs to be done to tackle the milestones.

A refreshing insight this morning was that I had already weeded out many superfluous files and manuscript versions of the book on the print and transmission history of the Ethica Complementoria. The most crucial documents are neatly stored in a public GitHub repository: the final version of the manuscript as published on the epub-server at the Herzog-August Library in Wolfenbüttel. The transcriptions of several prints in XML. Preliminary collation results and the code I used to create them. An ok README file describing the status quo and plans. One of these plans was to find a secure place for the documents which cannot be stored publicly, like scans I purchased for the publication and other admin stuff. I will put them in a private repository so I have access remotely.

While preparing the files for transfer into their new home, I witnessed a common problem of practical data management: Creating meaningful yet short and machine-friendly filenames! I used to teach data organisation and documentation in my previous job, and since then, I made it a daily practice only to use informative, machine-friendly filenames. For scans and digital photos of archival materials, manuscripts, or old books, I find it helpful to embed information about the source in the filename, e.g. which library holds this book? What is its shelf mark or reference number? What part of the book is the scan of? Is there any additional information relevant to see at first glance? With this in mind, and adhering to a few principles of good data management, like shallow directory structures, short names for files and folders, and no use of any special characters apart from underscore _ and hyphen – and ASCII letters and numbers. I made a little infographic about filenames for scans of archival materials and shared it with the researchers at the Dept. of Archaeology, Conservation, and History and my colleagues at the library. I will adopt this method now for the Ethica project and similar ones and rename all files accordingly. And add this info to the README so I might have a chance of remembering what I did and why next time!

The image shows a visualisation of a useful filename structure for a scan of the title page of a rare book. It has four parts, divided by underscores. The first part is the library information using its standard abbreviation, the second part is the shelfmark, the third page indicates the page number or part of the document, and the fourth part contains additional information in a short form.

Infographic depicting a filename structure for scans of archival materials.

Co-workers

The Ethica project is a single-person project for most of the time. I recently got funding for a research assistant to help me transcribe and prepare the bilingual digital scholarly edition of the Danish print of the Ethica from 1678. The assistant will mainly work on automatic text recognition, using Transkribus and building on the HTR model NorFraktur created by the National Library of Norway. NorFraktur is a multilingual model trained on printed Norwegian, Danish, and German texts from the early modern period. It performs quite well, and I hope to increase the performance slightly.

More interesting than the work, although, is collaborating with another researcher. Lots of the motivation behind proper data management and documentation stems from the need to work together on something, to share files and folders and workflows. The Ethica project is old, conceived in 2009, and has grown wildly since. So, for the following weeks to be a smooth and fruitful collaboration, I must create a usable data management plan (DMP) and set up a shared and version-controlled repository. And that’s why I had to go through all the “old stuff” – to know where things are, what is useful in this stage and which mistakes not to make again!

I will draft a DMP using DMPtool again – if I get lucky, I can do it today and publish it on this site, too. Update: the DMP is public and published here: https://dmphub.cdlib.org/dmps/doi:10.48321/D1RP93.

Training HTR-Models for a Bilingual Digital Edition of the Ethica Complementoria

In mid-June, I will start a small sub-project and finally get to work with the Ethica Complementoria again! I won a small grant from the University of Oslo’s “Teksthub+DH“, a network to support digital humanities focusing on language and textual sources. The network itself is part of a university-wide endeavour to strengthen IT in research. With the grant money, I was able to hire a research assistant. This is what we are going to do:

We aim to create a dataset to be used as the basis for a bilingual (Danish/German) digital scholarly edition of one of the most popular books on ‘etiquette’ in early modern Germany and Northern Europe: the Ethica Complementoria.

Originally written in German, the book made its way to the Nordic region through the Danish translation from 1678. This first Danish print will be published in parallel with the German version used for the translation.

The transcription project is part of a larger project on the book and revision history of the Ethica Complementoria, led by Annika Rockenberger and will be conducted by Håvard Loeng. An overview of all editions has been published digitally at the Herzog August library: http://diglib.hab.de/ebooks/ed000738/start.htm.

Manual transcription of two 300+ page texts is not feasible anymore. However, traditional Optical Character Recognition (OCR) yields inferior results for older printed books. Therefore, we aim to test, evaluate, improve, and build upon the NorFraktur model from the National Library of Norway. NorFraktur is a public Handwritten Text Recognition (HTR) model in Transkribus. It was trained on the HTR algorithm developed by READ Coop to recognise manuscripts and older prints automatically.

The development project contributes to both a digital scholarly edition with open access (planned as part of the publications by the Norwegian Language and Literature Society at bokselskap.no) and to the improvement and expansion of an open HTR model that the scholarly community can reuse for early modern prints in Norwegian (including Danish and German).