How to Train a Grounded OCR Model: GutenOCR (2026)

What GutenOCR is and why we built it

Jessica Carbino, BevayaHunter, Ben, and Yosheb, the floor is yours.

Hunter Heidenreich, BevayaAwesome. Thank you so much, Jessica. I'm very excited to tell you guys a bit about what we built with GutenOCR. let's jump in.

So, today we're talking about GutenOCR. And what we have on the right here is a little radar diagram profile of how our model ultimately performs in the end. in building this we were really striving to build something open, sharable and that can kind of fold into production workflows in a grounded way. and along the way we found that we needed to sometimes generate new data and release it to the open source community.

in doing so we created artifacts like PubMed-OCR demonstrated on the left here. so we got a lot to talk about with how we went about building this model. but before we dive too deeply into it, I want to hand off to Ben to tell us a little bit about kind of why we care about this problem.

Why document representation matters in insurance

Ben Elliott, BevayaThanks, Hunter. So, yeah, I'm going to give a little bit of background about sort of some of the problems that we're used to solving and and how solving these problems, informed our decisions and and training GutenOCR. So, we work primarily in insurance. and the classic sort of LLM problem is is document extraction.

we're I think we're unique in in the sense that we like to fine-tune. So, we identify very difficult problems that generalist LLM have trouble solving. We curate the appropriate data and then we train on this data to surpass general accuracy levels to good effect so far. And so here, this document on the left, this is an example of an insurance loss run.

Just a specific but complex and domain knowledge intensive kind of document. This is tended to be our bread and butter. But as we sort of push higher levels of accuracy on this problem, we have to get down into the weeds really about what's going on. And part of that is thinking about how do we represent documents, textual information to LLMs vision models, as we'll talk about later, like they bring about a lot of technical cost.

And so if we can get a near lossless or good enough representation of a document in pure text form, that's what we do today. And and that would be an efficient use of of the technology at you. So currently on the right here we have an example of like something that OCR might do. Currently we use standard OCR pipelines but as we again as we got up to that higher accuracy threshold we we start to see some problems.

so part of our motivation was to nail down a really good document representation which we can then feed into our extraction models. This is something that OCR does a good job doing. We we are good at producing standard OCR type outputs which involve JSONs with bounding boxes and lines or words some form of text. But we can also push beyond this and and there are other applications as well. For example,

Why AI extractions need to be verifiable

Ben Elliott, BevayaSo what we do fundamentally is automate the reading of documents. But automating the reading of a document is somewhat distinct from simply training a model and trying to achieve good scores on a benchmark. We'll talk a lot about benchmarks in the future and also what they mean for our loss runs model, but more than anything, automation is a a human process, right? It's a business process.

And so we need to be able to like make our models interface with humans. Well, for example, if we're looking at this loss run, a reasonable question someone might ask an extraction model is, okay, what was the most expensive claim in this document currently today? Gemini or something like this will tell you okay it's $65,249 and that may be accurate or it may not but independent of its accuracy it needs to be easily verifiable right or else people won't trust the model and we aren't really automating anything so down here I use time complexity notation but really I'm talking about what it costs for a human to go about this process and so if a human needs to verify that $65,249 is the correct answer. They're going to have to read the document in the same way that they would have had to read it if they didn't ask an LLM to make this extraction.

So, in this case, like the the complexity of verification is pretty much the same as the complexity of the human just doing the task themselves. we don't think this is a long run like viable way of going about things. And and so one we need OCR for like the best representation that we can get of a document but then on top of that these grounding models which is the other component grounded OCR VLM right grounding models are going to help as verifiers and and help humans make use of of this LLM extraction process.

What makes loss runs hard for AI

Ben Elliott, BevayaSo now I'm going to talk a little bit about sort of the details about what our loss runs model does because it motivates the way we think not just in terms of why we need an OCR model but how do we go about training models to be efficient with token representation for example so and also what are the types of problems that are good for fine-tuning in like an industrial sense they're the hard problems it turns out open source or I I'm sorry. closed source state-of-the-art models tend to be pretty good at simpler tasks that don't require big token budgets either on the input or the output. But here, it's not the case for loss runs. They're very long and complex documents, 20 to 100 pages.

The outputs can be tens of thousands of tokens for these longer longer documents. Models need to be able to understand different components of of these documents and be able to relate them together. For example, we have this idea of resolution, a claim versus a policy. A loss run has information on both.

So, we need to make statements about each that that agree with each other, right? And this requires domain knowledge. requires a lot of annotation as well. on top of this, there are accounting rules that the outputs need to follow.

This is part of domain knowledge, but also has something to do with how we think about our data as well. And finally, back to the automation point. nobody's interested in a 90% correct extraction of a loss run, right? These are documents which have real bearing on people's lives.

They're used to price insurance policies, for example. And so 98% really is the threshold we need to reach and also a 98% that humans can trust and believe in. Going back to the grounded aspect of our of our model of our OCR model, I mentioned also we like to think about evaluation a lot. Hunter did a lot of work evaluating the OCR model.

We had to think a lot about what do these metrics mean and are the the right the correct metrics to chase. so just a little sort of technical details here. What we do is calculate cell-wise accuracy across different tables that we extract which have varying num numbers of columns. We have our data set fancy X here.

We have this indicator that determines if an extraction is a valid JSON. And so then what we do is we sort of average the cells across a table here in these two inner sums and then we average that average table accuracy across our data set. So another thing thinking about how we want to fine-tune or train models a lot of it comes down to how we define the data that we're interested in outputting and in the same way that we want to represent information in an efficient way that goes in we also need to represent information that comes out efficiently. So I think the standard way of thinking about this representation might be a list of dictionaries.

It's very pythonic, very robust to missing rows for example. But this is very expensive in terms of the tokens you incur. And so we can make a trade-off to a list of list JSON array format where the first list of that array is a is a list of column names that we're interested in extracting and then we just have n many rows for however many data points are in the underlying document. So for a little example oh right and the problem with list of list is that we have commas everywhere.

We have commas inside fields and also commas as delimiters. And LLMs are typically not good at counting. it's something that they've gotten better at, but it requires, and this is part of the reason to fine-tune as well. It requires a model which has seen this counting operation a lot and can do it accurately.

I suppose a little visualization of of what this might look like. You can sort of see on the right hand side here our list of list format. We shrink down tokens by quite a bit. and so when we thought about how what exactly we want our OCR model to output and which tasks it can be good at, we were making a lot of decisions about, okay, how can we maximally give out information but also be easy on the model in terms of context, especially with the limitation of taking in images.

Here's an example of what this actually looks like. this is a non-PII loss run here. so no worries there. But as you can see it gets difficult to start counting and aligning each of these rows with the appropriate or each field with the appropriate column. And so models have to be trained how to do this.

Loss run benchmark results

Ben Elliott, BevayaI spoke a little bit about the success of the model previously. So this is a benchmark we had run maybe eight months ago on the best models of the time. Gemini 2.5 Flash was the most impressive. But as you can see we do very well and this success gave us confidence to apply the same formula to OCR and vision language models.

looking a little more deeply into like particular fields which would be a column in in a like the name of a column in that list of list representation or a key for one of those dictionaries. these are some of the fields that we think require the most domain knowledge which is just the model sees the correct answers which we annotate and then it figures out these rules on its own. but we do quite well here especially on something like claim number. Going back to the automation task, right?

Some of these fields are very important for being able to match entries with you know various profiles or whatnot in insurance claim systems. And so focusing on these things is critical for delivering an industry grade solution.

Where markdown OCR loses information

Ben Elliott, BevayaHowever, like I mentioned before, a lot of the problem with accuracy once you have spent a lot of time annotating your data and understanding the problem comes down to the representation. So if we take a look at this example of a loss run like it's not clear to me what is the best way to represent this that is both efficient and accurate. For example, a lot of open source and closed source OCR solutions currently like to translate things into markdown, but it's markdowns can have tables within tables for example and I think that structure exists on this document. So this was the question which we posed ourselves how can we go about this and of course like it's not a lot of the problem is not a learning problem.

It's not a problem with SFT as an algorithm. It is a problem with the accuracy of the data that you're annotating and if the patterns that exist in the document as we see it, if they themselves are represented in the textual version. So here I took this that document I just showed you and I put it into a markdown OCR style model closed source from LandingAI and I I think we're pretty impressed with how they go about it and I when I first saw this representation I was quite impressed but if you look closely you'll notice that there are things that are off here. So for example, claim number this is the fourth column in this markdown.

But down here the claim number is the fifth column. And if a model is trained to count columns and then make predictions based on those counts, it very well might get this wrong. For example, as well there's this column to the left of these subtables in the underlying document here versus inc colon PD colon. These are separate columns but here they've been conflated and this might be losing information that the model then makes a mistake with.

Moreover, right going back to the grounding problem as opposed to the representation problem, how can we be sure about where certain fields that got extracted were? As you can see, there's this big blue box drawn around the loss run. It's called table here. So LandingAI sees this, they call it a table.

they populate it with information and and we could use this JSON on the right here to like back out where each of these fields are and we could go about this in a programmatic way but it's still quite a bit of effort and I think that the gold standard would be a model that can understand this natively right that can say okay the total incurred for this particular claim was this amount of money and I found that here in this box so these are the things that we've been thinking about as we train a model to do these tasks and fit in with our sort of automation goal. I think with that in mind we can move on to more particulars about how training happened and evaluation happened for this OCR model. So I'll hand off to Hunter.

What counts as OCR?

Hunter Heidenreich, BevayaYeah, I think you'll have kind of heard in in Ben's comments that we're starting to circle this drain of, you know, what is OCR? How should a document be represented? And one of the things that we found and I I guess kind of confounded us is there's not a unified definition here. you have OCR as it's traditionally defined usually with lines and words, sometimes characters even.

in industry it's just accepted how OCR operates. It's just like taken as like when I OCR I get data that I need. I shouldn't have to like think harder about it. and then you have in like LLM AI circles where it seems that right now OCR to them is markdown, which is a little bit at odds with the historical definition of what OCR is.

and so there seems to be this tension, at least to us, we feel like there's this tension between what is OCR? Is it just the text? Is it grounded? grounded at what level?

and to kind of belabor this a little bit. let's say we have this nice fancy academic document. it's column and we get some OCR like this. you might say, well, that's pretty good.

It is telling me that the section title is different than the body text. That's that's pretty nice. but then we kind of go down the column and we hit this point where it wraps back around and you know it still is capturing the section hierarchy. It looks very nice, but we've lost something here.

It may be that we don't need what's lost, but there is information that's been lost. namely that these two columns are side by side and that when we get past the sentence at the bottom of the first column, we're actually traveling quite a bit on the page to the next column. and this is going to be relevant if what we care about is grounding.

Where GPT-5 and Gemini miss on grounding

Hunter Heidenreich, Bevayafurthermore, one of the things that we were trying to do is just kind of get a feel for like these closed source models. And so we took this document and we at the time gave it to GPT-5 thinking. We turned on extended mode and we said hey you know has reverse engineering been applied in this domain. that's what this paper this section is about.

please answer and then draw your support. And so it correctly answers. but in fact the box that it's highlighting here is not in the right location. the the quote that it's highlighting is a couple sentences above where it's highlighted.

So, it's close, but it's inaccurate. And inaccuracy is going to build distrust. and it's not it's not just GPT. No.

it's also is Gemini. And so, what we have here is a very simple case. we're gonna ask Gemini, you know, where is this math equation? It tells us what it is.

It gets it right. but the box, it's nowhere near where we see it on the page. So clearly Gemini is not doing this particular version of the task quite well either. and so what we've ultimately kind of come around to is that well LLM, VLM, OCR, it's not really equivalent to traditional OCR, at least not in the way that we've thought about it in industry.

and that our customers have thought about it. and even when a SOTA model, the closed source model is asked these questions, they tend to struggle. it's worse with open weights in the testing that we've done. And while you could make the argument that yeah, you can get traditional OCR with the word level grounding or the line level grounding and then you can give it to a reasoning model and have it back out where things are.

that's like still not an end to end solution. That's a pipelined approach. and it does kind of at least for us it made us ask like how did the community miss this? Are they missing something?

The problem with today's OCR benchmarks

Hunter Heidenreich, Bevayaso we we the first place we looked were actually at benchmarks that we have. OCRBench v2 is is one of the the newer benchmarks of late. and the reason why I highlight this sample is because, this is one of their instances that they argue is testing OCR abilities. but if you look at this, this is just a linearization of the text.

and in fact, it actually has some typos if you really squint at it. Yet, this is treated as like golden ground truth benchmark data. not ideal. and it's not just one sample.

there are other choices that benchmarks make like when they get around to testing something like localized reading where they give us a region and a model is supposed to like zoom in and read that region. they're making they're making choices for the models. They're making choices about how coordinates should be represented. And just to be honest, there isn't this kind of standardization today in vision language modeling.

so it makes it difficult to use the benchmark and it's not even really testing the core OCR things that we care about. I don't want to just pick on OCRBench v2. So here's another OCR benchmark. it's got text that it's reading.

It's got grounding to this text. You might be like, "Wow, finally we're getting somewhere." unfortunately, I'd argue we're not quite there with this benchmark because this is not very dense text. It's interesting text.

it's laid out in a in a compelling way. but if we're talking about academic documents or dense business contracts or anything like that, this isn't going to really measure how well we can do those kinds of OCR tasks. the other thing I would just call out with this is that if you look at the boxes, I wouldn't argue that these boxes are very accurate and they almost seem heuristically defined in a way that isn't good, which is, you know, granted it is coming from a model, so we can't expect perfection, but maybe we should aim for our benchmarks to have a little bit of a higher bar. because as we've kind of hit on, when you have a bad represent representation produced by OCR, you're going to limit your ability to do the core business tasks you care about.

You're going to break up tables in ways that don't make sense. You're going to glob together information in a way that makes it unparsable. And so we we kind of feel that better benchmarks are needed. something that is like really pushing the boundary of of grounding localization and getting models to like cite the information.

one idea that we were we've been thinking about is just something that can cite at different levels maybe page and and regions. we see GPT and Gemini easing into this space. So we we think it's pretty salient to kind of like push on this and build benchmarks ahead of those models kind of building these capabilities. but it does seem to be one of the things missing.

enough said about benchmarks and and kind of like what we've been missing. I think now's the time that we should start talking a bit about GutenOCR and I'll hand it off to Yosh.

What GutenOCR is designed to do

Yosheb Getachew, BevayaHunter. yeah, so we've heard a lot from Ben and Hunter about why GutenOCR is a very practable practical and desirable model. We've talked about like how there was this kind of like amorphous what what is OCR? How should we define it?

Now that we've done that, we should now talk about how our team went about trying to resolve this by building model. We wanted GutenOCR to be a very versatile model. one that was quite flexible to variety of queries from users. a model that was not also just pigeonholed into performing one simple OCR task, but one that had this availability to provide you things such as all the bounding boxes for all the text.

One that could give you localized information on a specific sentence section of your document. One that could give you very clean outputs that could tie text in to their bounding boxes for some sort of downstream purposes or just give you the text itself but format it in a in a manner that preserves things like relative spacing. just the overall layout of the document. And so to do that, we came together and we thought about the DAHM model as it's humorously named.

this acronym stands for data algorithms, hardware models. And these are the things that you should be thinking about when you're starting to develop model for any use but particularly for some sort of commercial usage. we wanted to think about the data how we could collect it if it was publicly available and if it wasn't how we would go about developing that for ourselves and then eventually sharing that with the broader community. We wanted to think about the algorithms which is how our model would learn whether we using techniques such as SFT or RL.

We want to think about the hardware where the learning occurs. these things are hardware itself is a very important concept to think about because it can be a severe limiter if you're not incredibly intentional from the start and all throughout the training process. we ended up doing single node training on an 8x H100 rack. And furthermore, we wanted to use this hardware setup because we wanted this process to be reproducible.

We released our code for anyone in the community to attempt their own trainings. and not just releasing the code but collecting this hardware would enable most most individuals or most small or large organizations to reproduce what we did. We've seen in other OCR models deprecation of information that would be essential for recreating the experiment or the model and and we want to be different there. And then finally the model itself the base like who learning what are we selecting to be the backbone of this process.

We wanted to be we wanted it to be permissively licensed because we wanted to be you know broadly usable implementable.

Training data: five sources

Yosheb Getachew, Bevayaso for the data we built up our training corpus by combining these five sources here. this large-scale real-world collection of documents in OCR-IDL, TabMe++ and PubMed-OCR complemented by synthetic data from sources such as SynthDoG grounding and grounded or Grounded LaTeX source. These data sources all explicitly would teach our model grounding behaviors such as line detection, math localization, paragraph detection, just everything we wanted for an incredibly robust OCR models. But real data sets as I'll call them are the data sets that are aggregates of real data cover this very pervasive spectrum of noisy business documents as well as long scientific papers.

For example, OCR-IDL and TabMe++ gave us this very diverse this collection of very diverse messy forms with line and word level supervision. such as IDs, invoice forms, claims all with stamps, handwriting, and other artifacts. Our PubMed-OCR data set added dense multi-column academic layouts and most importantly of all, paragraph level annotations. And finally, our SynthDoG grounding and Grounded LaTeX data sources were leveraged target grounding for line level OCR and math.

Grounded LaTeX provided the clean localized mathematical equation data and SynthDoG gave us more dense line level boxes and structured reading. and also added some diversity to the data set because the collection of PDFs that these were sampled from more range from different domains. And now when we talk about data variety data diversity we want to ensure that that isn't merely confined to just the data sources themselves but also how we engage with the data. We want we want to be very comprehensive in our setting up the model so that when it's engaged with by a user it's not it's not you know set in one very rigid conversational flow.

we had four separate task categories which we could talk about more in depth in the Q&A section. I have this slide here that shows one of them conditional detection which that task can just be simply described as if you're given some sort of string X return all the bounding boxes for all matching lines in that image. So you give the model the image and the text you want and it gives you back where that word or that string shows up. Don't want to just simply in our prompting give it if you look at the first column one one method of interpreting that entire task.

We don't want to say where is this text in this image and have the model only respond to that because people ask questions in numerous different ways. we want the model to be exposed to that and that's why we here developed this degeneracy for this single task and how the the question is prompted.

Full-weight SFT and curriculum learning

Yosheb Getachew, BevayaNext algorithms. we had a lot of things to think about there. to keep it more concise, we ended up leveraging full weight SFT, not LoRA training. we experimented with whether we should freeze or unfreeze the vision layers.

and we also utilized a curriculum learning approach where we started our training with our low resolution image and our shorter sequent documents and then we trained up to higher resolution and longer sequence length. And one way to kind of motivate that is if you think about for a VLM the image token length it's of big O of like the the width time height. Basically the image token length is taking into account the area of your entire image. And so when you start to move to higher resolution that's going to take up more space.

so you want to start small and build up, allow the model to develop an intuition and fortify itself before you just jump into the deep end. Right here we have a breakdown for the staging that we performed our training in. given our SynthDoG, Grounded LaTeX, OCR-IDL, TabMe++, and PubMed OCR data, we broke down our training into four different phases. In the first phase, we focused on training the model small length samples which we derived from SynthDoG, Grounded LaTeX, IDL and TabMe++.

And this kind of established the foundation as the core training. From there we transitioned into our first specialization phase where we provided more IDL data, more TabMe++ data or those noisy longer business documents in the length range of 2 to 8k tokens. From there we finally converged on the final stage which we actually decomposed into a stage 3A and a stage 3B. And this is when we specialize the model to our longer denser academic PubMed data.

First in the short length range and then in the long length range. and we found actually in our stage 3A that's where we achieved our highest performances as you'll see later on

Training on 8x H100 GPUs

Yosheb Getachew, BevayaFor our hardware. as I alluded to earlier we trained on DGX 8x H100 racks via Google Cloud. One thing that we really focused on as I also mentioned earlier was we wanted to keep our GPU footprint small because this training process is incredibly sensitive to how much GPU memory is being allotted and is being consumed at any given time. It can make or break the training as as we learned from experience very handily.

so we started using DDP and then transitioned into FSDP for how we were distributing this training across our different H100s. And even in doing so, even in kind of performing this going procedure where we had to tweak our hyperparameters to our data to each training phase, we still encountered multiple encumbering scaling pains. which is to be expected. To circumvent those, we actually extracted a lot of useful information from this Ultra-Scale Playbook from the Nanotron team, which we highly recommend.

Anyone here, if you're ever performing your own independent training, you should dig into this. It was a very useful resource and we were very glad to use it.

Choosing the Qwen backbone

Yosheb Getachew, BevayaAnd then finally, the model, the backbone. there exists multiple different families of model backbones that exists in the machine learning community. all of them have their own unique architectures, sizes, strengths and weaknesses and for this specific use case. They're deliberating over which models would be best for this specific OCR use case.

Our like initial literature review kind of yielded that we should leverage the Qwen family. We found that the Qwen family is an incredibly scalable family. They provide strong variety in the size of the models while furthermore already supporting long context multimodal inputs. And most research in this OCR space has already shown that using Qwen as a backbone can yield results that we wanted to, you know, be in the same range of.

notable also for the Qwen family is its vision encoder. and that uses a native resolution input which means no cropping or no tilting. Preserving this global layout and reading order is critical for strongly aligned grounding. So we found this to be the ideal choice for the task.

And in our training, we used both the 3 billion and the 7 billion 2.5 vision language models because this allowed us to study the OCR quality as a function of model scale while keeping everything else fit. How did we do? Well, I'm going to provide a little sneak peek here before I pass the baton back to Hunter to provide a more rigorous explanation of our evaluation. Here we have couple examples.

I'll direct you to look at the top right figure first. one thing we found is that our model would perform extremely well on OCRing small difficult to see text. in this image, the red boxes correspond to the ground truth and the blue correspond to what our GutenOCR highlighted. In this case, it's the three billion parameter model that's doing the OCRing.

And what you'll notice is if you look at the upper upper if you look at the upper right image top diagram if you look at the bottom of the top diagram you'll see there are some boxes that are covered in a blue highlighted square but not a red. this is because we found that our model would sometimes identify text that wasn't even contained in the ground truths themselves. you could see that also in the top left text figure 7A under IDL. There are some signatures at the at the top that are included in the ground truth, but also that aren't that our model still identifies.

This was extremely promising to us. and I guess I don't want to jump down too quickly, so I'll pass the baton to Hunter to explain what exactly we did for our evaluation suite.

How we evaluated GutenOCR

Hunter Heidenreich, BevayaThanks, Yosh. Yeah. you don't want to get too dazzled by a sample taken here or there. and really we we spent a lot of time thinking about evaluation and how we can get inside of it.

I mean, we talked a lot about benchmarks and we were a little bit frustrated with them. So, we wanted to do better with our own eval. so with a cross-section of tasks that we we're looking at, we inherit a lot of the metrics directly from the community, when we're thinking about basic text reading, we're thinking about character error rates and word error rates. when we're thinking about just detection, we're thinking about the precision and F1 and recall of the boxes produced by the model.

And one of the things we wanted to do is not just, you know, take the representation, produce a markdown, and then score that markdown, but we wanted to actually fold together the grounded information with the text detected. and so we came up with these character error rates that were a little bit more end to end. and we we tried a slate of different open-weight models. we wanted to look kind of in the cohort that we were training with.

we wanted to make sure that, you know, we're doing better than the base model. we're competitive with others that are using similar base models. and we threw in a couple of the more interesting releases like DeepSeek and PaddlePaddle. and one of the things that we we found when we really got into our in domain performance eval is that a lot of these models aren't very good at at grounding.

many of the ones that specialize for quote unquote OCR, they do so at the benefit of reading capabilities, but at the expense of their grounding and localization capabilities. it's not true cross spectrum that this is always the case but it it does seem that for many of the ones like olmOCR that hyper-specialize into a particular reading order hemorrhage a lot of their previous capabilities. something that we found to be super interesting is that the three billion model was quite competitive compared to the 7 billion parameter model. it is it remains true that we find the seven billion is a better reader.

but is the extra four billion parameters worth it? It's not entirely obvious to us that it is especially when if anything we want to find even lighter models that we can produce with higher precision. so we we hope to keep pushing in this domain with like two three billion param models and we don't necessarily see there being a major need for these seven or eight billion per although they are still the best just looking at the raw numbers.

Results on Fox and OmniDocBench

Hunter Heidenreich, Bevayawe don't want to just do our own internal eval. So we did try a couple of the other benchmarks that are out there. one of the things that we wanted to do is is bring them into more of like a VLM like ecosystem so that others could more easily rerun these evaluations and maybe scaffold their own models. a lot of these benchmarks are kind of hodge podge all around with their own scaffolding.

and so we're hoping that with more like open-source pushing like we're doing in GutenOCR, we can kind of bring them better together and make it easier to eval. that said, the Fox benchmark, one of the ones that we were exploring is the focus on anything benchmark. and it it tests a model's ability to read local regions or regions that are pointed at or regions highlighted in a colored box. as well as just the global reading.

And what we found is that our our model was able to maintain these local reading abilities and improve upon them over the base model. whereas some of these models that went very narrow in their OCR lost all capability of doing so. a really cool thing that we found is that we we didn't train our model to even consume points like XY coordinates and read anything about or near them. we just trained the model to read box regions.

nonetheless because of how we trained it and like the variation in the the prompting when we tested it on these okay read this text next to this point it worked really well and we were pleasantly surprised at this like moderate generalization even in this like tiny three billion parameter model. The point remains that like one of the our core motivations is that we want to be able to say that our model can like ground where text is and draw boxes and and give you evidence, right? and so we went to OmniDocBench v1.5 to better test our models abilities at this problem of text detection. There's a little bit of a kind of definitional mismatch here with their benchmark.

they make the assumption that let's say you have a figure or table. They make the assumption that you're not going to OCR the words or lines within that. You're going to detect that as table and figure and then route that in a pipeline to something else and it's going to then render that in proper structure. or they assume that your model's just going to read the whole thing and not treat a table or figure like OCR.

it'll just it'll do a different task there and embed it in a document. so that's not what we directly targeted and we still wanted to use this high quality data as a ruler for how well our model does and did we improve the text detection capabilities. So we approach this through a recall lens where if a line of text is highlighted in OmniDocBench we treat it as golden truth and if our model overpredicts some of the figure stuff we're not holding it against it in this particular test. and what we saw is that broadly our model greatly increased its text recall capabilities.

these weren't across the board. if you look there's a couple of key areas that our model is is a little bit weaker in. you can see like the exam column or the notes column. These represent kind of document types that weren't really in our our open data that we're figuring out ways that we might either obtain similar data or broaden out the the data pipeline to better handle these these kinds of let's say longtail document types.

nonetheless, we see this as like doing exactly what we were hoping to do, drive the text detection performance to higher levels. and we hope to keep pushing on this further. just as a plug, we have our GutenOCR GitHub. this is not the end of the story.

This is the beginning. and we we hope to keep pushing on this to keep adding new evaluations, new benchmarks. we got things in the pipeline planned for new model releases and and more data. So I would say please join us on on the the GitHub.

if you try to use it and you want to drop an issue, drop us an issue. You want to do a PR. Wow, that's amazing. but we we hope that you'll you'll keep engaging with us and will help us continue to drive this.

Q&A: Extracting XHTML

Hunter Heidenreich, BevayaI think that now is probably a good time to maybe pivot to sort of questions and answers. I do see that we have a couple in the chat already. and maybe I will just take this first one. but so we have a question about GutenOCR and its abilities to extract xhtml.

I would say today we are not targeting that as an output representation. Although one of the things that we've been thinking about really really heavily is like what are the core things that a model needs to be able to extract or reproduce such that you can use them to then recreate any document representation or route conditionally into workflows that you care about. a lot of these models, it it seems that they're built kind of with this idea that, oh, you're always going to want a page level representation or you're always going to want a full document level representation. And simply that's just not true from a lot of the things we see in like real business workflows.

You might have something that's a little more nuanced or need a little bit of flexibility. And so what we really are thinking about is like can we satisfy all of what something like XHTML might need us to produce such that then you can render that document if that is the final desired output representation while not sacrificing the ability to like do these core unit capabilities in this OCR space like reading and conditional detection.

Q&A: Pages with mixed text and images

Hunter Heidenreich, BevayaLet's see. We got another question here about mixed pages with text and image images. what are your plans on representing the contents of the images and providing those coordinates? Yeah, I think that's a great question.

one of the big things that we haven't done here that a lot of OCR models in this space are doing is is like a layout detection layer. You want a model that doesn't just read the words on the page, but it also kind of has a a loose semantic understanding of like what are these? Is it a table? Is it a heading?

Is it in a figure? What have you. and so I think what we want to do is is build that as another layer of capability in in a training pipeline like this. we'll teach the model to be able to read the words but also be able to detect those general layout blocks in a very similar fashion to what PaddlePaddle is doing and what DeepSeek is doing.

and then we'll go a little bit further. we don't want to just say you know this is a figure here. We want to be able to provide a I don't want to say lossless. You can't get a lossless caption of a of a figure really, but to the best of the model's ability kind of producing something that when an LLM receives it downstream, there's nothing that it's like hungry for that would have been in that image or that figure to start with.

so it's going to kind of require us to curate data around captioning these things. and then building that as just like another task layer and pairing that with the core layout detection we see as really driving the next iteration

Ben Elliott, BevayaSort of raises questions about like what are all the kinds of metadata about even just text that could be captured like for example one of the I talked about how the loss runs problem motivates us towards GutenOCR in many ways. One thing we noticed with some standard OCR pipeline solutions is that they'll just forget to or fail to capture text that is blue for some reason. And so it sort of got us thinking like, well maybe then this is information that could be useful for an LLM downstream. Like it opens up new avenues for prompting for example like oh this piece of information tends to be blue so take a look for that.

And then we have a little bit of data that says color blue and this makes problems more tractable.

Hunter Heidenreich, BevayaYeah. And and maybe just like piggybacking off of that, you want a model, an OCR model maybe that it doesn't always give you that color information, but it conditionally does when you know you have a problem that you're going to need it for. so that that speaks a little bit to this flexibility that we're really striving for. We don't want to say, "Oh, you have to get the blues and the reds."

And we'll tag that in there. But we do want to say if you want that, yes, that's something that our model can can tag the data with.

Q&A: Using GutenOCR on screens

Hunter Heidenreich, BevayaOkay, we got a question about could this be used with Selenium to click buttons on a screen or enter text into a field if you fed GutenOCR an image of a screen as opposed to send it an image of a document? I I would say screens are probably maybe a little bit out of distribution for it today. although kind of like what we're thinking about intentionally as we build the next iteration is that what is a screen but just like a kind of digital document that maybe is like a little bit more involved. so I think the same capabilities that you need in in that kind of Selenium automation like application, we want to build those into the OCR model because there's no reason that the OCR model you're using to to read the documents can't also be used in that context.

would it be actually entering the text and clicking things? I'm not sure this model would really be what you would want that for, but maybe it's providing the useful context that smarter reasoning model is then utilizing to perform complex workflows.

Q&A: The biggest bottleneck to scaling

Hunter Heidenreich, BevayaMaybe we'll take one of these questions. It says of the axes like models, data, hardware, and algorithms. what is the most significant bottleneck to scaling the performance here? maybe I'll give a perspective and then if Ben or Yosh have any counter thoughts, they could jump in as well.

but I think for me, one of the biggest things that I didn't necessarily fully appreciate coming into this is that when you set yourself up to build something that's like open using open data, there's a lot you cannot use and the quality of data isn't always the best. so many of the the kinds of like really interesting document types we want maybe like newspapers they aren't always in a format or in in licensed in a way that makes them easily ingestable which is a little bit why these AI companies they do like their their data streams right but especially for building this building this further and continually building on this open data pipeline I think we're going to have to do some work on our end to to source more interesting data.

Yosheb Getachew, BevayaYeah. Not only you Oh, thanks Ben. yeah, also just kind of piv piggybacking off of that. I guess I would say I agree completely with Hunter, but also when you're talking about like scaling, it it it can be like kind of tricky how how how you're defining performance.

on one hand if you're defining performance as your accuracy I think there's an argument to be made for like each of these like a better data better performance like a maybe a bigger model better performance hardware obviously was a little difficult to navigate at times but one thing to remember is that they're all interconnected the way I see it when we were starting out we were collecting data like Hunter mentioned that was a very that could be quite cumbersome at times because sometimes there is not high quality data for a specific domain that you been searching for. sometimes you have to create it yourself as we did for PubMed and in doing so that of itself like from scratch creating your own robust data set is is quite intensive procedure. but the size and quantity quantity of your data will affect down the line your hardware how much GPU you're using up the algorithm you're selecting if you're choosing to do full weight SFT that means you're allowing a lot more free parameters that's going to occupy a lot more space as well that's going to affect your scaling the model itself right bigger or smaller that's going to change like what you can do So the way that I like to think about it is it is this pipeline and the decisions that you make early on continue to propagate and gather momentum upon the subsequent ones. typically you kind of witness the effect of it like at the end of the pipeline which is when you're doing your actual training and operating with the hardware itself. but it is kind of a composite of everything that occurred before.

Ben Elliott, BevayaLike for example, we have to feed all of this data, these millions of pages into the hardware. Is that a data problem or a hardware problem? You know, they're are very connected as Yosh is saying

Hunter Heidenreich, BevayaThat's a I think a great point to end it on. So, thanks everybody for attending.

Jessica Carbino, BevayaYes, absolutely. Thank you, Hunter, Yosh, and Ben for your time going through your your many lessons for us. We appreciate everyone for logging on. Again, any questions we didn't have time to cover, we'll reach out to you after the session.

As a reminder, a recording of today's session will be sent to you by tomorrow afternoon. Thanks again for joining. We hope you have a a great day.