diptanu

The tricky part of doing this in production is cloning sandboxes across nodes. You would have to snapshot the resident memory, file system (or a CoW layer on top of the rootfs), move the data across nodes, etc.

There was an unusual traffic spike around that time, if you try now it should be a lot faster. We were calling up but there was not enough GPU capacity at that time.

We haven’t tested Chandra yet, because it’s very new. Under the hood Tensorlake is very similar to Marker - it’s a pipeline based OCR API, we do layout detection, Text Recognition and Detection, Table Structure Understanding, etc. We then use VLMs to enrich the results. Our models are much bigger than marker, and thus takes a little longer to parse documents. We optimized for accuracy. We will have a faster API soon.

It does, we have users in Europe and Asia using it with non English languages. Can you please send me a message at diptanu at tensorlake dot ai, would love to see why it didn’t work.

OP mentioned Gemini and not Google’s Vertex OCR API which has very different performance and accuracy characteristics than Gemini

Hey! I am the founder of Tensorlake. We benchmarked the models that our customers consider using in enterprises or regulated industries where there is a big need for processing documents for various automation. Benchmarking takes a lot of time so we focussed on the ones that we get asked about.

On Gemini and other VLMs - we excluded these models because they don't do visual grounding - aka they don't provide page layouts, bounding boxes of elements on the pages. This is a table stakes feature for use-cases customers are building with Tensorlake. It wouldn't be possible to build citations without bounding boxes.

On pricing - we are probably the only company offer a pure on-demand pricing without any tiers. With Tensorlake, you can get back markdown from every page, summaries of figures, tables and charts, structured data, page classification, etc - in ONE api call. This means we are running a bunch of different models under the hood. If you add up the token count, and complexity of infrastructure to build a complex pipeline around Gemini, and other OCR/Layout detection model I bet the price you would end up with won't be any cheaper than what we provide :) Plus doing this at scale is very very complex - it requires building a lot of sophisticated infrastructure - another source of cost behind modern Document Ingestion services.

We parse PDFs to convert them to text in a linearized fashion. The use case for this would be to use the content for downstream use cases - search engine, structured extraction, etc.

Yeah we don't handle this yet.

Yes this! We training it on a ton of diverse document images to learn reading order and layouts of documents :)

There are many cases images are exported as PDFs. Think invoices or financial statements that people send to financial services companies. Using layout understanding and OCR based techniques leads to way better results than writing a parser which relies on the files metadata.

The other thing is segmenting a document and linearizing it so that an LLM can understand the content better. Layout understanding helps with figuring out the natural reading order of various blocks of the page.

Disclaimer - Founder of Tensorlake, we built a Document Parsing API for developers.

This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs.

We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recognition models on them, stitch them back together to get acceptable results for domains where accuracy is table stakes.

We started with using LLMs for parsing at Tensorlake (https://docs.tensorlake.ai), tried Qwen, Gemini, OpenAI, pretty much everything under the sun. My thought was we could skip 5-6 years of development IDP companies have done on specialized models by going to LLMs.

On information dense pages, LLMs often hallucinate half of the times, they have trouble understanding empty cells in tables, doesn't understand checkboxes, etc.

We had to invest heavily into building a state of the art layout understanding model and finally a table structure understanding for reliability. LLMs will get there, but there are some ways to go there.

Where they do well is in VQA type use cases, ask a question, very narrowly scoped, they will work much better than OCR+Layout models, because they are much more generalizable and flexible to use.

I don’t think what you are describing as heavy is that big of a deal if an external orchestration system is required only for deployment, while the workflow can be developed and tested without a server on a laptop or notebook.

Bringing in orchestration logic in the app layer means there is more code being bundled with the app, which has its own set of tradeoffs - like bringing in a different set of code dependencies which might conflict with application code.

In 2025, I would be surprised if a good workflow engine didn’t have a completely server-less development mode :)

Great points. Besides performance, centralized coordination and distributed dataplane is better for operability of schedulers as well. Some examples - Being able to roll out new features in the scheduler, tracing scheduling behavior and decisions, deploying configuration changes.

Even with a centralized scheduler it should be possible to create a DevEx that makes use of decorators to author workflows easily.

We are doing that with Indexify(https://github.com/tensorlakeai/indexify) for authoring data intensive workflows to process unstructured data(documents, videos, etc) - it’s like Spark but uses Python instead of Scala/SQL/UDFs. Indexify’s scheduler is centralized and it uses RocksDB under the hood for persistence, and long term we are moving to a hybrid storage system - S3 for less frequently updated data, and SSD for read cache and frequently updated data(on going tasks).

The scheduler’s latency for scheduling new tasks is consistently under < 800 microseconds on SSDs.

This is how schedulers have been designed traditionally that have a proven track record of working in production - Borg, Hashicorp Nomad, etc. There are many ways a centralized scheduler can scale out beyond a single machine - parallel scheduling across different by sharding jobs, node pools, and then linearizing and deduplicating conflicts during writes is one such approach.

Love DBOs and Hatchet! cheering for you @jedberg and @abelanger :-)

Disclaimer - I am the founder of Tensorlake, and worked on Nomad and Apache Mesos in the past.

We use grid search to figure out what's the best chunking strategy to use. Create a bunch of different strategies such as recursive chunking, semantic, etc, and parameterize them and see which one works best. The "best" chunking strategy depends on the nature of the documents and the questions being asked.

Rust is really good for building solid services. Indexfiy has never crashed for us in unexpected ways. The other option was to build this in C++ but the state of package management in C++ is really bad so I chose to use Rust.

Sorry just seeing this! There are various aspects of how it handles ingestion spikes - 1. The ingestion api writes to blob stores, which are horizontally scalable. Only when an ingestion finishes, we write the metadata to the replicated state machine. 2. The replicated state machine takes 100k IOPs on most commodity machines, and they can vertically scale. 3. The extractors can autoscale based on the number of tasks in the system. 4. The ingestion server cluster can autoscale also based on the amount of IOPs they are doing.

Hope this answers the question :)

Thanks! It's pretty general purpose. It has a retrieval API for RAG use cases. It can be used for building agents too, which might only care about data from certain data sources and get invoked on any changes. Some of our users use it for just data extraction from PDFs.

The patterns for embedding, structured extraction for different data types doesn't change much for each use-cases, if the underlying API and storage subsystem is flexible.

This approach was studied in a paper called Evaporate+ - https://www.vldb.org/pvldb/vol17/p92-arora.pdf They used active learning to pick the best function among candidate functions generated by the LLM on a sampled set of data.