Show HN: Documind – Open-source AI tool to turn documents into structured data

From the source, Documind appears to:

1) Install tools like Ghostscript, GraphicsMagick, and LibreOffice with a JS script. 2) Convert document pages to Base64 PNGs and send them to OpenAI for data extraction. 3) Use Supabase for unclear reasons.

Some issues with this approach:

* OpenAI may retain and use your data for training, raising privacy concerns [1].

* Dependencies should be managed with Docker or package managers like Nix or Pixi, which are more robust. Example: a tool like Parsr [2] provides a Dockerized pdf-to-json solution, complete with OCR support and an HTTP api.

* GPT-4 vision seems like a costly, error-prone, and unreliable solution, not really suited for extracting data from sensitive docs like invoices, without review.

* Traditional methods (PDF parsers with OCR support) are cheaper, more reliable, and avoid retention risks for this particular use case. Although these tools do require some plumbing... probably LLMs can really help with that!

While there are plenty of tools for structured data extraction, I think there’s still room for a streamlined, all-in-one solution. This gap likely explains the abundance of closed-source commercial options tackling this very challenge.

---

1: https://platform.openai.com/docs/models#how-we-use-your-data

2: https://github.com/axa-group/Parsr

themanmaran · 9 months ago

Disappointed to see this is an exact rip of our open source tool zerox [1]. With no attribution. They also took the MIT License and changed it out for an AGPL.

If you inspect the source code, it's a verbatim copy. They literally just renamed the ZeroxOutput to DocumindOutput [2][3]

[1] https://github.com/getomni-ai/zerox

[2] https://github.com/DocumindHQ/documind/blob/main/core/src/ty...

[3] https://github.com/getomni-ai/zerox/blob/main/node-zerox/src...

alchemist1e9 · 9 months ago

Are there any reputation mechanisms or github flagging systems to alert users to such scams?

It’s a pretty unethical behavior if what you describe is the full story and as a user of many open source projects how can one be aware of this type of behavior?

Tammilore · 9 months ago

Hello. I apologize that it came across this way. This was not the intention. Zerox was definitely used and I made sure to copy and include the MIT license exactly as it was inside the part of the code that uses Zerox.

If there's any additional thing I can do, please let me know so I would make all amendements immediately.

dontdoxxme · 9 months ago

For the MIT license to make sense it needs a copyright notice, I don’t actually see one in the original license. It just says “The MIT license” but then the text below references the above copyright notice, which doesn’t exist.

I think both sides here can learn from this, copyright notices are technically not required but when some text references them it is very useful. The original author should have added one. The user of the code could also have asked about the copyright. If this were to go to court having the original license not making sense could create more questions than it should.

tl;dr: add a copyright line at the top of the file when you’re using the MIT license.

sidmo · 9 months ago

If you are looking for the latest/greatest in file processing i'd recommend checking out vision language models. They generate embeddings of the images themselves (as a collection of patches) and you can see query matching displayed as a heatmap over the document. Picks up text that OCR misses. My company DataFog has an open-source demo if you want to try it out: https://github.com/DataFog/vlm-api

If you're looking for an all-in-one solution, little plug for our new platform that does the above and also allows you to create custom 'patterns' that get picked up via semantic search. Uses open-source models by default, can deploy into your internal network. www.datafog.ai. In beta now and onboarding manually. Shoot me an email if you'd like to learn more!

groby_b · 9 months ago

That's not what [1] says, though? Quoth: "As of March 1, 2023, data sent to the OpenAI API will not be used to train or improve OpenAI models (unless you explicitly opt-in to share data with us, such as by providing feedback in the Playground). "

"Traditional methods (PDF parsers with OCR support) are cheaper, more reliable"

Not sure on the reliability - the ones I'm using all fail at structured data. You want a table extracted from a PDF, LLMs are your friend. (Recommendations welcome)

niklasd · 9 months ago

We found that for extracting tables, OpenAIs LLMs aren't great. What is working well for us is Docling (https://github.com/DS4SD/docling/)

emmanueloga_ · 9 months ago

> That's not what [1] says, though?

Documind is using https://api.openai.com/v1/chat/completions, check the docs at the end of the long API table [1]:

> * Chat Completions:

> Image inputs via the gpt-4o, gpt-4o-mini, chatgpt-4o-latest, or gpt-4-turbo models (or previously gpt-4-vision-preview) are not eligible for zero retention."

1: https://platform.openai.com/docs/models#how-we-use-your-data

Deleted Comment

brianjking · 9 months ago

OpenAI isn't retaining your details sent via the API for training details. Stopp.

With such a system, how do you ensure that the extracted data matches the data in the source document? Run the process several times and check that the results are identical? Can it reject inputs for manual processing? Or is it intended to be always checked manually? How good is it, how many errors does it make, say per million extracted values?

glorpsicle · 9 months ago

Perhaps there's still value in the documents being transformed by this tool and someone reviewing them manually, but obviously the real value would be in reducing manual review. I don't think there's a world–for now–in which this manual review can be completely eliminated.

However, if you process, say, 1 million documents, you could sample and review a small percentage of them manually (a power calculation would help here). Assuming your random sample models the "distribution" (which may be tough to define/summarize) of the 1 million documents, you could then extrapolate your accuracy onto the larger set of documents without having to review each and every one.

danbruc · 9 months ago

You can sample the result to determine the error rate, but if you find an unacceptable level of errors, then you still have to review everything manually. On the other hand, if you use traditional techniques, pattern matching with regular expressions and things like that, then you can probably get pretty close to perfection for those cases where your patterns match and you can just reject the rest for manual processing. Maybe you could ask a language model to compare the source document and the extracted data and to indicate whether there are errors, but I am not sure if that would help, maybe what tripped up the extraction would also trip up the result evaluation.

vunderba · 9 months ago

OP, you've been accused of literally ripping off somebody's more popular repository and posing it as your own.

https://news.ycombinator.com/item?id=42178413

You may wanna get ahead of this because the evidence is fairly damning. Failing to even give credit to the original project is a pretty gross move.

Hi. This was definitely not the intention.

I made sure to copy and past the MIT license in Zerox exactly as it was into the folder of the code that uses it. I also included it in the main license file as well. If there's anything I could do to make corrections please let me know so I'd change that ASAP.

ankenyr · 9 months ago

Your initial commit makes it look like you wrote all the code. https://github.com/DocumindHQ/documind/commit/d91121739df038... This is because you copied and uploaded the code instead of forking. You could do a lot by restoring attribution. Your history would look the same as https://github.com/getomni-ai/zerox/commits/main/ and diverge from where you forked.

People are getting upset because this is not a nice thing to do. Attribution is significant. No one would care if you replaced all the names with the new ones in a fork because they would see commits that do that.

infecto · 9 months ago

Multimodal LLM are not the way to do this for a business workflow yet.

In my experience your much better of starting with a Azure Doc Intelligence or AWS Textract to first get the structure of the document (PDF). These tools are incredibly robust and do a great job with most of the common cases you can throw at it. From there you can use an LLM to interrogate and structure the data to your hearts delight.

disgruntledphd2 · 9 months ago

> AWS Textract to first get the structure of the document (PDF). These tools are incredibly robust and do a great job with most of the common cases you can throw at it.

Do they work for Bills of Lading yet? When I tested a sample of these bills a few years back (2022 I think), the results were not good at all. But I honestly wouldn't be surprised if they'd massively improved lately.

Have not used in on your docs but I can say that it definitely works well with forms and forms with tables like a Bill of Lading. It costs extra but you need to turn on table extract (at least in AWS). You then can get a markdown representation of that page include table, you can of course pull out the table itself but unless its standardized you will need the middleman LLM figuring out the exact data/structure you are looking for.

IndieCoder · 9 months ago

Plus one, using the exact setup to make it scale. If Azure Doc Intelligence gets too expensive, VLMs also work great

vinothgopi · 9 months ago

What is a VLM?

bob778 · 9 months ago

From just reading the README, the example is not valid JSON. Is that intentional?

Otherwise it seems like a prompt building tool, or am I missing something here?

Thanks for pointing this out. This was an error on my part.

I see someone opened an issue for it so will fix now.

assanineass · 9 months ago

Oof you’re right LOL

rkuodys · 9 months ago

Just this weekend was solving similar problem.

What I've noticed, that on scanned documents, where stamp-text and handwriting is just as important as printed text, Gemini was way better compared to chat gpt.

Of course, my prompts might have been an issue, but gemini with very brief and generic queries made significantly better results.

inexcf · 9 months ago

Got excited about an open-source tool doing this.

Alas, i am let down. It is an open-source tool creating the prompt for the OpenAI API and i can't go and send customer data to them.

I'm aware of https://github.com/clovaai/donut so i hoped this would be more like that.

_joel · 9 months ago

You can self host OpenAPI compatible models with lmstudio and the like. I've used it with https://anythingllm.com/

Hi. I totally get the concern about sending data to OpenAI. Right now, Documind uses OpenAI's API just so people could quickly get started and see what it is like, but I’m open to adding options and contributions that would be better for privacy.

That sounds great.

I'd recommend checking out vision language models. They generate embeddings of the images themselves (as a collection of patches) and you can see query matching displayed as a heatmap over the document. Picks up text that OCR misses. I built a simple API over it if you want to try it out: https://github.com/DataFog/vlm-api

turblety · 9 months ago

You might be able to use Ollama, which has a OpenAI compatible API.

Zambyte · 9 months ago

Not without chaning the code (should be easy though)

https://github.com/DocumindHQ/documind/blob/d91121739df03867...

slippy · 9 months ago

Legit question: By _removing the MIT license_ from the distribution and replacing it with the AGPL, how are you not violating the copyright and subject to a lawsuit?

The MIT license has just 2 conditions. They are pretty easy to read, and the fist one is:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

By replacing the license, you violate this very simple agreement.

Hi. Thanks for the question. To clarify, the MIT license was never removed or swapped. The license was and still is included in the folder that contains the code from the original project. In the root of the repository, I added the AGPL license for the new code I developed and made sure to explicitly acknowledge that the code in the folder is still under the MIT license.

I’ve also added a direct note acknowledging and linking back to the zerox project.