I tried using one of these dumps a year ago (wanted to play around and see what visualizations I could come up with based on text and the links between pages) and it was an incredibly unintuitive process.
It's not clear which files you need, and the site itself is (or at least, was when I tried) "shipped" as some gigantic SQL scripts to rebuild the database with enough lines that the SQL servers I tried gave up reading them, requiring another script to split it up into chunks.
Then when you finally do have the database, you don't have a local copy of Wikipedia. You're missing several more files, for example category information is in a separate dump. Also you need wiki software to use the dump and host the site. After a weekend of fucking around with SQL, this is the point where I gave up and just curled the 200 or so pages I was interested in.
I'm pretty sure they want you to "just" download the database dump and go to town, but it's such a pain in the ass that I can see why someone else would just crawl it.
> I tried using one of these dumps a year ago (wanted to play around and see what visualizations I could come up with based on text and the links between pages) and it was an incredibly unintuitive process.
More recently they starting putting the data up on Kaggle in a format which is supposed to be easier to ingest.
I think there are engineers working for crawler companies who are paid well enough to figure out how to do this without kneecapping the most well-known noncommercial projects still surviving on the capitalized internet.
a. unpack the first file
b. use the second file to locate specific articles within the first file; it maps page title -> file offset for the relevant bz2 stream
c. use a streaming decoder to process the entire Wiki without ever decompressing it wholly
4. Once you have the XML, getting at the actual text isn't too difficult; you should use a streaming XML decoder to avoid as much allocation as possible when processing this much data.
What they need to do is have 'major edits' push out an updated static render physical file like old school processes would. Then either host those somewhere as is, or also in a compressed format. (E.G. compressed weekly snapshot retained for a year?)
Also make a cname from bots.wikipedia.org to that site.
This probably is about on-demand search, not about gathering training data.
Crawling is more general + you get to consume it in its reconstituted form instead of deriving it yourself.
Hooking up a data dump for special-cased websites is much more complicated than letting LLM bots do a generalized on-demand web search.
Just think of how that logic would work. LLM wants to do a web search to answer your question. Some Wikimedia site is the top candidate. Instead of just going to the site, it uses this special code path that knows how to use https://{site}/{path} to figure out where {path} is in {site}'s data dump.
> This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models.
Sounds like the problem is not the crawling itself but downloading multimedia files.
The article also explains that these requests are much more likely to request resources that aren't cached, so they generate more expensive traffic.
I need to work with the dump to extract geographic information. Most mirrors are not functioning, take weeks to catch up or block, or only mirror english wikipedia. Every other month I find a work-around. It's not easy to work with the full dumps, but I guess/hope easier than crawling wikipedia website itself.
I don't see an obvious option to download all images from Wikipedia Commons. As the post clearly indicates, the text is not the issue here, its the images.
it seems like Wikimedia Foundation has always been protective of the image downloads since the 90s. So many drunken midnight scripters or new urban undergrad CEOs discovers that they can download cool images fairly quickly. AFAIK there has always been some kind of text corpus available in bulk because it is part of the mission of Wikipedia. But the image gallery is big on disk, big bandwidth compared to TEXT, and low hanging target for the uninformed, greedy and etc.
This phenomenon is the wilful destruction of valuable global commons at the hands of a very small number of companies. The number of individually accountable decision-makers driving this destruction is probably in the dozens or low hundreds.
Everybody and their dog are writing AI scrapers with Captcha passing functionality these days. None of that is new but the scale is unseen.
The thing is also that corporate scrapers are in comparison even the good guys, they respect robots.txt, have properly set user agents etc. Others might do neither and from residential IPs.
The issue isn't even new but any proper solution attempts have been postponed because CDNs seem such an easy solution.
This has become a concern for the Arch Linux wiki which now makes you pass a proof-of-work challenge to read it. Which my anti-fingerprinting browser fails at every time. Putting a burden on human readers that will be only a minor temporary annoyance for the bots. Think about it, the T in CAPTCHA stands for "Turing". What is the design goal of AI? To create machines that can pass a Turing test.
I fear the end state of this game is the death of the anonymous internet.
The point of (correctly done) proof-of-work is not to require Turing-level impersonation. It is to create a cost to a trawler that is going to hit thousands or more of your pages, and almost no cost to a human user.
Problem is, as you've discovered, it can have the cost that anti-fingerprinting browsers can't do the required work.
These are AI bots. Computational capacity is not a limiting factor. I'd argue that my desktop consumer PC is less capable of efficiently solving a PoW than a multi-GPU cluster in a data center.
Even if, as you say, crawlers will hit the PoW thousands of times more, the only way to make it a barrier is if the cost is higher than the profit to be gained. Otherwise it's merely an expense to be passed on to the customer.
A while back I wrote up a way to turn the big Wikipedia XML dump into a database. Not a generic table with articles but thousands of tables, one for each article "type". I'm not sure if this is still the best way to go about it.
Contrary to what most commenters assume, the high bandwidth usage is not coming from scraping text, but images. They are pretty clear about it:
> Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%. This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models.
There's two distinct problems caused by AI scrapers:
1. Bandwidth consumption - that's on scrapers downloading multimedia files
2. CPU resource exhaustion - AI scrapers don't take contextual clues into account. They just blindly follow each and every link they can find, which means that they hit a lot of pages that aren't cached but re-generated for each call. That's stuff like the article history but especially the version delta pages. These are very expensive to generate and are so rarely called that it doesn't make sense to cache them.
You want images only be available to users with a Wikipedia login? This would mean by far most people would no longer see images in Wikipedia articles.
Maybe this is an insane idea, but ... how about a spider P2P network?
At least for local AIs it might not be a terrible idea. Basically a distributed cache of the most common sources our bots might pull from. That would mean only a few fetches from each website per day, and then the rest of the bandwidth load can be shared amongst the bots.
Probably lots of privacy issues to work around with such an implementation though.
Wikimedia's recent post completely misses the mark. What they're experiencing isn't merely bulk data collection – it's the unauthorized transformation of their content infrastructure into a free API service for commercial AI tools.
It's not crawling for training that is the issue...and it's an over simplification stating that AI companies are "training" on someone's data.
When systems like Claude and ChatGPT fetch Wikimedia content to answer user queries in real time, they're effectively using Wikimedia as an API – with zero compensation, zero attribution, and zero of the typical API management that would come with such usage. Each time a user asks these AI tools a question, they may trigger fresh calls to Wikimedia servers, creating a persistent, on-demand load rather than a one-time scraping event.
The distinction is crucial. Traditional search engines like Google crawl content, index it, and then send users back to the original site. These AI systems instead extract the value without routing any traffic back, breaking the implicit value exchange that has sustained the web ecosystem.
Wikimedia's focus on technical markers of "bot behavior" – like not interpreting JavaScript or accessing uncommon pages – shows they're still diagnosing this as a traditional crawler problem rather than recognizing the fundamental economic imbalance. They're essentially subsidizing commercial AI products with volunteer-created content and donor-funded infrastructure.
The solution has been available all along. HTTP 402 "Payment Required" was built into the web's foundation for exactly this scenario. Combined with the Lightning Network's micropayment capabilities and the L402 protocol implementation, Wikimedia could:
- Keep content free for human users
- Charge AI services per request (even fractions of pennies would add up)
- Generate sustainable infrastructure funding from commercial usage
- Maintain their open knowledge mission while ending the effective subsidy
Tools like Aperture make implementation straightforward – a reverse proxy that distinguishes between human and automated access, applying appropriate pricing models to each.
Instead of leading the way toward a sustainable model for knowledge infrastructure in the AI age, Wikimedia is writing blog posts about traffic patterns. If your content is being used as an API, the solution is to become an API – with all the management, pricing, and terms that entails. Otherwise, they'll continue watching their donor resources drain away to support commercial AI inference costs.
I suspect several factors contribute to this resistance:
Ideological attachment to "free" as binary rather than nuanced: Many organizations have built their identity around offering "free" content, creating a false dichotomy where any monetization feels like betrayal of core values. They miss that selective monetization (humans free, automated commercial use paid) could actually strengthen their core mission.
Technical amnesia: The web's architects built payment functionality into HTTP from the beginning, but without a native digital cash system, it remained dormant. Now that Bitcoin and Lightning provide the missing piece, there's institutional amnesia about this intended functionality.
Complexity aversion: Implementing new payment systems feels like adding complexity, when in reality it simplifies the entire ecosystem by aligning incentives naturally rather than through increasingly byzantine rate-limiting and bot-detection schemes.
The comfort of complaint: There's a certain organizational comfort in having identifiable "villains" (bots, crawlers, etc.) rather than embracing solutions that might require internal change. Blog posts lamenting crawler impacts are easier than implementing new systems.
False democratization concerns: Some worry that payment systems would limit access to those with means, missing that micropayments precisely enable democratization by allowing anyone to pay exactly for what they use without arbitrary gatekeeping.
The irony is that Wikimedia already pays for the cost of serving pages — it's just invisible to users because donors cover it. Micropayments via Lightning aren't about "charging for knowledge," they're about sustainable access models in the face of high-frequency bot loads (especially from AI). If AI crawlers are consuming massive resources, it's not unreasonable to explore accountability — not for readers, but for automated extractors.
And, even better, those micropayments could be shared with the volunteers. How about a big party for them, or gifts on Amazon for good behavior? How about a simple birthday card? There's a lot that can be done with resources like this!
Because I evolved the prompt from something that was off to something that made sense. I don't see using any resource as a problem as long as the content is bang on.
The "prompt" as you call it, isn't a sinlge prompt. It's a long discussion that includes the article (which I copy pasta'd) and other references I've worked on recently (I write crawlers and have for years)
Readit News
So it's not like you need to crawl the sites to get content for training your models...
It's not clear which files you need, and the site itself is (or at least, was when I tried) "shipped" as some gigantic SQL scripts to rebuild the database with enough lines that the SQL servers I tried gave up reading them, requiring another script to split it up into chunks.
Then when you finally do have the database, you don't have a local copy of Wikipedia. You're missing several more files, for example category information is in a separate dump. Also you need wiki software to use the dump and host the site. After a weekend of fucking around with SQL, this is the point where I gave up and just curled the 200 or so pages I was interested in.
I'm pretty sure they want you to "just" download the database dump and go to town, but it's such a pain in the ass that I can see why someone else would just crawl it.
More recently they starting putting the data up on Kaggle in a format which is supposed to be easier to ingest.
https://enterprise.wikimedia.com/blog/kaggle-dataset/
https://dumps.wikimedia.org/kiwix/zim/wikipedia/
1. Go to https://dumps.wikimedia.org/enwiki/latest/ (or a date of your choice in /enwiki)
2. Download https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page... and https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.... The first file is a bz2-multistream-compressed dump of a XML containing all of English Wikipedia's text, while the second file is an index to make it easier to find specific articles.
3. You can either:
4. Once you have the XML, getting at the actual text isn't too difficult; you should use a streaming XML decoder to avoid as much allocation as possible when processing this much data.The XML contains pages like this:
so all you need to do is get at the `text`.Also make a cname from bots.wikipedia.org to that site.
Crawling is more general + you get to consume it in its reconstituted form instead of deriving it yourself.
Hooking up a data dump for special-cased websites is much more complicated than letting LLM bots do a generalized on-demand web search.
Just think of how that logic would work. LLM wants to do a web search to answer your question. Some Wikimedia site is the top candidate. Instead of just going to the site, it uses this special code path that knows how to use https://{site}/{path} to figure out where {path} is in {site}'s data dump.
Deleted Comment
Sounds like the problem is not the crawling itself but downloading multimedia files.
The article also explains that these requests are much more likely to request resources that aren't cached, so they generate more expensive traffic.
The thing is also that corporate scrapers are in comparison even the good guys, they respect robots.txt, have properly set user agents etc. Others might do neither and from residential IPs.
The issue isn't even new but any proper solution attempts have been postponed because CDNs seem such an easy solution.
I fear the end state of this game is the death of the anonymous internet.
Problem is, as you've discovered, it can have the cost that anti-fingerprinting browsers can't do the required work.
Even if, as you say, crawlers will hit the PoW thousands of times more, the only way to make it a barrier is if the cost is higher than the profit to be gained. Otherwise it's merely an expense to be passed on to the customer.
https://feder001.com/exploring-wikipedia-as-a-database-part-...
> Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%. This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models.
1. Bandwidth consumption - that's on scrapers downloading multimedia files
2. CPU resource exhaustion - AI scrapers don't take contextual clues into account. They just blindly follow each and every link they can find, which means that they hit a lot of pages that aren't cached but re-generated for each call. That's stuff like the article history but especially the version delta pages. These are very expensive to generate and are so rarely called that it doesn't make sense to cache them.
At least for local AIs it might not be a terrible idea. Basically a distributed cache of the most common sources our bots might pull from. That would mean only a few fetches from each website per day, and then the rest of the bandwidth load can be shared amongst the bots.
Probably lots of privacy issues to work around with such an implementation though.
(91 points, 30 days ago, 101 comments) https://news.ycombinator.com/item?id=43555898
(49 points, 29 days ago, 45 comments) https://news.ycombinator.com/item?id=43562005
It's not crawling for training that is the issue...and it's an over simplification stating that AI companies are "training" on someone's data.
When systems like Claude and ChatGPT fetch Wikimedia content to answer user queries in real time, they're effectively using Wikimedia as an API – with zero compensation, zero attribution, and zero of the typical API management that would come with such usage. Each time a user asks these AI tools a question, they may trigger fresh calls to Wikimedia servers, creating a persistent, on-demand load rather than a one-time scraping event.
The distinction is crucial. Traditional search engines like Google crawl content, index it, and then send users back to the original site. These AI systems instead extract the value without routing any traffic back, breaking the implicit value exchange that has sustained the web ecosystem.
Wikimedia's focus on technical markers of "bot behavior" – like not interpreting JavaScript or accessing uncommon pages – shows they're still diagnosing this as a traditional crawler problem rather than recognizing the fundamental economic imbalance. They're essentially subsidizing commercial AI products with volunteer-created content and donor-funded infrastructure.
The solution has been available all along. HTTP 402 "Payment Required" was built into the web's foundation for exactly this scenario. Combined with the Lightning Network's micropayment capabilities and the L402 protocol implementation, Wikimedia could:
Tools like Aperture make implementation straightforward – a reverse proxy that distinguishes between human and automated access, applying appropriate pricing models to each.Instead of leading the way toward a sustainable model for knowledge infrastructure in the AI age, Wikimedia is writing blog posts about traffic patterns. If your content is being used as an API, the solution is to become an API – with all the management, pricing, and terms that entails. Otherwise, they'll continue watching their donor resources drain away to support commercial AI inference costs.
I suspect several factors contribute to this resistance:
Ideological attachment to "free" as binary rather than nuanced: Many organizations have built their identity around offering "free" content, creating a false dichotomy where any monetization feels like betrayal of core values. They miss that selective monetization (humans free, automated commercial use paid) could actually strengthen their core mission.
Technical amnesia: The web's architects built payment functionality into HTTP from the beginning, but without a native digital cash system, it remained dormant. Now that Bitcoin and Lightning provide the missing piece, there's institutional amnesia about this intended functionality.
Complexity aversion: Implementing new payment systems feels like adding complexity, when in reality it simplifies the entire ecosystem by aligning incentives naturally rather than through increasingly byzantine rate-limiting and bot-detection schemes.
The comfort of complaint: There's a certain organizational comfort in having identifiable "villains" (bots, crawlers, etc.) rather than embracing solutions that might require internal change. Blog posts lamenting crawler impacts are easier than implementing new systems.
False democratization concerns: Some worry that payment systems would limit access to those with means, missing that micropayments precisely enable democratization by allowing anyone to pay exactly for what they use without arbitrary gatekeeping.
But so does unrestricted AI use. I guess the nice things era is over.
And, even better, those micropayments could be shared with the volunteers. How about a big party for them, or gifts on Amazon for good behavior? How about a simple birthday card? There's a lot that can be done with resources like this!
The "prompt" as you call it, isn't a sinlge prompt. It's a long discussion that includes the article (which I copy pasta'd) and other references I've worked on recently (I write crawlers and have for years)